Developing Students' Self-regulated Learning Skills with Teacher Classroom Analytics Enhancing Teachers' Direct Instruction of Self-regulated Learning Strategies

Abstract

Developing self-regulated learning (SRL) skills is crucial for students. They develop these skills as early as during the primary school years. Nonetheless, previous research indicates that students struggle with monitoring and controlling their learning. Teachers play a substantial role in developing SRL. Young students benefit from direct instruction in SRL strategies. However, for teachers monitoring students' SRL and providing appropriate and timely support are challenging tasks. Adaptive learning technologies (ALTs) are widely used in educational settings to support math learning. Although ALTs externally regulate the learning process by adapting the difficulty of problems, students are still responsible for choosing the appropriate level of effort, monitoring their accuracy, and setting learning goals. Teacher dashboards visualizing the learning process on ALTs may provide teachers with information on the learners' progress which might help them monitor students' SRL processes effectively. However, most dashboards do not provide information on SRL and target teachers' instruction. Thus, we aim to develop a teacher dashboard that provides visualized information on classroom-level SRL to facilitate teachers' instruction of SRL strategies. Subsequently, we will investigate whether this classroom-level teacher dashboard enhances teachers' instruction of SRL strategies, which may increase the SRL skills of primary school students during math learning.

Keywords

• eol>Self-regulated learning
• direct strategy instruction
• learning analytics
• teacher dashboards
• adaptive learning technologies1

1. Introduction

Effective use of self-regulated learning (SRL) enhances the academic achievement of students [ 1 ]. Therefore, it is crucial for learners to develop SRL skills. Selfregulated learners are characterized by being able to plan, monitor, and control their learning [ 2 ]. However, students tend to have utilization deficiency [ 3 ], meaning they have difficulties activating the monitor and control loops while learning [ 4 ]. Hence, they may benefit from external support, especially from teachers and learning technologies [ 5 ]. Recently, more than half of the primary school students in the Netherlands practice math using adaptive learning technologies (ALTs) [ 6 ]. When students fail to internally regulate, ALTs partially take over the monitoring and control loops by adjusting the difficulty of the problem based on learners’ knowledge and selecting appropriate tasks [ 7 ]. Although ALTs externally regulate the process, students still need to apply appropriate effort to enhance their accuracy while engaging in the tasks, which is a crucial component of self-regulated learning (SRL) process [ 8 ]. Teachers should scaffold strategy use in SRL until students learn to self-regulate their own learning [ 9 ]. Teachers can promote students’ SRL by teaching learning strategies [ 1 ]. Teacher dashboards may contribute to the development of self-regulated learning in students by providing teachers cues regarding the learning processes of students. Thus, the goal of this project is to examine whether teacher dashboards visualizing information about students’ SRL during learning enhance teachers’ strategy instruction of SRL, which in turn increases student learning and SRL skills. In other words, we focus on examining the role of teacher dashboards in improving teachers’ strategy instruction that fosters students’ SRL.

1.1. Self-regulated learning

According to the COPES model [ 10 ], self-regulating learners go through a loosely sequenced and recursive pattern that consists of four phases: (1) defining the task in which learner develop an understanding of the task, (2) setting goals and plans in which learner set and plan goals, (3) engagement in which learner work on their plans, control and monitor their progress, and (4) large-scale adaptation in which learner makes adjustments if the progress does not match with the plans [ 11 ]. Learners are not required to go through phases in a sequence, meaning that they can move from one phase to another anytime [ 10 ]. Control and monitoring loops are in the key point of SRL. These loops facilitate students’ evaluation of the effectiveness of their learning and help them adjust their effort appropriately [ 7 ]. However, most students fail to regulate this process, and this utilization deficiency leads to less effective and efficient learning [ 12 ].

1.2. The role of teachers and ALTs in promoting self-regulated learning

Research indicates that students tend to either overestimate or underestimate their performance [ 4 ], which may stem from poor SRL skills. External feedback from teachers and learning technologies may enhance students’ judgements regarding their learning process [ 5 ]. A large number of intervention studies show that SRL skills can be fostered in primary schools [ 13 ]. As younger students have less experience in learning, they may need more direct instruction of SRL strategies than the older ones [ 9 ]. Teachers may contribute to students’ SRL skills directly by instructing strategies implicitly or explicitly [ 1 ]. Implicit strategy instruction involves teachers’ modeling the strategy use without explicitly mentioning the strategy while explicit instruction comprises teachers’ demonstrating the strategy by referring to the strategy [ 2 ]. It is stressed that implicit and explicit strategy instructions build on each other and both are crucial for students’ SRL [ 14 ].

In addition to teacher instruction of SRL strategy, ALTs intervene the regulation process of students by taking over the monitoring and controlling loops. They select appropriate tasks suitable to learners’ goals and adjust the difficulty of the tasks based on students’ performance [ 7, 15 ]. Although ALTs externally regulate learning process, students are still responsible for adjusting their effort and monitor their accuracy in solving problems, which are closely related to having better SRL skills.

1.3. Teacher dashboards

While learning with ALTs, students leave traces of data containing rich source of information regarding their learning processes [ 16 ]. The trace data stores logs showing which activities students engage in and how they progress, thus, it provides a source regarding their SRL [ 17 ]. Learning analytics dashboard is defined as “a single display that aggregate different indicators about learner(s), learning process(es) and/or learning context(s) into one or multiple visualization” [ 18 ]. Teacher dashboards are used as a tool to capture and visualize the trace data regarding students’ learning. They may enhance teachers’ understanding of the situation in the class and their awareness of student needs by providing visual information [ 19 ], which may support teachers in taking appropriate pedagogical actions. Indeed, researchers found that teacher dashboards had an effect on teachers’ daily teaching practices [ 20 ], supporting this notion. Teacher dashboards display aggregated and real-time information regarding students, which may help providing appropriate and timely support [ 19 ]. However, although studies show that teachers take pedagogical actions based on the dashboard information, the majority of teacher dashboards were not designed to include SRL data [ 21 ] and none of them target improving teachers’ direct strategy instruction. Most of the dashboards targeting students’ SRL were developed to be used in higher education settings by students [ 22 ] although developing SRL skills is crucial for young students. There is also lack of theoretical grounding in most of the dashboard designs [ 23 ].

Thus, with this research project, we propose to design a teacher dashboard and test its effectiveness in informing teachers about students’ SRL processes during learning. A teacher dashboard may contribute to teachers’ decision process of when and how to instruct SRL strategies at a classroom-level. We expect that when teachers provide direct strategy instruction for SRL, students’ SRL skills will increase consequently.

1.4. Design and development of teacher dashboards

It is crucial for teachers to identify student needs quickly and accurately to improve their SRL. Teacher dashboards facilitate teachers’ monitoring of students’ progress by informing them through visualizations [ 19 ]. Therefore, the design of the teacher dashboards is important to make the interpretation of the information on the dashboard easier for teachers. The design of the teacher dashboard plays also an important role in the effectiveness of the dashboard [ 24 ].

In this project, we will develop a classroom-level teacher dashboard to enhance teachers’ strategy instruction of SRL skills using a user-centered approach. Research indicates that a successful implementation of dashboards into teachers’ practices requires a solid fit between the information on the dashboard and the teachers’ beliefs, motivations, and teaching habits [ 20 ]. Thus, in addition to the previous literature on SRL indicators and COPES model, we will also consult teachers’ ideas and experiences regarding the SRL indicators and visualization of them in the design process by conducting one-on-one semistructured interviews with teachers. In addition, research highlights the lack of theoretical grounding in the development of learning analytics dashboards [ 23 ]. In this project we will use the COPES model as a theoretical basis for the SRL to visualize students’ learning process, as this model is widely used in research on technology supported learning [ 25 ]. We will also follow a framework developed by van Leeuwen and colleagues [ 15 ] to help teachers turning dashboard data into action. According to this framework, teachers should be aware of the information displayed on the dashboard (awareness), make sense of the information shown on the dashboard (interpretation), and finally turn the interpretation into pedagogical action (enactment). Hence, teachers’ awareness, interpretation, and enactment of the teacher dashboard information will be prioritized in the design process. As well as teachers’ needs and suggestions regarding relevant SRL indicators, we will build on previous studies to provide visualizations [ 7, 19, 26 ]. The different learner profiles and paths depicted by “moment-by-moment learning curves” can be used as an indicator of SRL while learning with ALTs [ 7 ]. The effectiveness of using these visualizations have been shown in interventions regarding SRL [ 11, 27 ]. the Figure 1 illustrates how moment-by-moment learning curves can be presented through a teacher dashboard providing an overview of the class. Based on the classroom-level information regarding the student activity on the dashboard, teachers can determine the needs of students and provide strategy instruction based on these needs.

1.5. Evaluation of the classroomlevel teacher dashboard

The usability and effectiveness of the classroom-level teacher dashboards will be evaluated through a lab study and short- and long-term classroom experiments. Lab and field experiments provide fair comparisons between conditions and consequently increase the validity of the research. Thus, we will first conduct a lab study after designing the teacher dashboard. This study will provide a preliminary insight into how teachers may shape their direct instruction of SRL strategies based on the dashboard data. The lab study will be followed by short- and longterm classroom experiments to gain insight into actual use of the dashboard in the classroom setting. Classroom experiments will allow us to investigate the effects of teacher dashboards on teacher behavior and consequent student learning and SRL skills. The longterm experiment followed by the short-term experiment will provide teachers and students enough time to get used to working with the dashboards and allow novelty effects to wear off. Besides, it will give us an opportunity to observe the hypothesized effects of the classroom-level teacher dashboard on students’ development of SRL skills in time. We propose developing and testing teacher dashboards visualizing learning analytics data to enhance teachers’ direct instruction of SRL strategies. The scope and quality of teachers' support for students' SRL are often constrained as they do not have much insight into students’ SRL during learning. Thus, this project focuses on classroom-level teacher dashboards to contribute to teachers’ role in the development of SRL skills. The overarching research question is: “How do teacher dashboards support the development of primary school students’ SRL skills?”. Throughout the project, we will address the following research questions (RQs):

[RQ1] Which SRL indicators are relevant and actionable for primary school teachers to provide direct strategy instruction during math classes to support students’ SRL?

[RQ2] What are teachers’ preferences and concerns regarding the presentation and aggregation of the SRL information in dashboard prototypes? [RQ3] How do teachers evaluate the indicators and visualizations shown on the dashboard prototypes concerning their clarity and actionability to inform teachers’ instruction of SRL strategies, and how can they be optimized?

[RQ4] How can the teacher dashboard enhance teachers’ direct instruction of SRL strategies during math classes?

[RQ5] What are the short-term effects of classroom-level teacher dashboard on teachers’ direct strategy instruction during math classes and students’ SRL?

[RQ6] What are the long-term effects of classroomlevel teacher dashboard on teachers’ direct strategy instruction during math classes and students’ SRL?

2. Methodology

In this study, we will follow a mixed methods approach, as we aim to address the preferences and concerns of teachers iteratively in the design process and empirically test the effectiveness of the tool. To accomplish that, we will collect both qualitative and quantitative data by conducting semi-structured interviews and setting up a vignette study in the lab and field experiments in classrooms throughout the project. In the first phase, a literature study will be conducted on (1) existing SRL dashboards and visualizations and (2) indicators regarding students’ self-regulation during learning. Following a usercentered approach, we will invite teachers for one-onone semi-structured interviews to investigate the relevant and actionable SRL indicators. During the first round of interviews, teachers (n = 10) will be shown storyboards [ 28 ] depicting possible scenarios they may face while assessing students’ self-regulation and will be asked to share ideas on the use of teacher dashboards in facilitating students’ SRL at the classroom level. Storyboards help designers prioritize the needs of targeted users and give a clear idea of room for innovation [ 29 ]. Afterwards, reflective questions regarding the SRL indicators and design aspects will be posed to obtain deeper understanding of teachers’ preferences and needs (RQ1). Based on these interviews, the first low-fidelity prototypes of the teacher dashboard will be created using Miro software. We will create two prototypes to investigate teachers’ preferences regarding the aggregation of the classroom data. While one of the prototypes will aggregate SRL data mainly on the class and group level, the other prototype will provide also a closer look to the individual level SRL next to the classroom and group level information. In the second round, a group of teachers (n = 10) will be invited for one-on-one semi-structured interviews to evaluate and optimize the features of these low-fidelity prototypes. By posing interview questions supported by the prototypes and classroom scenarios, we will explore whether teachers are able to understand and use the data shown on the dashboards to improve their direct strategy instruction (RQ2, RQ3). After the second interview session, a clickable medium-fidelity classroom-level dashboard will be created based on teachers’ evaluations and suggestions so that teachers can interact with the dashboard. This medium-fidelity prototype will be tested in a lab study.

In the lab study, the classroom-level teacher dashboard prototype will be tested to examine how it may enhance teachers’ direct instruction of SRL strategies during math learning (RQ4). Vignettes depicting various actual learning situations in the class will be used to investigate teachers’ instruction practices systematically. Each vignette will correspond to one of the SRL phases. Using the teacher dashboard, we will present simulated data at the classroom-level and ask teachers (n = 10) to prepare a lesson plan based on the information displayed on the teacher dashboard. The lesson plans will be coded using Assessing How Teachers Enhance Self- regulated Learning (ATES) [ 9 ] instrument by the researchers.

In the next phase, a 1-week experiment will be conducted to investigate the short-term effects of classroom dashboard on teachers’ direct strategy instruction and students’ SRL (RQ5). Three experimental conditions will be compared: no dashboard, classroom- and individual-level dashboard, and only classroom-level dashboard. Each condition will involve ten teachers (n = 30) and their classes. As this is an interlinked project, we will collaborate with the researchers at Radboud University who are developing the individual-level teacher dashboard to test its effects on teachers’ feedback practices during data collection process to increase the feasibility. This study will follow a strictly controlled setup with three pre-selected learning goals and four lessons in Gynzy (a widely used ALT in the Netherlands). Figure 3 presents the study setup. Students’ learning will be measured with curriculumspecific pre-, post-, and transfer-test. Students’ SRL skills will be measured using Metacognitive Strategy Inventory for Learning with Hypermedia (MESH) [ 30 ] questionnaire. Classroom observations will be conducted for each teacher during lessons 2 or 3 to assess teachers’ direct strategy instruction using the ATES.

In the last phase of the study, a two-month longterm classroom experiment will be conducted to investigate the long-term effects of the classroom-level teacher dashboard on teachers’ direct strategy instruction (RQ6). Additional two months is expected to provide enough time for teachers and students to get used to the dashboards. It also allows researchers enough time to observe hypothesized effects of the dashboards on students’ development of SRL skills. This study will follow the same setup as the phase 3 except that a sequence of 6 lesson blocks will be used. Learners will work with Gynzy app for 6 consecutive weeks on 6 different math topics and curriculumspecific pre-, post-, and transfer- tests will be conducted at the start and end of the 6 weeks. Students’ SRL skills are measured with the MESH questionnaire at the start and the end of each lesson.

The data collection for this study will be done in collaboration with the research team in Radboud University. Teachers (n = 40) will be assigned to either no dashboard or classroom- and individuallevel dashboard conditions. Classroom observations will be conducted to investigate teaches’ strategy instruction at three timepoints (week 1, 3 and 6). Teachers’ behaviour will be coded using ATES instrument similarly to Study 3. Figure 4 shows the planning of the research project.

2.1. Data Analysis Plan

Semi-structured interviews will be audiotaped and transcribed. Transcriptions will be coded based on the four phases of SRL in the COPES model [ 3 ], classroom instruction, and dashboard design. The guidelines of Saldaña [ 31 ] will be followed during the coding process. The interview data will be analyzed using a deductive content analysis approach. Subsequently, a more inductive approach will be used to have a closer look in teachers’ preferences and needs while monitoring SRL of the class [ 32 ].

Lesson plans prepared by teachers based on the displayed vignettes during the lab study will be systematically coded using ATES instrument following the coding scheme. Teacher scores will be operationalized as the degree to which teacher promotes SRL in students through explicit instruction.

Since the short- and long-term experiment data will have a hierarchical structure, multilevel modeling for repeated measures will be used to analyse the data. Measurement occasions (level 1) will be nested within students (level 2), and students will be nested within classrooms (level 3).

3. Progress so far

We conducted the first phase of the semi-structured interviews supported with storyboards and reflective questions with 10 primary school teachers teaching students aged 9-12 years. We transcribed, coded, and analyzed the interviews. Currently, we are designing the low-fidelity prototype of the classroom-level teacher dashboard based on the input from teachers and the literature to be used in the second phase of the semi-structured interviews.

4. Theoretical and Practical

Contributions

This research project will facilitate teachers’ role in monitoring crucial SRL skills through a theory-driven and empirically-tested design of a classroom-level teacher dashboard, which in turn is expected to contribute to students’ learning. To our knowledge, this is the first teacher dashboard to target improving teachers’ direct instruction of SRL strategies. We will develop new data visualizations for teachers using learning analytics solutions and educational data mining techniques through which we will be able to investigate the transitional process from teachers’ instruction to students’ SRL skills. Therefore, this research will provide an understanding of how teachers use dashboards to support their instructional practices. This understanding may contribute to the further development of technology-enhanced learning instruments.

Acknowledgements

This research is supported by a grant from the Netherlands Initiative for Educational Research (NRO), with a grant number 40.5.20300.009. It is a part of an interlinked project involving researchers from Utrecht University and Radboud University. The research project in Utrecht University is promoted by Prof. Dr. Liesbeth Kester and supervised by Dr. Jeroen Janssen and Dr. Anouschka van Leeuwen. The research project in Radboud University is led by Susan Janssen, MSc, promoted by Prof Dr. Eliane Segers, and supervised by Prof. Dr. Inge Molenaar and Dr. Carolien Knoop-van Campen.

References

[1] S. Kistner , K. Rakoczy , B. Otto , C. Dignath-van Ewijk , G. Büttner , E. Klieme, Promotion of selfregulated learning in classrooms: Investigating frequency, quality, and consequences for student performance , Metacognition Learn. 5 . 2 ( 2010 ) 157 - 171 . doi: 10 .1007/s11409-010-9055-3.

[2] C. Dignath , M. V. J. Veenman , The role of direct strategy instruction and indirect activation of self-regulated learning-evidence from classroom observation studies, Educ . Psychol. Rev. ( 2020 ). doi:10.1007/s10648-020-09534-0.

[3] Studying as Self-Regulated Learning, in: Metacognition in educational theory and practice , Routledge , 1998 , pp. 291 - 318 . doi: 10 .4324/ 9781410602350 - 19 .

[4] I. Molenaar , A. Horvers , R. Dijkstra , Young learners' regulation of practice behavior in adaptive learning technologies , Front. Psychol . 10 ( 2019 ). doi: 10 .3389/fpsyg. 2019 . 02792 .

[5] W. Matcha , N. A. Uzir , D. Gasevic , A. Pardo , A systematic review of empirical studies on learning analytics dashboards: A self-regulated learning perspective , IEEE Trans. Learn. Technol. 13.2 ( 2020 ) 226 - 245 . doi: 10 .1109/tlt. 2019 . 2916802 .

[6] Kennisnet , Onderwijs in een kunstmatig intelligente wereld - Kennisnet Technologiekompas 2019-2020 , 2019 .

[7] I. Molenaar , A. Horvers , R. S. Baker , What can moment-by-moment learning curves tell about students' self-regulated learning?, Learn. Instr. ( 2019 ) 101206 . doi: 10 .1016/j.learninstruc. 2019 . 05 .003.

[8] P. H. Winne , Improving measurements of selfregulated learning , Educ. Psychol . 45 .4 ( 2010 ) 267 - 276 . doi: 10 .1080/00461520. 2010 . 517150 .

[9] C. Dignath , G. Büttner, Teachers' direct and indirect promotion of self-regulated learning in primary and secondary school mathematics classes - insights from video-based classroom observations and teacher interviews , Metacognition Learn. 13.2 ( 2018 ) 127 - 157 . doi: 10 .1007/s11409-018-9181-x.

[10] P. H. Winne , Theorizing and researching levels of processing in self-regulated learning , Br. J. Educ. Psychol . 88 .1 ( 2017 ) 9 - 20 . doi: 10 .1111/bjep.12173.

[11] I. Molenaar , A. Horvers , R. Dijkstra , R. S. Baker , Personalized visualizations to promote young learners' SRL, in: LAK '20: 10th international conference on learning analytics and knowledge , ACM, New York, NY, USA, 2020 . doi: 10 .1145/3375462.3375465.

[12] R. Azevedo , D. C. Moos , J. A. Greene , F. I. Winters , J. G. Cromley , Why is externally-facilitated regulated learning more effective than selfregulated learning with hypermedia? , Educ. Technol. Res. Dev. 56.1 ( 2007 ) 45 - 72 . doi: 10 .1007/s11423-007-9067-0.

[13] M. De Smul , S. Heirweg , G. Devos, H. Van Keer , School and teacher determinants underlying teachers' implementation of self-regulated learning in primary education , Res. Pap. Educ. 34.6 ( 2018 ) 701 - 724 . doi: 10 .1080/02671522. 2018 . 1536888 .

[14] Zelfregulerend leren gaat niet vanzelf: Maar hoe dan wel? Openbare les , Hogeschool Rotterdam, Rotterdam, 2023 .

[15] A. van Leeuwen , C. A. N. Knoop-van Campen , I. Molenaar , N. Rummel , How teacher characteristics relate to how teachers use dashboards: Results from two case studies in K12 , J. Learn. Anal. 8 . 2 ( 2021 ) 6 - 21 . doi: 10 .18608/jla. 2021 . 7325 .

[16] D. Gašević , S. Dawson , G. Siemens, Let's not forget: Learning analytics are about learning , TechTrends 59.1 ( 2014 ) 64 - 71 . doi: 10 .1007/s11528-014-0822-x.

[17] E. Araka , E. Maina, R. Gitonga , R. Oboko, Research trends in measurement and intervention tools for self-regulated learning for e-learning environments-systematic review (2008-2018), Res . Pract. Technol. Enhanc. Learn. 15.1 ( 2020 ). doi:10.1186/s41039-020-00129-5.

[18] B. A. Schwendimann , M. J. Rodriguez-Triana , A. Vozniuk , L. P. Prieto , M. S. Boroujeni , A. Holzer , D. Gillet , P. Dillenbourg , Perceiving learning at a glance: A systematic literature review of learning dashboard research , IEEE Trans. Learn. Technol. 10.1 ( 2017 ) 30 - 41 . doi: 10 .1109/tlt. 2016 . 2599522 .

[19] A. van Leeuwen , N. Rummel , T. van Gog, What information should CSCL teacher dashboards provide to help teachers interpret CSCL situations? , Int. J. Comput. Collab. Learn. 14.3 ( 2019 ) 261 - 289 . doi: 10 .1007/s11412-019- 09299-x.

[20] I. Molenaar , C. A. N. Knoop-van Campen , How teachers make dashboard information actionable , IEEE Trans. Learn. Technol. 12.3 ( 2019 ) 347 - 355 . doi: 10 .1109/tlt. 2018 . 2851585 .

[21] M. D. Wiedbusch , V. Kite , X. Yang , S. Park , M. Chi , M. Taub , R. Azevedo , A theoretical and evidencebased conceptual design of metadash: An intelligent teacher dashboard to support teachers' decision making and students' selfregulated learning , Front. Educ . 6 ( 2021 ). doi: 10 .3389/feduc. 2021 . 570229 .

[22] R. Perez-Alvarez , I. Jivet , M. Perez-Sanagustin , M. Scheffel , K. Verbert , Tools Designed to Support Self-Regulated Learning in Online Learning Environments: A Systematic Review , IEEE Trans. Learn . Technol. ( 2022 ) 1 - 18 . doi: 10 .1109/tlt. 2022 . 3193271 .

[23] K. Verbert , X. Ochoa , R. De Croon, R. A. Dourado , T. De Laet, Learning analytics dashboards , in: LAK ' 20 : 10th international conference on learning analytics and knowledge , ACM, New York, NY, USA, 2020 . doi: 10 .1145/3375462.3375504.

[24] A. F. Wise , Y. Jung , Teaching with analytics: Towards a situated model of instructional decision-making , J. Learn. Anal. 6 . 2 ( 2019 ). doi: 10 .18608/jla. 2019 . 62 .4.

[25] E. Panadero , A review of self-regulated learning: Six models and four directions for research , Front. Psychol . 8 ( 2017 ). doi: 10 .3389/fpsyg. 2017 . 00422 .

[26] C. Knoop-van Campen , I. Molenaar , How teachers integrate dashboards into their feedback practices , Frontline Learn. Res . ( 2020 ) 37 - 51 . doi: 10 .14786/flr.v8i4. 641 .

[27] R. S. J. D. Baker , A. B. Goldstein , N. T. Heffernan , Detecting learning moment-by- moment , Int. J. Artif. Intell. Educ . 21 .1- 2 ( 2011 ) 5 - 25 .

[28] B. M. Hanington , Universal methods of design: 100 ways to research complex problems, develop innovative ideas, and design effective solutions , Rockport Publishers, Beverly, Mass, 2012 .

[29] S. Davidoff , M. K. Lee , A. K. Dey , J. Zimmerman , Rapidly exploring application design through speed dating , in: UbiComp 2007 : Ubiquitous computing, Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 429 - 446 . doi: 10 .1007/978-3- 540 -74853-3_ 25 .

[30] M. Bannert , E. Pieger , S. Christoph, MESH-Ein verfahren zur erfassung metakognitiver strategien beim lernen mit hypermedien [metacognitive strategy inventory for learning with hypermedia], Dep . Educational Sciences, LS f. Lehren und Lernen mit Digitalen Medien , 2015 .

[31] J. Saldaña , Coding manual for qualitative researchers , SAGE Publications, Limited , 2012 .

[32] M. Skjott Linneberg , S. Korsgaard , Coding qualitative data: A synthesis guiding the novice , Qual. Res. J. 19.3 ( 2019 ) 259 - 270 . doi: 10 .1108/qrj-12-2018-0012.