Conceptual framework for process-oriented feedback through Learning Analytics Dashboards

Abstract

The number of students enrolled in online higher education courses is increasing, and as a result, more data on their learning process is being generated. By exploring this student behavior data through learning analytics, both student and teacher can be provided with process-oriented feedback in the form of dashboards. However, little is known about the typology of relevant feedback in the dashboard to different learning objectives, students and teachers. Although most dashboards and the feedback they provide are based solely on student performance indicators, research shows that such feedback is not sufficient. This article attempts to define a conceptual model that visualizes the relationships between the design of a Learning Analytics Dashboard (LAD) and the concepts of learning science in order to provide process-oriented feedback that supports the regulation of learning. The aim of the work is not to propose a specific design of the LAD to provide feedback, but rather a conceptual framework for the choice of concepts for that design, and therefore to help understand future data needs as a basis for the educational feedback of the dashboards. As a conclusion of our research, we can say that having LADs adapted to any profile (student, teacher, etc.) can improve decision-making processes by showing each user the information that interests them most in the way that best enables them to understand it.

Keywords

• learning analytics dashboards
• process-oriented feedback
• learning sciences

-

The number of students in online courses has increased in the last decade [ 1 ]. Therefore, the data generated in their learning process within the online learning spaces are also growing [ 2 ]. Learning Analytics (LA) emerges with the goal of using data on learner activity in Learning Management Systems (LMS) to increase understanding of the learning experience and better support learners [ 3 ].

The rapid advancement of educational technologies and online courses has generated greater interest in exploring data on student behavior to provide learning process1 Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). oriented feedback mechanisms [ 4 ]. Examining how students interact within LMSs (i.e. with each other, with teachers, with the environment...) provides opportunities to reveal where things are progressing well and where problems may arise. Using this information, process-oriented feedback can be generated that can help teachers and students improve engagement and achievement [ 5 ]. This feedback can be presented in the form of visualizations on various teacher- and student-oriented dashboards [ 6, 7, 8 ].

Dashboards are seen as tools that aim to improve decision-making by directing cognition and capitalizing on human perceptual capacities [ 9 ]. However, despite the popularity of dashboards, little is known about their effectiveness, for example, the typology of feedback needed for different learning objectives, different students and a teacher [ 10 ].

According to [ 11 ], most research on educational dashboards lacks both the theoretical support of recent advances in the learning sciences and an evidence base for choosing data that can help observe and evaluate learning processes to identify the feedback needs of students and/or teachers. As a result, the information provided by the dashboards regarding the learning process, instead of being useful, may be non-existent and even negative [ 12 ]. Furthermore, current dashboard solutions are mainly based on student performance indicators, which do not seem to contribute to student motivation and engagement [ 13, 14 ]. Recent research reveals that when performance-oriented dashboards are used, the orientation of student mastery decreases [ 15 ]. This suggests that such goal orientations must be carefully considered in the design of any intervention, as the resulting instruments may affect students' interpretations of their data and consequent academic success [ 15 ].

These goals may be mastery or performance oriented. While students with mastery goals are usually interested in learning as an end in itself, students with performanceoriented goals are usually interested in learning as a means of demonstrating their ability or competence in the subject [ 16 ]. In this regard, it is important to note the importance of these guidelines in enabling students to define their learning objectives.

All of the LADs for providing feedback that exist in the literature have in common the lack of theoretical support based on the learning sciences and research on feedback and the mechanisms underlying learning processes [ 17 ].

This article attempts to define a conceptual model that visualizes the relationships between the design of an LAD and the concepts of learning science in order to provide process-oriented feedback that supports the regulation of learning. The aim of the work is not to propose a specific design of the LAD to provide feedback, but rather a conceptual framework for the choice of concepts for that design, and therefore to help understand future data needs as a basis for the educational feedback of the dashboards.

In section 2, a brief analysis has been made of the different approaches that exist in the current literature on LADs, and of the importance of personalizing them when designing them and defining the visualizations that are made of them. Afterwards, the concept of process-oriented feedback has been introduced as a differentiating and very important element in the design of an LAD for the student and specifically for the student's self-regulation process. To finish with the section of conclusions and future research to be carried out as a continuation of this research work.

Learning Analytics Dashboards (LAD)

In recent years, many LADs have been implemented to facilitate the understanding of student learning data. The objectives of these dashboards should include providing feedback on learning activities, encouraging reflection and decision-making, increasing engagement and motivation, and supporting learning regulation [ 4, 6, 7, 8, 18, 19, 21 ]. These LADs apply information visualization techniques to help teachers and students explore and understand relevant user traces collected in various LMSs. The overall goal is to enhance the learning process [ 22 ].

As for the objectives observed in the dashboards, in most of the studies carried out so far, these are limited to the results of student performance [ 6 ], positioning students in comparison with the performance specified by the teacher and/or peers. As noted above, this data is collected through records of the LMSs used by students [ 23 ].

Representations of such results are generally limited to graphs, tables or other diagrams without providing supporting mechanisms to facilitate interpretation [ 6, 24 ]. On the other hand, several studies show that a change in behavior and an improvement in performance were observed when the student was supported in the interpretation of the visualizations [ 4, 25, 26 ].

We cannot claim to have a single approach for all types of users [ 27 ]. In the educational context, LADs not only seek to inform teachers about student performance, but can also become tools to motivate students [ 28 ]. They can even serve as tools for students to self-regulate and compare their own results. However, not all students may respond in the same way to the information shown in an LAD about their performance [ 27, 29 ].

LADs should be personalized to provide the most effective information needed. In fact, a study by [ 30 ] confirmed the widespread desire of students for LADs that can be personalized to their liking, giving them the option of configuring them to display the information they are most interested in or see as most useful [ 31, 32, 33, 34 ].

Finally, and as far as LADs are concerned, another question that arises is that of evaluating the instrument or tool in a constant manner. This validation could review whether the instruments are fulfilling their intended purpose, whether they are actually having a positive effect on learning, or whether they are helping more efficient and/or effective learning [ 35 ]. The evaluation of information visualization systems is essential.

Thus, common to all of these feedback LADs is the lack of theoretical support grounded in the learning sciences [ 17 ]. Therefore, we see the need to analyze what concepts are needed to design a LAD that provides feedback so that it is possible to observe the learning processes with regard to possible feedback needs (e.g. learning regulation) of different students for different learning objectives. 3

Process-oriented feedback

Learning regulation and performance regulation is central to research on feedback [ 36 ]. Learning regulation is defined as an intentional and goal-directed metacognitive activity in which students take strategic control of their actions (behavior), thinking (cognitive), and beliefs (motivation, emotions) to complete a task [ 37 ].

In practice, self-regulated learning represents experimenting and learning about effective strategies for regulating one's own aspects such as planning, goal setting, organization, monitoring and adaptation [ 38 ].

In summary, following regulation guidelines during a learning process can be useful to determine possible feedback needs during a learning process [ 39 ].

As already indicated in the introduction to this research paper, research on dashboards lacks theoretical support from recent developments in the field of learning sciences and feedback research [ 23 ]. Furthermore, current LADs are mainly based on student performance indicators, leading to a lesser orientation of the domain [ 15 ]. The conceptual model to be defined in this paper aims to address this gap and therefore the concept of learning process-oriented feedback needs to be further analyzed [ 10 ].

Feedback can be defined as an interactive process in which the result or effect of an action is returned ("feedback") to modify the next action towards achieving a goal. In order to link students' past and future work and help them create a path of progressive development, "timeliness" must be central to any action or discussion about feedback [ 40 ].

Research on feedback shows that the earlier students receive information about what they have done, the more effective it is for their learning [ 41 ]. When we raise the idea that LADs can provide feedback on learning regulation, we intend to inform students about the needs for regulation during the learning process, and more specifically about the phases of planning (definition of objectives), monitoring and adaptation (readaptation of objectives in real time or during the itself process) [ 39 ].

As regards the typology of feedback, the different approaches described in the research analyzed translate into two main forms: explanations aimed at improving the cognitive dimensions of knowledge and orientations to influence student behavior [ 4 ].

Cognitive feedback provides information to students about the success or failure of a particular task through pointers, comments and/or questions, which help students reflect on the quality of work done on a particular task [ 42 ].

Unlike cognitive feedback, behavioral feedback aims at changing behavior. This type of feedback relates to the student's learning objectives and goals, improving awareness of learning progress and potential regulation needs during the learning process [ 40 ].

As we have seen in the section on regulation of learning, planning is the first phase within the regulation process, and the setting of objectives is a very important part of that phase [ 43 ]. Depending on the types of objectives set by the teacher or planned by the students, learning outcomes will be directed at different levels of knowledge (competences), or simply at the completion of tasks [ 38 ]. When teachers set explicit learning goals, students have a clear idea of course expectations and focus efficiently on achieving those goals [ 44 ]. However, if students set or plan their own learning goals, it can improve learning and students' own motivation [ 45 ]. It is in this case, where students have clear objectives, that they are most likely to seek or need feedback to close the gap between their knowledge or skills and the desired goal [ 46 ].

Knowing the learning objectives and how much effort (regulatory) the learner has put into achieving the objectives is not enough to determine the possible time when the feedback will be most relevant to an end user [ 47 ]. LADs should on the one hand enable the student to monitor his/her learning progress, and on the other hand assist in the objectives planned by the student and/or teacher [ 48 ]. 4

Conclusions and Further Research

This research paper discusses the design implications for an LAD that can provide feedback and preliminary answers on how such feedback can be based on the learning sciences. The research contributes to the learning sciences with respect to the lack of methodologies for designing and building LADs, the lack of experience in processoriented early feedback or learning goals, and data and information sciences with respect to the type of data concepts needed to store and track learning processes in relation to feedback.

In summary, from the perspective of the learning sciences, the learning process can be positively influenced by the feedback provided by LADs if the regulatory mechanisms underlying the learning processes are taken into account [ 46 ]. This feedback can be built on the basis of the phases of the learning regulation process that encompass planning, monitoring and adaptation activities, which allows for the detection of inefficient learning processes and/or objectives. Furthermore, LMSs should consider student learning objectives to broaden the scope of LAD feedback to support mastery orientation, in addition to performance orientation, which is the main goal of existing solutions. By complementing feedback with the concepts of effectiveness and efficiency of learning processes, it is also possible to track learning progress and refine detection mechanisms for potential intervention time by allowing for the detection of ineffective or inefficient processes during learning [ 4 ]. However, detailed mechanisms for user intervention in feedback remain a challenge.

The conceptual design proposed in the paper will make it possible to provide students with personalized process-oriented feedback through LAD, as opposed to the traditional outcome and performance-oriented feedback of the student, which usually occurs during learning after a learning task has been completed, indicating whether the results are correct or not [ 36 ].

Another thing to keep in mind is that IAD feedback should be bi-directional, allowing the student to observe and improve his or her learning with respect to his or her own need for regulation (self-regulation), on the one hand, and allowing the teacher to observe the individual needs of the students to obtain specific feedback, on the other hand [ 49 ].

The future research direction thus includes these challenges: a)

Explore mechanisms of analysis that take into account the personal characteristics of students, different personality patterns and the emotional experience lived by the student during the learning process, when designing the corresponding feedback model within the LAD.

Analyze the possibilities of feedback by integrating the socio-emotional context of learning based on multimodal data that can be collected, for example, from wearable sensors, audio/video flow analysis, etc.

To review other research, with the aim of obtaining new techniques of data visualization and analysis, through which one can work better in supporting the feedback ideas presented in this paper.

References

1. Arriaran , I. , Ipiña , N.: Indicadores necesarios para diseñar un dashboard desde la perspectiva de los profesores: un estudio cualitativo . Revista Iberoamericana de Educación 80 ( 1 ), 157 - 166 ( 2019 ).

2. Means , B. : What research tells us about whether, when and how . In: Learning Online. New York: Routledge ( 2104 ).

3. Dawson , S. , Gasevic , D. , Siemens , G. , Joksimovic , S.: Current state and future trends: a citation network analysis of the learning analytics field . In: International Conference on Learning Analytics and Knowledge . Indianapolis ( 2014 ).

4. Sedrakyan , G.: Process-oriented feedback perspectives based on feedback-enabled simulation and learning process data analytics . PhD thesis . KU Leuven ( 2016 ).

5. Gasevic , D. , Dawson , S. , Rogers , T. : Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success . The Internet and Higher Education 28 , 68 - 84 ( 2016 ).

6. Bodily , R. , Verbert , K. : Review of research on student-facing learning analytics dashboards and educational recommender systems . IEEE Transactions on Learning Technologies 10 ( 4 ), 405 - 418 ( 2017 ).

7. Hu , Y.-H. , Lo , C.-L. , Shih , S.-P. : Developing early warning systems to predict students' online learning performance . Computers in Human Behavior 36 , 469 - 478 ( 2014 ).

8. Mottus , A. , Graf , S. , Chen , N.-S.: Use of dashboards and visualization techniques to support teacher decision making . In: Ubiquitous Learning Environments and Technologies , pp. 181 - 199 . Springer Berlin Heidelberg ( 2015 ).

9. Yigitbasioglu , O. M. , Velcu , O.: A review of dashboards in performance management: Implications for design and research . Journal of Accounting Information Systems 13 ( 1 ), 41 - 59 ( 2012 ).

10. Sedrakyan , G. , Järvelä , S. , Kirschner , P. : Conceptual framework for feedback automation and personalization for designing learning analytics dashboards . In: Conference EARLI SIG 27 , online measures of learning processes ( 2016 ).

11. Saywer , R. K. : The future of learning: Grounding educational innovation in the learning science . In: The cambridge handbook of the learning sciences , pp. 1 - 19 . Cambridge, UK: Cambridge University Press ( 2014 ).

12. Corrin , L., de Barba , P.: How do students interpret feedback delivered via dashboard? In: International Conference on Learning Analytics and Knowledge . Poughkeepsie, NY ( 2015 ).

13. Blumenfeld , P. C. : Classroom learning and motivation: Clarifying and expanding goal theory . Journal of Educational Psychology 84 , 272 - 281 ( 1992 ).

14. Elliot , A. , Harackiewicz , J.: Approach and avoidance achievement goals and intrinsic motivation . Journal of Personality and Social Psychology 73 , 171 - 185 ( 1996 ).

15. Lonn , S. , Aguilar , S. J. , Teasley , S. D.: Investigating student motivation in the context of a learning analytics intervention during a summer bridge program . Computers in Human Behavior 47 , 90 - 97 ( 2015 ).

16. Dweck , C. S. , Leggett , E. L.: A social-cognitive approach to motivation and personality . Psychological Review 95 ( 2 ), 256 - 273 ( 1998 ).

17. Sedrakyan , G. , Malmberg , J. , Noroozi , O. , Verbert , K. , Järvelä , S. , Kirschner , P. : Designing a learning analytics dashboards for feedback to support learning regulation ( 2017 )

18. Duval , E. , Klerkx , J. , Verbert , K. , Nagel , T. , Govaerts , S. , Parra Chico, G. A. , et al.: Learning dashboards & learnscapes . In: Educational interfaces, software, and technology , pp. 1 - 5 ( 2012 ).

19. Dyckhoff , A. L. , Zielke , D. , Bültmann , M. , Chatti , M. A. , Schroeder , U. : Design and implementation of a learning analytics toolkit for teachers . Educational Technology & Society 15 ( 3 ), 58 - 76 ( 2012 ).

20. Verbert , K. , Duval , E. , Klerkx , J. , Govaerts , S. , Santos , J. L. : Learning analytics dashboard applications . American Behavioral Scientist ( 2013 ).

21. Verbert , K. , Govaerts , S. , Duval , E. , Santos , J., Van Assche , F. , Parra , G. , et al.: Learning dashboards: An overview and future research opportunities . Personal and Ubiquitous Computing 18 ( 6 ), 1499 - 1514 ( 2014 ).

22. Pardo , A. , Jovanovic , J. , Dawson , S. , Gasevic , D. , Mirriahi , N.: Using learning analytics to scale the provision of personalised feedback . British Journal of Educational Technology ( 2017 ).

23. Schwendimann , B. A. , Rodriguez Triana , M. J. , Prieto Santos , L. P. , Shirvani Boroujeni , M. , Holzer , A. C. , Gillet , D. , et al.: Understanding learning at a glance: An overview of learning dashboard studies . In: Proceedings of the sixth international conference on learning analytics & knowledge , pp. 532 - 533 . ACM ( 2016 ).

24. Park , Y. , Jo , I.-H.: Development of the learning analytics dashboard to support students' learning performance . Journal of Universal Computer Science 21 , 110 - 133 ( 2015 ).

25. Sedrakyan , G. , & Snoeck , M. : Design and evaluation of feedback automation architecture: Enriching model execution with feedback to support testing of semantic conformance between models and requirements . In: Modelsward 2016 - 4th international conference on model-driven engineering and software development ( 2016 ).

26. Sedrakyan , G. , Poelmans , S. , Snoeck , M. : Assessing the influence of feedback-inclusive rapid prototyping on understanding the semantics of parallel UML statecharts by novice modellers . Information and Software Technology 82 , 159 - 172 ( 2017 ).

27. Tesley , S. D.: Student facing dashboards: One size fits all? Technology, Knowledge and Learning 22(3 ), 377 - 384 ( 2017 ).

28. Vázquez-Ingelmo , A. , Therón , R.: Beneficios de la aplicación del paradigma de líneas de productos software para generar dashboards en contextos educativos . RIED. Revista Iberoamericana de Educación a Distancia 23 ( 2 ), 169 - 185 ( 2020 ).

29. Vázquez-Ingelmo , A. , García-Peñalvo , F. J. , Therón , R. , Conde , M. A. : Representing data visualization goals and tasks through meta-modeling to tailor information dashboards . Appl. Sci . 10 , 2306 ( 2020 ).

30. Roberts , L. D. , Howell , J. A. , Seaman , K. Give me a customizable dashboard: Personalized learning analytics dashboards in higher education . Technol. Knowl. Learn . 22 , 317 - 333 ( 2017 ).

31. Mutlu , B. , Veas , E. , Trattner , C. : Vizrec: Recommending personalized visualizations . ACM Trans. Interact. Intell. Syst . 6 , 31 ( 2016 ).

32. Saket , B. , Endert , A. , Demiralp , C. : Task-based effectiveness of basic visualizations . IEEE Trans. Vis. Comput. Graph . 25 , 2505 - 2512 ( 2018 ).

33. Vartak , M. , Huang , S. , Siddiqui , T. , Madden , S. , Parameswaran , A. : Towards visualization recommendation systems . ACM Sigmod Rec . 45 , 34 - 39 ( 2017 ).

34. Vázquez-Ingelmo , A. , García-Peñalvo , F.J. , Therón R. , Conde , M. A. : Extending a dashboard meta-model to account for users' characteristics and goals for enhancing personalization . LASI Spain 2019 : Learning Analytics in Higher Education (Vigo , Spain, 2019 ) 35 - 42 ( 2019 ).

35. Ferguson , R. , Brasher , A. , Clow , D. , Cooper , A. , Hillaire , G. , Mittelmeier , J. , et al.: Research evidence on the use of learning Analytics: Implications for education policy . In: Joint Research Centre, Seville, Spain ( 2016 ).

36. Butler , D. L. , Winne , P. H. : Feedback and self-regulated learning: A theoretical synthesis . Review of Educational Research 65 ( 3 ), 245 - 281 ( 1995 ).

37. Zimmerman , B. J. , Schunk , D. : Motivational sources and outcomes of self regulated learning and performance . In: B. Zimmerman , & D. Schunk (Eds.), Handbook of self-regulation of learning and performance , pp. 49 - 64 . New York, Routledge ( 2011 ).

38. Winne , P. H. , Hadwin , A. F. , Perry , N. E. : Metacognition and computer supported collaborative learning . In: C. Hmelo-Silver , A. O 'Donnell , C. Chan , & C. Chinn (Eds.), International handbook of collaborative learning , pp. 462 - 479 . New York: Taylor & Francis ( 2013 ).

39. Sedrakyan , G. , Malmberg , J. , Verbert , K. , Järvelä , S. , & Kirschner , P. : Linking learning behavior learning analytics and learning science concepts: Designing a learning analytics dashboard for feedback to support learning regulation . In: Computers in Human Behavior ( 2018 )

40. Eyers , D. , Jordan , J. , Hendry , K. : What are student perceptions of the timeliness of feedback ? In: Cardiff University ( 2016 ).

41. Irons , A. : Enhancing learning through formative assessment and feedback . In: Abingdon, UK: Routledge ( 2008 ).

42. van Merriënboer, J. J. G. , Kirschner , P. A. : Ten steps to complex learning: A systematic approach to four-component instructional design . Mahwah, N.J: Lawrence Erlbaum Associates ( 2012 ).

43. Stevens , C. K. , Gist , M. E.: Effects of self-efficacy and goal-orientation training on negotiation skill maintenance: What are the mechanisms? Personnel Psychology 50 ( 4 ), 955 - 978 ( 1997 ).

44. Turkay , S. : Setting goals: Who, why, how? In: Harvard University ( 2014 ).

45. Zimmerman , B. J. : Self-regulated learning and academic achievement: An overview . Educational Psychologist 2 , 3 - 17 ( 1990 ).

46. Hattie , J. , Timperley , H.: The power of feedback . Review of Educational Research 77 ( 1 ), 81 - 112 ( 2007 ).

47. Frøkjaer , E. , Hertzum , M. , Hornbaek , K. : Measuring usability: Are effectiveness, efficiency, and satisfaction really correlated? . In: Proceedings of the sigchi conference on human factors in computing systems , pp. 345 - 352 . ACM ( 2000 ).

48. Winne , P. H. , Hadwin , A. F. : Studying as self-regulated learning . Metacognition in educational theory and practice 93 , 27 - 30 ( 1998 ).

49. Lockyer , L. , Heathcote , E. , Dawson , S. : Informing pedagogical action: Aligning learning analytics with learning design . American Behavioral Scientist 57 ( 10 ), 1439 - 1459 ( 2013 ).