Towards a Teacher-Oriented Framework of Visual Learning Analytics by Scenario-Based Development

Abstract

Visual Learning Analytics (VLA) tools (such as dashboards) serve as a centralized hub for monitoring and analyzing educational data. Dashboards can assist teachers in data-informed pedagogical decision-making and/or students in following their own learning progress. However, the design of VLA tools should include features of trust in order to make analytics overt among its users. In this study, we propose a framework for the development of VLA tools from beginning to end that describes how we intend to develop the digital and technical infrastructure in our project for teachers. With that aim, we ofer one scenario describing how data is managed, transferred, analyzed, and visualized by teachers. The suggested framework intends to make it easier for developers to understand the various steps involved in co-designing and developing a reliable VLA tool and to comprehend the importance of the teacher's participation in design. VLA tools developed based on the proposed framework have the potential to assist teachers in understanding and analyzing educational data, monitoring students' learning paths based on their learning outcomes and activities, simplifying regular tasks, and giving teachers more time to support teaching/learning and growth.

Keywords

• Development

-

1. Introduction ogy has allowed for the collection of a wide range of data on students. Educational data can be large, complex, and heterogeneous [ 1 ], coming from sources such as text, quizzes, timestamps, and behavioral data on the use of Digital Learning Material (DLM) or student data of an administrative nature from digital administrative systems. In this study, DLM refers to all digital materials that hold an educational purpose and are accessible through a computer or a tablet. Due to the fact that educational data are typically spread over a variety of diferent digital resources, it may be challenging for a informed decision-making.

Once the educational and administrative data (student data such as grades and attendance) is safely moved to a unified storage, matched, and anonymized, a process for extracting insights (such as identifying a student’s learning path, monitoring and predicting their performance, Proceedings of the Doctoral Consortium of the 18th European Conference on Technology Enhanced Learning (ECTEL), 4th September 2023, LGOBE

https://lnu.se/en/staff/zeynab.mohseni/ (Z. Mohseni); https://lnu.se/personal/italo.masiello/ (I. Masiello); https://lnu.se/personal/rafael.martins/ (R. M. Martins)

0000-0002-3297-0189 (Z. Mohseni); 0000-0002-3738-7945 (I. Masiello); 0000-0002-2901-935X (R. M. Martins) [ 4 ] (p. 1) as “the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs”. As a developing area, LA draws pedagogical patterns from data related to a learner’s interactions and activities to guide decisionmaking and assessments [ 5 ].

Visual Learning Analytics (VLA) tools visualize data (usually via dashboards) to help teachers better understand LA [ 1, 6, 7 ]. The definition of a VLA tool given gregates diferent indicators about learner(s), learning process(es), and/or learning context(s) into one or multiple visualizations”. By using VLA tools, one may create a visual depiction of what a teacher or student can do to enhance motivation, self-direction, learning efectiveness, student performance, and teacher engagement [ 9, 10, 11 ].

However, for the successful integration and efective utilization of analytics in supporting learning and teaching, the design of VLA tools should consider a spectrum of human factors, encompassing the reasons behind their usage and the methods by which they will be applied [ 12, 13 ]. In addition, the functionality of analytics challenges the accountability of students’ learning processes that are simply transformed into non-transparent processes for teachers [ 14 ], requiring therefore other design components of trust not addressed before. The research teacher to use them strategically for pedagogical data- by Schwenimann et al. [ 8 ] is “a single display that ag© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License question that drives our work is formulated as follows: Attribution 4.0 International (CC BY 4.0).

How can a VLA development framework be used to estab- documenting common and pivotal user activities early lish a digital and technical infrastructure for primary and on and consistently throughout the development process secondary school teachers? We chose to work with pri- which facilitate the analysis of usage situations, even mary and secondary school teachers because there is little before those situations come into existence [ 21, 22, 23 ]. research on VLA tools in this context [ 10 ]. Our proposed For example, Siddiqui et al. [ 24 ] developed a scenarioframework goes through a detailed scenario that includes: based e-learning solution to improve the efectiveness of 1) the diferent steps required in co-designing and devel- an introductory technology course. The scenarios’ main oping a reliable VLA tool; 2) the long-term engagement goal is to enable decision-makers to consider the consenecessary to build a solution based on the actual needs quences of diferent assumptions of how systems might and trade-ofs that design decisions can amount to; 3) respond to various causes of change [ 25 ]. For instance, the educational data collected from various resources Zurek & Henrichs [26] have claimed that scenarios can be required for creating a more comprehensive overall view used to: 1) help us structure the choices we need to make of the students’ learning and learning process; and 4) the by identifying any potential long-term consequences, 2) necessary building blocks required for the construction by giving people a place to consider the implications of of a technological solution that is transparent to the user. diferent options in the face of upcoming uncertainty, Developing a VLA tool based on the suggested frame- a platform for strategic planning and decision-making work that incorporates all data gathered from numerous is provided, 3) allowing stakeholders to express opposstakeholders and adheres to the same data structure has ing viewpoints and opinions will help to facilitate their the potential to streamline everyday activities and pro- participation in the strategic formulation process. vide teachers with more time to support students’ growth In this paper, we provided a scenario-based frameand learning [ 15 ]. work that describes how data is managed, transferred,

The rest of the article is organized as follows. Section analyzed, and visualized for teachers in primary and sec2 describes the background of the scenario-based models. ondary education. The purpose of the suggested user Section 3 describes the elements of the digital and tech- scenario is to demonstrate how teachers can be involved nical infrastructure. Section 4 presents the user scenario. in the co-design of a VLA tool by considering the infrasSection 5 concludes the paper. tructure, and relevant building blocks and bringing value to the educational system. Researchers may also use the entire infrastructure presented in the framework as a 2. Background guideline for the development of a reliable VLA tool. In the study by Buckingham Shum et al. [ 12 ], they presented principles of Human-Centered Design (HCD) from related fields that can be incorporated and customized to facilitate the advancement of human-centered learning analytics. Furthermore, Dimitriadis et al. [ 16 ] presented three HCD principles for LA solutions, including positioning teachers and other stakeholders as agents, incorporating the learning design cycle into the LA design process, and leveraging educational theories to inform the design and implementation of LA solutions. Within the larger context of HCD, scenario-based management and policy support technologies can serve as valuable assets. Scenarios play a ubiquitous role in the software development process and hold significance for any system operating in a complex environment, extending beyond systems solely reliant on human-computer interfaces [ 17, 18 ]. In requirements engineering, scenarios promote multidisciplinary learning, but they also serve as a mechanism for labor division, with significant consequences for project management and artifact integration [ 19 ]. According to Polasky et al. [ 20 ], scenarios are “sets of plausible stories, supported with data and simulations, about how the future might unfold from current conditions under alternative human choices”. Scenarios are straightforward approaches that involve actively envisioning and

3. The Digital and Technical Infrastructure

There are a few necessary building blocks that make up the technical infrastructure. The term “building block” refers to a group of “components” that can include technical capabilities, services, standardized models, frameworks, and patterns and that are intended to address common infrastructure problems in a given focus area [27]. We considered the following five building blocks for the scenario: 1) Authorization: this building block intends to assist the remaining four building blocks by enabling standardized digital functions for secure information exchange. The digital infrastructure and services needed to securely and digitally authorize individuals, groups, and smart objects are included in the authorization building block; 2) Traceability: this building block aims to provide a framework for traceability, add to a shared list of specifications, and look into and create operational capabilities and processes for traceability; 3) Trust framework: by establishing requirements and monitoring capabilities for a trust-promoting environment, the framework for “organizational trust” intends to make it easier for stakeholders to establish and share digital information in a secure manner over a long term, 4) API: there should be preconditions in digital processes for co-creation between the public sector, individuals, and the private sector in a modern digital ecosystem where a large number of stakeholders (producers and consumers) participate with services and information cast in components for information interchange. Application Programming Interfaces (APIs) are essential in facilitating this information sharing. The defined standards and guidelines are made available through APIs, a building block that also provides associated applications, programming interfaces, and visualizations. 5) Access control: where there is a recommendation to ensure access (a login system) for an authorized user at the right time. These five building blocks described in this section are employed in the development of our VLA tool. The building blocks proposal intends to provide users—in this case, teachers— with secure access to information and a way to trace where the information is coming from. These enable the Development Team (DT) to monitor user access and exercise some control over the information’s availability, but also show where the information comes from for teachers wanting to understand the “hidden” processes of the analyses.

The development phase of this project is carried out by considering a teacher scenario. Our scenario describes how data is managed, transferred, analyzed, and visualized for teachers. The purpose is to acknowledge the significance of the teacher’s role in design and demonstrate how the relevant building blocks form the development of our VLA tool.

4. Scenario: “Rose is a teacher who wants to make pedagogical decisions based on digital data”

This scenario demonstrates what connections are established between the steps the teacher performs, the necessary building blocks that make up the infrastructure, and the datasets that feed into the infrastructure. In this scenario, Rose, a teacher ( ), knows that diferent digital tools provide diferent information about her students.

However, those digital tools do not communicate with each other, as it is often the case with proprietary digital systems in primary and secondary education [28]. Rose is in a situation where she needs assistance in making pedagogical decisions and analyzing student data collected from the various digital resources that she uses in her teaching and learning activities. For properly evaluating the students’ performance, she needs a mediating VLA tool—a dashboard—helping her to visualize and interpret the collected data. The user, Rose’s activities in the scenario are shown in figure 1. This scenario includes and finally test diferent DLMs. dashboard for her that meets all her requirements 3. Starts the use of the selected DLM : Rose employs and encourages daily use.

the purchased DLM in the classroom and eval- 8. Makes a better decision about educational data: the uates it by considering the tasks and activities ifnal step in the development phase will give Rose the students may complete with it. Using the access to a teacher dashboard with visualizations DLM regularly allows the EdTech company to that will assist her in better understanding the collect trustworthy data on the online activities students’ educational data collected from various and learning behavior of her students. resources, evaluating their performance, deter4. Completes and submits feedback about the DLM : mining the best learning path, making better desince no DLM is perfect, Rose takes this step to cisions, and assisting the students in improving provide feedback to the EdTech company every their performance. month addressing errors and bugs in the DLM or provide suggestions for how to improve the This scenario-based development helps to better comlearning material. prehend the various steps involved in creating a reliable and trustworthy VLA tool and recognize the importance 5. pTohseadlse:vewlohpemnednetvteealompicnogmaesnuepwwdiitghitsaollut otiooln, iptrios- of the teacher’s role in design as the primary user. necessary to identify requirements [27, 29]. The Developing a VLA tool by considering the proposed DT identifies and analyzes the needs that Rose framework has the potential to streamline daily activihas and captures the requirements that should be ties and ofer teachers more time to support students’ featured in the tool for it to be safe and secure. growth and learning [ 15 ]. The framework considers To achieve this, the team first conducts Design- several datasets obtained from multiple sources such as Based Research (DBR) [30] with Rose using pa- schools’ digital systems and EdTech companies and adper and clickable prototypes [29] then interviews heres to the same data structure. During the development Rose to better understand her needs and the trade- of our dashboard, we have followed a scenario to help ofs that the team must apply for a viable design. us detect all possible structures needed for the VLA tool. This step needs to be repeated multiple times be- We are currently testing the dashboard in various schools fore the team is ready for the next step. After and with many teachers, meaning we are on step 4, but reviewing the data gathered from the interviews we do not yet have preliminary results. and the DBR, the team determines what data are required in the service. The DT identifies sev- 5. Conclusion eral sources of data that can be used to provide decision-making support to Rose, i.e., data from Today, many educational systems allow access to a large the DLM, as well as data collected in the adminis- amount of information about students and their educatrative digital systems such as attendance, weekly tional activities. However, it might be dificult to use academic schedule, learning objectives, etc. The these datasets strategically because they are usually disteam also considers how existing infrastructure persed across multiple distinct digital services. As a building blocks might be linked to be used in con- consequence, teachers fail to take advantage of the opjunction with tool development. portunity to make decisions based on evidence. In this 6. Using the proposed VLA tool to evaluate the stu- study, we provided a VLA development framework that dents’ performance: The DT analyses the difer- illustrates how we expect to develop the digital and techent datasets with machine learning techniques nical infrastructure in our project. In order to achieve to discover more informative students’ patterns this and answer our research question, we provided a in specific contexts. This step is done in strict scenario for teachers that illustrates the use of data mancollaboration with Rose, so that Rose can clearly agement, transfer, analysis, and visualization. The estabunderstand the analysis performed and trust the lishment of a VLA tool based on the suggested framedeveloped VLA tool. Rose evaluates the visu- work, which incorporates the information gathered from alizations of the relevant data. The data con- diferent datasets, has the potential to support teachers tains students’ data from the EdTech company, in understanding and analyzing educational data, moniDBR and interview data, and data regarding the toring students’ learning paths based on their learning students’ progress and final grade gathered by outcomes and activities, streamlining teachers’ routine Rose/school/municipality. tasks, and finally giving teachers more time to support 7. Regular meetings to discuss the needs: the DT teaching/learning and development. We are currently meets with Rose to go over her requirements. Re- engaged in a research project in collaboration with teachpeating this process a few times will allow the ers, school principals and EdTech companies with which developer to create an eficient and trustworthy we are going to utilize the proposed framework to craft a solution that aligns with actual needs of teachers and considers design trade-ofs. [26] M. B. Zurek, T. Henrichs, Linking scenarios across geographical scales in international environmental assessments, Technological forecasting and social change, Elsevier 74 (2007) 1282–1295. [27] DIGG, Agency for digital government (digg): Värdeskapande byggblock, 2023. URL: https://www.digg.se/ledning-och-samordning/ ena---sveriges-digitala-infrastruktur/byggblock. [28] C. Aguerrebere, H. He, M. Kwet, M.-J. Laakso,

C. Lang, C. M. D. Price-Dennis, H. Zhang, Global perspectives on learning analytics in k12 education, by Charles Lang, Alyssa Friend Wise, Agathe Merceron, Dragan Gaševic, and George Siemens. 2nd ed. Vancouver, Canada: SOLAR (2022). [29] J. Moonen, Prototyping as a design method, Tj.

lomp & DP Ely (Eds.), International encyclopedia of educational technology (2nd ed.) (1996) 186–190. [30] S. Barab, Design-Based Research: A Methodological

Toolkit for the Learning Scientist., 2006.

References

[1] Z. Mohseni , R. M. Martins , I. Masiello , Sbgtool v2 . 0: An empirical study on a similarity-based grouping tool for students' learning outcomes , Data 7 ( 2022 ) 98 .

[2] M. Aruvee , A. Ljalikova , E. Vahter , L. P. Prieto , K. Poom-Valickis , Learning analytics to inform and guide teachers as designers of educational interventions , in: EDULEARN18 Proceedings , IATED, 2018 , pp. 2457 - 2465 .

[3] D. Clow , An overview of learning analytics , Teaching in Higher Education 18 ( 2013 ) 683 - 695 .

[4] G. Siemens , R. S. d. Baker, Learning analytics and educational data mining: towards communication and collaboration , in: Proceedings of the 2nd international conference on learning analytics and knowledge , 2012 , pp. 252 - 254 .

[5] K. E. Arnold , M. D. Pistilli , Course signals at purdue: Using learning analytics to increase student success , in: Proceedings of the 2nd international conference on learning analytics and knowledge , 2012 , pp. 267 - 270 .

[6] Z. Mohseni , R. M. Martins , I. Masiello , Sbgtool: Similarity-based grouping tool for students' learning outcomes , in: 2021 Swedish Workshop on Data Science (SweDS) , IEEE, 2021 , pp. 1 - 7 .

[7] Z. Mohseni , R. M. Martins , I. Masiello , Savis: a learning analytics dashboard with interactive visualization and machine learning , in: Nordic Learning Analytics (Summer) Institute 2021 , KTH Royal Institute of Technology, Stockholm, 23 August 2021 , volume 2985 , ceur-ws.org, 2021 .

[8] B. A. Schwendimann , M. J. Rodriguez-Triana , A. Vozniuk , L. P. Prieto , M. S. Boroujeni , A. Holzer , D. Gillet , P. Dillenbourg , Perceiving learning at a glance: A systematic literature review of learning dashboard research , IEEE Transactions on Learning Technologies 10 ( 2016 ) 30 - 41 .

[9] K. Verbert , X. Ochoa , R. De Croon, R. A. Dourado , T. De Laet, Learning analytics dashboards: the past, the present and the future , in: Proceedings of the tenth international conference on learning analytics & knowledge , 2020 , pp. 35 - 40 .

[10] Z. Mohseni , R. M. Martins , S. Nordmark , I. Masiello , Visual learning analytics for educational interventions in primary, lower and upper secondary schools: A scoping review, 2023, Under review .

[11] H. He , B. Dong , Q. Zheng , D. Di , Y. Lin , Visual analysis of the time management of learning multiple courses in online learning environment , in: 2019 IEEE Visualization Conference (VIS) , IEEE, 2019 , pp. 56 - 60 .

[12] S. Buckingham Shum , R. Ferguson , R. MartinezMaldonado, Human-centred learning analytics , Journal of Learning Analytics 6 ( 2019 ) 1 - 9 .

[13] T. Ley , K. Tammets , G. Pishtari, P. Chejara , R. Kasepalu , M. Khalil , M. Saar , I. Tuvi, T. Väljataga , B. Wasson , Towards a partnership of teachers and intelligent learning technology: A systematic literature review of model-based learning analytics , Journal of Computer Assisted Learning ( 2023 ).

[14] M. Utterberg Modén , Teaching with Digital Mathematics Textbooks-Activity Theoretical Studies of Data-Driven Technology in Classroom Practices , 2021 .

[15] J. Hylén , J. Karlén , Standarder för datadrivna processer i skolan , 2019 .

[16] Y. Dimitriadis , R. Martínez-Maldonado , K. Wiley, Human-centered design principles for actionable learning analytics, Research on E-learning and ICT in education: Technological, pedagogical and instructional perspectives ( 2021 ) 277 - 296 .

[17] W. L. Johnson , Specification via scenarios and views ., in: ISPW , 1986 , pp. 61 - 63 .

[18] K. M. Benner , M. S. Feather , W. L. Johnson , L. A. Zorman , Utilizing scenarios in the software development process, in: Information system development process , Elsevier , 1993 , pp. 117 - 134 .

[19] K. Weidenhaupt , K. Pohl , M. Jarke , P. Haumer , Scenarios in system development: current practice , IEEE software 15 ( 1998 ) 34 - 45 .

[20] S. Polasky , S. R. Carpenter , C. Folke , B. Keeler , Decision-making under great uncertainty: environmental management in an era of global change , Trends in ecology & evolution, Elsevier 26 ( 2011 ) 398 - 404 .

[21] J. M. Carrol , Five reasons for scenario-based design , in: Proceedings of the 32nd annual hawaii international conference on systems sciences. 1999 . hicss -32. abstracts and cd-rom of full papers , IEEE, 1999 , pp. 11 -pp.

[22] J. Sun , Q. V. Liao , M. Muller , M. Agarwal , S. Houde , K. Talamadupula , J. D. Weisz , Investigating explainability of generative ai for code through scenariobased design , in: 27th International Conference on Intelligent User Interfaces , 2022 , pp. 212 - 228 .

[23] M. B. Rosson , J. M. Carroll , Scenario based design, Human-computer interaction . boca raton, FL ( 2009 ) 145 - 162 .

[24] A. Siddiqui , M. Khan , S. Akhtar , Supply chain simulator: A scenario-based educational tool to enhance student learning , Computers & Education 51 ( 2008 ) 252 - 261 .

[25] N. Ash , Ecosystems and human well-being: a manual for assessment practitioners , Island Press, 2010 .