Workshop on human-centred learning analytics: A critical analysis based on the discussion of two case studies

Abstract

There is a recently growing subcommunity of LA focusing on human-centeredness that proposes a more user-centred approach to the learning analytics practice: Human-Centred Learning Analytics (HCLA). This approach focuses on three key elements that come into play and interact under the human activity umbrella: human factors, social factors and technology factors. Therefore, the main focus shifts from assuring that the artifact works as intended to enabling many individual or cultural conceptions to unfold into uninterrupted interfaces with technology. The workshop was organized by SNOLA (Spanish Network of Learning Analytics) and introduced two cases used to prompt the discussion of two LA tools designed following the HCLA approach. This paper presents the workshop and its results based on the discussion around the advantages and disadvantages of the two cases and their methodologies, objectives and findings according to participants' insights and collaboration. The two main findings this paper contributes are focused on: 1. The HCLA approach, viewed and understood from the point of view of non-experts and experts of different levels of expertise and 2. The implications that HCLA has on the design process of the LA tools and artifacts. These two findings open more doors for research on the enhancement of the HCLA approach and effectiveness of the design process, therefore guiding the future directions of research on the same line of the presented findings.

-

1

Introduction

Learning Analytics (LA) is defined as the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs [ 1 ], with this work mainly focusing on the needs of two major stakeholders: teachers and learners. The foundation of LA sits at the convergence of three main principles: Theory, Data Science and Design. Thus, only when all three principles are considered and applied do the results reach a high level of validity and effectiveness [ 2 ].

The most popular uses of LA research focus on student performance, prediction of academic success, identification of students who are at risk of failing or dropping out of their studies, among others [ 3 ]. Yet, as the LA approach evolved over the last decade, more uses were put in the foreground, making LA an even more powerful and solid approach that is able to help with complex learning and teaching requirements. Supporting the development of the student’s lifelong learning, personalizing learning plans, enhancing learning skills such as collaboration and critical thinking, and supporting quality learning and teaching by providing empirical evidence are some of the actual goals LA has, just to name a few [ 3 ]. As LA continues growing and getting more complex, especially from the perspective of considering user needs, the necessity to have a more human centered approach was recently born, giving way to the Human Centered subcommunity of LA: The Human-Centred Learning Analytics (HCLA) [ 4, 5 ].

Human-Centeredness (HC) has been identified in other fields as a characteristic of systems that have been carefully designed by determining three main points: critical stakeholders (i.e., human factors), their relationships (i.e., social factors) and the context they interrelate in (i.e., technology factors) [ 5 ]. HCLA assumes this perspective and integrates concepts and practices like participatory design and co-design in the design process, giving a critical and influential role to the user in the design and creation of the LA tools, switching the design logic from a “design for users” to a “design with users” [ 6 ].

The integration of the HC and design approaches with the user as a critical role stems from fundamental early findings in HCLA research [ 4, 5 ] and LAK21 first workshop on HCLA: First International Workshop on Human-Centred Learning Analytics (HCLA) [ 7 ]. As HCLA is a topic still under discussion, it is important to apply a Human Centered approach to collect feedback from the LA community to understand the benefits and the pain points of the HCLA methodology. For this reason, in this paper we present the results of a workshop organized in the context of the LASI Spain 2021 conference [ 8 ]. It is important to mention that this workshop is an adaptation of the first HCLA workshop organized in LAK’21 [ 7 ] mentioned above.

The organization of this workshop was conducted by members of SNOLA (Spanish Network of Learning Analytics) [ 9 ], in response to one of the objectives of the network, which aims at reaching a better understanding of users’ needs with respect to LA with special emphasis on the TEL (Technology Enhanced Learning) Spanish context. As part of this objective, this workshop was organized building on the series of HCLA workshops offered in international conferences (LAK’21 and ECTEL’21 HCLA workshops [ 7, 10 ] and in the global Learning Analytics Summer Institute (LASI) [ 8 ] of the global LA community (SOLAR) [ 11 ], to discuss with the LA community of practitioners in Spain the advantages and disadvantages of adopting an HCLA approach. In particular, this workshop provides a more specific view on the critical discussion of case studies’ tool design processes co-authored by members of the SNOLA network. The findings of the first LAK’21 workshop on HCLA served as a starting point for further discussion on HCLA among the community of practitioners in the field.

Some early findings from HCLA research (which should be understood better through the community expertise in both LA and education) orbit around (but not limited to): 1. The complexity of learning: The process of learning is a complex one. Moreover, the needs that teachers and students have are not always strictly data related. 2. Data literacy: Most of the actual LA tools require a level of data-literacy in order to be able to perform with it, which is perceived as a critical obstacle. Furthermore, from a Human-Centered Design (HCD) perspective it is advisable to adapt the tool to the teachers’ and students’ needs rather than contrariwise. 3. LA design choices from non-data experts: Non-data experts are unlikely to be aware of the implications of LA design choices. For instance, prior studies have shown that students find it hard to interpret charts and data visualizations [ 12 ]. Hence why teachers and students should be considered non-data experts. 4. Involving stakeholders: Just as important as it is involving the social and technology factors, the human factor also constitutes a critical element that interacts with the aforementioned two. Therefore, even though involving stakeholders may seem to be a difficult, time-consuming and expensive task to carry out during the different stages of the design process, it can easily turn out to be a key element for the correct functioning, success, agency and adoption of the resulting tool. 5. Ethics in LA: LA tools should not be an imposed element on learners and teachers who are subjected to be passive beneficiaries of said tools, but rather be an agentic and added value to the whole learning process. Therefore, the LA role should be eventually limited to one of awareness and recommendation [ 13 ]. All in all, the involvement of these aspects in the design process may vary from one tool to another, but they shape solid points of reflection during said process.

Thus, this paper focuses on presenting the workshop on HCLA at LASI Spain ’21 and the corresponding discussion with a group of members of the LA community on the advantages and disadvantages of the HCLA approach. 2

Workshop structure

The 90 minute workshop consisted of two main segments: the first segment focused on the introduction of the main concepts of Human Centeredness and Learning Analytics, and how both merge through HCLA; followed by the second segment with the introduction of two case studies based on LA tools to be used as the object of evaluation, critique and assessment through a group discussion. The use of case studies is the major difference with respect to the LAK’21 HCLA workshop which focused on a design challenge from scratch. The first case study is based on the design and use of Teacher Action Planner (TAP), a tool that supports teachers’ orchestration actions in inquiry-based K-12 science learning [ 14, 15 ]. The second case study is based on a LA dashboard focused on communities of learning design practice [ 16 ].

Since the workshop was organized in a hybrid format, the creation and supervision of the groups of work mainly took into account that aspect, making it easier for both the participants to interact with each other and enhancing the sharing of ideas and participatory experience further, and for the workshop facilitators to give better feedback.

As a means to efficiently use the short workshop duration (1,5 hours), a prequestionnaire was distributed prior to the event in order to have information on the participants’ profiles, their prior knowledge and motivation and if they were joining online or face to face. The results of the participating profiles were varied, mostly being doctoral students (52,9%). Other profiles included postdoctoral researchers (23,5%), data scientists and lecturers (17,6% each), and university professors, designers and teaching assistants (11,8% each). It is important to note that despite the local focus of LASI Spain ’21, participants came from six different countries (Spain, Estonia, Greece, Hungary, Malaysia and Cyprus).

Participants were also asked about their motivation behind joining the workshop and the answers were mainly on the interest in learning more about Human-Centered methodologies and Learning Analytics research.

The total number of participants was 17, making a total of 5 groups (3 face to face groups, 1 fully online group and 1 hybrid group with online participants and onsite facilitator). The case studies were assigned as follows: for the face to face participants, two groups has the case 1 (Design and use of Teacher action Planner in inquiry-based K-12 science learning) and one group had the case 2 (Design and use of an LA dashboard for a Learning Design community), and each group of the online participants worked on a different case.

The main activity consisted in a group discussion where the participants had to answer a series of questions in a survey. Both cases had the same questions but were supported with their respective introductions. The survey had two main sections: The first section had an introduction to the respective case study and focused on the critique and assessment of 4 main elements. These 4 elements are: (i) Objectives and goals of the LA tools, (ii) Research methodology applied, (iii) Results and findings, and (iv) Reflection space (an open question where the participants could add reflections and bring forth any possible critique). The first three elements were assessed through a Likert scale each and an open question to justify the given score. The fourth element was specifically left open for any reflections the participants might have.

As for the second activity, it consisted in reflecting about one advantage and one disadvantage participants considered to be relevant in the HCLA approach.

The participants had a total of 25 minutes to discuss and write their reflections in the survey before submitting it. The role of the workshop facilitator (one per group) was to answer and clarify any possible questions the participants might have during the activity. Once the activity was submitted, a debriefing session took place where one spokesperson for each group shared their thoughts on the workshop, followed by a synthesis of the main findings.

Results

Prior to the workshop, participants stated in the pre-workshop survey that they were interested in LA and HCLA research. They were able to provide good definitions of HCD and more than half of them already had used HCD approaches in their work. However, a smaller number had used HCD approached for LA.

They were also asked to think about what are the advantages and disadvantages that HCLA has in order to have comparative data, since they were asked again this same question during the group discussion in the workshop questionnaire.

The advantages presented by the participants centered around aspects related to: (i) involving key stakeholders for a better understanding of the context, need and requirements of the user, (ii) achieving higher user adoption, (iii) providing more optimized measurements and conclusions for the end product users and (iv) resulting overall better usability and accessibility of the product thanks to the HCD perspective. One particular answer of a participant could sum up the advantages presented: “The meaning, interaction opportunities, functions, and attributes associated with the system will be defined by the people for whom the system is intended and therefore they will be better”.

On the other hand, the disadvantages mainly focused on two big aspects: (i) the approach is highly time-consuming and it requires a lot of effort and resources and (ii) the objective is difficult to be achieved when applying HC approaches, since users may be misinformed about their real needs and potential solutions, so the final solution might be too biased.

Finally, the most important pointers and aspect of each study case will be introduced in the following subsections. 3.1

Design and use of Teacher Action Planner (TAP) in inquiry-based K-12 science learning

The first case study is based on Teacher Action Planner (TAP), a LA tool that supports teachers’ orchestration actions [ 14, 15 ]. This project corresponds to a designbased research (DBR) study, consisting of a 2-year partnership involving three researchers, three system developers, and five middle school science teachers. The study goal was to develop an activity-centered LA solution for a Web-based Inquiry Science Environment (WISE) unit on global climate change. The resulting dashboard was heavily based on: the Knowledge Integration (KI) theoretical framework of learning; the functional integration of the Learning Design of the WISE unit together with meaningful analytics; the storytelling principles that focused on challenges posed by the teachers themselves. The evaluation results showed increased adoption and enhanced learning outcomes, especially for the teachers whose Technology, Pedagogy Content and Knowledge (TPACK) profile was aligned with underlying design decisions.

Overall, the three groups that participated in the discussion of this case study gave positive feedback when evaluating the objectives of the tool. Though, one of the groups pointed out that even though the objective of the tool is clear, what is not clear to them was if the implementation of the tool actually meets said objective and to which degree. They stressed on the importance of having this information in order to properly assess the tool.

The research methodology was deemed as structured, especially focusing on the positive aspects of the importance of involving the stakeholders in the design process. Though, the time and resources required can be a setback if not managed properly, especially if there is no management on how much time is the process expected to take and how many resources are expected to be used.

However, on the results and findings, there were some criticism from the participants related to their lack of sufficient familiarity with the tool and suggested that some time for tool exploration would have been necessary. They also pointed out that it may be due to the limited time constraints of the workshop. The storytelling feature is also mentioned as a helpful way to help teachers interpret data, yet resource wise it may be too consuming to automate the storytelling process. 3.2

Design and use of an LA dashboard for a Learning Design community

The second case study is based on a LA dashboard focused on communities of learning design practice supported by a social platform named ILDE. The dashboard aims at visualizing the participation behavior of the community members (e.g., number of comments, number of created learning designs), members’ interactions with the shared artifacts (e.g., number of re-used learning designs) and the learning design tools (e.g., frequency of using a specific tool). The community members can select a specific time period to investigate the activity of the community and explore the specific member, artefact or tool that draws their attention. The dashboard was designed and evaluated following a Design-Based Research (DBR) methodology and aimed at providing better community awareness in the context of learning design communities [ 16 ].

The two working groups also positively evaluated the second case with respect to the objective, research methodology and overall findings. However, answers to open questions and the discussion between the groups’ facilitator and members provided more information for the HCLA approach.

Regarding the objectives, participants mentioned the alignment between the main goal and final integration but a definition of sub-objectives could lead to the development of different/varied visualizations. According to the working groups, the overall DBR methodology was well elaborated and implemented but it could be timeconsuming. The development of the visualizations could include more stakeholder involvement (e.g., teachers with varied profiles and experiences) and focus on priorities according to the end-users’ needs and the overall aim of the visualization.

When discussing the results participants stated that the definition of indicators would help in this context and a space for discussing the use of the dashboard with annotations would be meaningful. After elaborating the case, participants mentioned as advantages of the HCLA methodology, the stakeholder involvement helps to understand the complexity of the learning situation in a particular context. They also pointed out that time, effort and access to a number of dedicated stakeholders can be a drawback. 4

Conclusions

Regarding the workshop activity results there were some interesting answers that strengthened a number of actual early findings in HCLA and some points that raised some critics to the approach that could give hints for the future steps of HCLA. One interesting aspect of the workshop was the discussion on two completely different HCLA cases that led to similar conclusions with respect to the advantages (e.g., stakeholders’ involvement being a positive aspect) and disadvantages (e.g., the highly time-consuming process being the main drawback) of the HCLA approach. However, the presentation and elaboration of cases led to contextualized discussion that could create a database for case specific findings and show more value in participatory approaches in HCLA research. Some strengthened findings examples are: HCLA looks for LA solutions (mainly used in the Human Centered Design field) designed for humans with humans, considering the human perspective. Participants also addressed that through participation of stakeholders in the design process, the adoption of the tool may be wider and the innovation in LA may correspond better to the users’ needs and wishes. On the other hand, and regarding the disadvantageous aspects of the HCLA process, participants expressed that using techniques from the HCLA compendium requires a lot of resources.

The critical issues that showed up during the discussion are mainly centered around two points: The first one discusses that the design process may be difficult to understand if that process is not followed properly (taking into account the three main factors of HCLA: human factors, social factors and technology factors). And linked to the first point, the second point touches on some doubts linked to the HCLA approach methodology, especially coming from a non-specialized public. Said doubts address possible loops that may be created through constant research and the iterative process, thus not finding the “stopping point” and not having a clear timing on how long the process would take due to it. This first point may be connected to one of the early findings in HCLA (LAK’21 workshop) [ 7 ] previously introduced: “3. LA design choices from non-data experts: non-data experts are unlikely to be aware of the implications of LA design choices” which aside from impacting the use of time and resources, gaps or loops in the design process might also be harder to avoid. Another doubt (or more like a criticism) points out that if the feedback of the stakeholders and involved parts is not captured in a structured way, it might be excessively chaotic and out of scope, rendering the approach impractical.

These two critical points may hint at making HCLA approach more accessible and easier to understand for non-specialized users, which brings us to one of the previously mentioned findings about teachers and students: they should be considered nondata experts. Therefore, they should probably be considered non-design experts too, which raises the need to make HCLA as an approach more transparent and friendly for the non-specialized users involved in the design process. The series of HCLA workshops could incorporate this challenge, by focusing on identifying tools and practices that make the approach easier to grasp by the community. There are some examples of tools extending on this last aspect, namely the LA tutorial by NYU Learn and LASI’21 tutorial [ 17 ].

Furthermore, these two arguments led to some questions that may be regarded as critical for the HCLA community, inviting practitioners to reflect about further refinement of the approach: How accessible of an approach is the design process of HCLA? Are the contributions of all the involver parts (i.e., users and stakeholders) being considered with the same level of importance despite the differences in the level of expertise? And how does that affect (in regard to time and resources) the design process itself? Therefore, and as a final reflection, these questions might be a starting point guiding the future steps on research regarding HCLA as an approach.

Acknowledgements

This work has been partially funded by the EU Regional Development Fund and the National Research Agency of the Spanish Ministry of Science and Innovation under project grants RED2018-102725-T, TIN2017-85179-C3-2-R, TIN2017-85179-C3-3R, and PID2020-11584RB-C23, PID2020-112584RB-C33. D. Hernández-Leo (Serra Húnter) acknowledges the support by ICREA under the ICREA Academia program. P. Santos acknowledges the support under the Ramón y Cajal program.

References

1. Siemens , G. & Long , P. ( 2011 ). Penetrating the Fog: Analytics in Learning and Education . EDUCAUSE Review . 5 . 30 - 32 . 10 .17471/ 2499 -4324/195.

2. Gašević , D. , Kovanović , V. , & Joksimović , S. ( 2017 ). Piercing the learning analytics puzzle: A consolidated model of a field of research and practice . Learning: Research and Practice , 3 ( 1 ), 63 - 78 .

3. Clow , D. ( 2013 ). An overview of learning analytics . Teaching in Higher Education 18 ( 6 ), 683 - 695 , https://doi.org/10.1080/13562517. 2013 .827653

4. Prieto Alvarez , C. , Martinez-Maldonado , R. , Anderson , T. ( 2018 ). Co-designing learning analytics tools with learners . In Jason M. Lodge , Jared Cooney Horvath, & L. Corrin (Eds.), Learning analytics in the classroom: translating learning analytics research for teachers . London: Routledge 10 .4324/9781351113038-7.

5. Buckingham Shum , S. , Ferguson , R. & Martinez-Maldonado , R. ( 2019 ). Human-Centred Learning Analytics . Journal of Learning Analytics, JLA , 6 ( 2 ) https://doi.org/10.18608/jla. 2019 . 62 .1

6. Michos , K. , Lang , C. , Hernández-Leo , D. , & Price-Dennis , D. ( 2020 ). Involving teachers in learning analytics design: Lessons learned from two case studies . In Proceedings of the Tenth international conference on learning analytics & knowledge (pp. 94 - 99 ).

7. Martínez-Maldonado , R. , et al. ( 2021 ). LAK21 First International Workshop on HumanCentred Learning Analytics (HCLA) . https://sites.google.com/view/hcla21

8. https://lasi21.snola.es/

9. https://snola.es/

10. https://ea-tel. eu/ectel2021

11. https://www.solaresearch.org/

12. Maltese , A.V. , Harsh , J.A. , & Dubravka , S. ( 2015 ). Data visualization literacy: Investigating data interpretation along the novice-expert continuum . Journal of College Science Teaching , 45 ( 1 ) 84 - 90 .

13. Tchounikine , Pierre. ( 2019 ). Learner's agency and CSCL technologies: towards an emancipatory perspective . International Journal of Computer-Supported Collaborative Learning , 14 ( 2 ), 237 - 250 .

14. Dimitriadis , Y. , Martínez-Maldonado , R. , & Wiley, K. ( 2021 ). Human-Centered Design principles for actionable Learning Analytics . In: Tsiatsos T., Demetriadis S. , Mikropoulos A. , Dadgilelis V . (eds) Research on E-Learning and ICT in Educations. Springer, Cham. 277 - 296 , https://doi.org/10.1007/978-3- 030 -64363-8-15

15. Wiley, K. , Dimitriadis , Y. , Bradford , A. , & Linn , M. ( 2020 ). From theory to action: Developing and evaluating Learning Analytics for Learning Design, Learning Analytics and Knowledge Conference (LAK 2020 ), pp. 569 - 578 https://doi.org/10.1145/3375462.3375540

16. Michos , K. , & Hernández-Leo , D. ( 2018 ). Supporting awareness in communities of learning design practice . Computers in Human Behaviour , 85 , 255 - 270 .

17. Sarmiento , J.P , and Campos , F. ( 2021 ). Power to the user. 6 Techniques for participatory design of learning analytics with students and teachers . Tutorial at LASI 2021 . https://www.solaresearch.org/events/lasi/lasi21/workshops-tutorials/