=Paper=
{{Paper
|id=Vol-3029/paper08
|storemode=property
|title=Workshop on human-centred learning analytics: A critical analysis based on the discussion of two case studies
|pdfUrl=https://ceur-ws.org/Vol-3029/paper08.pdf
|volume=Vol-3029
|authors=Khadija El Aadmi-Laamech,Yannis Dimitriadis,Patricia Santos,Davinia Hernández-Leo,Konstantinos Michos,Alejandra Martínez-Monés
|dblpUrl=https://dblp.org/rec/conf/lasi-spain/Aadmi-Laamech0S21
}}
==Workshop on human-centred learning analytics: A critical analysis based on the discussion of two case studies==
https://ceur-ws.org/Vol-3029/paper08.pdf
Workshop on human-centred learning analytics: A
critical analysis based on the discussion of two case
studies
Khadija El Aadmi-Laamech1, Yannis Dimitriadis2, Patricia Santos1,
Davinia Hernández-Leo1, Konstantinos Michos3 and Alejandra Martínez-
Monés2
1
Universitat Pompeu Fabra, Plaça de la Mercè 10-12, 08002 Barcelona, Spain
[khadija.elaadmi, patricia.santos, davinia.hernandez-
leo]@upf.edu
2
Universidad de Valladolid, Paseo Belén 15, 47011 Valladolid, Spain
yannis@tel.uva.es
amartine@infor.uva.es
3
University of Zurich, Rämistrasse 71, 8006 Zürich, Switzerland
konstantinos.michos@ife.uzh.ch
Abstract. There is a recently growing subcommunity of LA focusing on hu-
man-centeredness that proposes a more user-centred approach to the learning
analytics practice: Human-Centred Learning Analytics (HCLA). This approach
focuses on three key elements that come into play and interact under the human
activity umbrella: human factors, social factors and technology factors. There-
fore, the main focus shifts from assuring that the artifact works as intended to
enabling many individual or cultural conceptions to unfold into uninterrupted
interfaces with technology. The workshop was organized by SNOLA (Spanish
Network of Learning Analytics) and introduced two cases used to prompt the
discussion of two LA tools designed following the HCLA approach.
This paper presents the workshop and its results based on the discussion around
the advantages and disadvantages of the two cases and their methodologies, ob-
jectives and findings according to participants’ insights and collaboration. The
two main findings this paper contributes are focused on: 1. The HCLA ap-
proach, viewed and understood from the point of view of non-experts and ex-
perts of different levels of expertise and 2. The implications that HCLA has on
the design process of the LA tools and artifacts. These two findings open more
doors for research on the enhancement of the HCLA approach and effectiveness
of the design process, therefore guiding the future directions of research on the
same line of the presented findings.
Keywords: Learning Analytics, Human-Centred Learning Analytics, Human-
Centered Design, Co-Design, Participatory Design
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
82 Learning Analytics in times of COVID-19: Opportunity from crisis
1 Introduction
Learning Analytics (LA) is defined as the measurement, collection, analysis and re-
porting of data about learners and their contexts, for purposes of understanding and
optimizing learning and the environments in which it occurs [1], with this work main-
ly focusing on the needs of two major stakeholders: teachers and learners. The foun-
dation of LA sits at the convergence of three main principles: Theory, Data Science
and Design. Thus, only when all three principles are considered and applied do the
results reach a high level of validity and effectiveness [2].
The most popular uses of LA research focus on student performance, prediction of
academic success, identification of students who are at risk of failing or dropping out
of their studies, among others [3]. Yet, as the LA approach evolved over the last dec-
ade, more uses were put in the foreground, making LA an even more powerful and
solid approach that is able to help with complex learning and teaching requirements.
Supporting the development of the student’s lifelong learning, personalizing learning
plans, enhancing learning skills such as collaboration and critical thinking, and sup-
porting quality learning and teaching by providing empirical evidence are some of the
actual goals LA has, just to name a few [3]. As LA continues growing and getting
more complex, especially from the perspective of considering user needs, the necessi-
ty to have a more human centered approach was recently born, giving way to the Hu-
man Centered subcommunity of LA: The Human-Centred Learning Analytics
(HCLA) [4, 5].
Human-Centeredness (HC) has been identified in other fields as a characteristic of
systems that have been carefully designed by determining three main points: critical
stakeholders (i.e., human factors), their relationships (i.e., social factors) and the con-
text they interrelate in (i.e., technology factors) [5]. HCLA assumes this perspective
and integrates concepts and practices like participatory design and co-design in the
design process, giving a critical and influential role to the user in the design and crea-
tion of the LA tools, switching the design logic from a “design for users” to a “design
with users” [6].
The integration of the HC and design approaches with the user as a critical role
stems from fundamental early findings in HCLA research [4, 5] and LAK21 first
workshop on HCLA: First International Workshop on Human-Centred Learning Ana-
lytics (HCLA) [7]. As HCLA is a topic still under discussion, it is important to apply
a Human Centered approach to collect feedback from the LA community to under-
stand the benefits and the pain points of the HCLA methodology. For this reason, in
this paper we present the results of a workshop organized in the context of the LASI
Spain 2021 conference [8]. It is important to mention that this workshop is an adapta-
tion of the first HCLA workshop organized in LAK’21 [7] mentioned above.
The organization of this workshop was conducted by members of SNOLA (Span-
ish Network of Learning Analytics) [9], in response to one of the objectives of the
network, which aims at reaching a better understanding of users’ needs with respect to
LA with special emphasis on the TEL (Technology Enhanced Learning) Spanish con-
text. As part of this objective, this workshop was organized building on the series of
HCLA workshops offered in international conferences (LAK’21 and ECTEL’21
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
Learning Analytics in times of COVID-19: Opportunity from crisis 83
HCLA workshops [7, 10] and in the global Learning Analytics Summer Institute
(LASI) [8] of the global LA community (SOLAR) [11], to discuss with the LA com-
munity of practitioners in Spain the advantages and disadvantages of adopting an
HCLA approach. In particular, this workshop provides a more specific view on the
critical discussion of case studies’ tool design processes co-authored by members of
the SNOLA network. The findings of the first LAK’21 workshop on HCLA served as
a starting point for further discussion on HCLA among the community of practition-
ers in the field.
Some early findings from HCLA research (which should be understood better
through the community expertise in both LA and education) orbit around (but not
limited to): 1. The complexity of learning: The process of learning is a complex one.
Moreover, the needs that teachers and students have are not always strictly data relat-
ed. 2. Data literacy: Most of the actual LA tools require a level of data-literacy in
order to be able to perform with it, which is perceived as a critical obstacle. Further-
more, from a Human-Centered Design (HCD) perspective it is advisable to adapt the
tool to the teachers’ and students’ needs rather than contrariwise. 3. LA design choices
from non-data experts: Non-data experts are unlikely to be aware of the implications
of LA design choices. For instance, prior studies have shown that students find it hard
to interpret charts and data visualizations [12]. Hence why teachers and students
should be considered non-data experts. 4. Involving stakeholders: Just as important as
it is involving the social and technology factors, the human factor also constitutes a
critical element that interacts with the aforementioned two. Therefore, even though
involving stakeholders may seem to be a difficult, time-consuming and expensive task
to carry out during the different stages of the design process, it can easily turn out to
be a key element for the correct functioning, success, agency and adoption of the
resulting tool. 5. Ethics in LA: LA tools should not be an imposed element on learners
and teachers who are subjected to be passive beneficiaries of said tools, but rather be
an agentic and added value to the whole learning process. Therefore, the LA role
should be eventually limited to one of awareness and recommendation [13]. All in all,
the involvement of these aspects in the design process may vary from one tool to an-
other, but they shape solid points of reflection during said process.
Thus, this paper focuses on presenting the workshop on HCLA at LASI Spain ’21
and the corresponding discussion with a group of members of the LA community on
the advantages and disadvantages of the HCLA approach.
2 Workshop structure
The 90 minute workshop consisted of two main segments: the first segment focused
on the introduction of the main concepts of Human Centeredness and Learning Ana-
lytics, and how both merge through HCLA; followed by the second segment with the
introduction of two case studies based on LA tools to be used as the object of evalua-
tion, critique and assessment through a group discussion. The use of case studies is
the major difference with respect to the LAK’21 HCLA workshop which focused on a
design challenge from scratch. The first case study is based on the design and use of
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
84 Learning Analytics in times of COVID-19: Opportunity from crisis
Teacher Action Planner (TAP), a tool that supports teachers’ orchestration actions in
inquiry-based K-12 science learning [14, 15]. The second case study is based on a LA
dashboard focused on communities of learning design practice [16].
Since the workshop was organized in a hybrid format, the creation and supervision
of the groups of work mainly took into account that aspect, making it easier for both
the participants to interact with each other and enhancing the sharing of ideas and
participatory experience further, and for the workshop facilitators to give better feed-
back.
As a means to efficiently use the short workshop duration (1,5 hours), a pre-
questionnaire was distributed prior to the event in order to have information on the
participants’ profiles, their prior knowledge and motivation and if they were joining
online or face to face. The results of the participating profiles were varied, mostly
being doctoral students (52,9%). Other profiles included postdoctoral researchers
(23,5%), data scientists and lecturers (17,6% each), and university professors, design-
ers and teaching assistants (11,8% each). It is important to note that despite the local
focus of LASI Spain ’21, participants came from six different countries (Spain, Esto-
nia, Greece, Hungary, Malaysia and Cyprus).
Participants were also asked about their motivation behind joining the workshop
and the answers were mainly on the interest in learning more about Human-Centered
methodologies and Learning Analytics research.
The total number of participants was 17, making a total of 5 groups (3 face to face
groups, 1 fully online group and 1 hybrid group with online participants and onsite
facilitator). The case studies were assigned as follows: for the face to face partici-
pants, two groups has the case 1 (Design and use of Teacher action Planner in in-
quiry-based K-12 science learning) and one group had the case 2 (Design and use of
an LA dashboard for a Learning Design community), and each group of the online
participants worked on a different case.
The main activity consisted in a group discussion where the participants had to
answer a series of questions in a survey. Both cases had the same questions but were
supported with their respective introductions. The survey had two main sections: The
first section had an introduction to the respective case study and focused on the cri-
tique and assessment of 4 main elements. These 4 elements are: (i) Objectives and
goals of the LA tools, (ii) Research methodology applied, (iii) Results and findings,
and (iv) Reflection space (an open question where the participants could add reflec-
tions and bring forth any possible critique). The first three elements were assessed
through a Likert scale each and an open question to justify the given score. The fourth
element was specifically left open for any reflections the participants might have.
As for the second activity, it consisted in reflecting about one advantage and one
disadvantage participants considered to be relevant in the HCLA approach.
The participants had a total of 25 minutes to discuss and write their reflections in
the survey before submitting it. The role of the workshop facilitator (one per group)
was to answer and clarify any possible questions the participants might have during
the activity. Once the activity was submitted, a debriefing session took place where
one spokesperson for each group shared their thoughts on the workshop, followed by
a synthesis of the main findings.
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
Learning Analytics in times of COVID-19: Opportunity from crisis 85
3 Results
Prior to the workshop, participants stated in the pre-workshop survey that they were
interested in LA and HCLA research. They were able to provide good definitions of
HCD and more than half of them already had used HCD approaches in their work.
However, a smaller number had used HCD approached for LA.
They were also asked to think about what are the advantages and disadvantages
that HCLA has in order to have comparative data, since they were asked again this
same question during the group discussion in the workshop questionnaire.
The advantages presented by the participants centered around aspects related to: (i)
involving key stakeholders for a better understanding of the context, need and re-
quirements of the user, (ii) achieving higher user adoption, (iii) providing more opti-
mized measurements and conclusions for the end product users and (iv) resulting
overall better usability and accessibility of the product thanks to the HCD perspective.
One particular answer of a participant could sum up the advantages presented: “The
meaning, interaction opportunities, functions, and attributes associated with the sys-
tem will be defined by the people for whom the system is intended and therefore they
will be better”.
On the other hand, the disadvantages mainly focused on two big aspects: (i) the
approach is highly time-consuming and it requires a lot of effort and resources and (ii)
the objective is difficult to be achieved when applying HC approaches, since users
may be misinformed about their real needs and potential solutions, so the final solu-
tion might be too biased.
Finally, the most important pointers and aspect of each study case will be intro-
duced in the following subsections.
3.1 Design and use of Teacher Action Planner (TAP) in inquiry-based K-12
science learning
The first case study is based on Teacher Action Planner (TAP), a LA tool that sup-
ports teachers’ orchestration actions [14, 15]. This project corresponds to a design-
based research (DBR) study, consisting of a 2-year partnership involving three re-
searchers, three system developers, and five middle school science teachers. The
study goal was to develop an activity-centered LA solution for a Web-based Inquiry
Science Environment (WISE) unit on global climate change. The resulting dashboard
was heavily based on: the Knowledge Integration (KI) theoretical framework of learn-
ing; the functional integration of the Learning Design of the WISE unit together with
meaningful analytics; the storytelling principles that focused on challenges posed by
the teachers themselves. The evaluation results showed increased adoption and en-
hanced learning outcomes, especially for the teachers whose Technology, Pedagogy
Content and Knowledge (TPACK) profile was aligned with underlying design deci-
sions.
Overall, the three groups that participated in the discussion of this case study gave
positive feedback when evaluating the objectives of the tool. Though, one of the
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
86 Learning Analytics in times of COVID-19: Opportunity from crisis
groups pointed out that even though the objective of the tool is clear, what is not clear
to them was if the implementation of the tool actually meets said objective and to
which degree. They stressed on the importance of having this information in order to
properly assess the tool.
The research methodology was deemed as structured, especially focusing on the
positive aspects of the importance of involving the stakeholders in the design process.
Though, the time and resources required can be a setback if not managed properly,
especially if there is no management on how much time is the process expected to
take and how many resources are expected to be used.
However, on the results and findings, there were some criticism from the partici-
pants related to their lack of sufficient familiarity with the tool and suggested that
some time for tool exploration would have been necessary. They also pointed out that
it may be due to the limited time constraints of the workshop. The storytelling feature
is also mentioned as a helpful way to help teachers interpret data, yet resource wise it
may be too consuming to automate the storytelling process.
3.2 Design and use of an LA dashboard for a Learning Design community
The second case study is based on a LA dashboard focused on communities of learn-
ing design practice supported by a social platform named ILDE. The dashboard aims
at visualizing the participation behavior of the community members (e.g., number of
comments, number of created learning designs), members’ interactions with the
shared artifacts (e.g., number of re-used learning designs) and the learning design
tools (e.g., frequency of using a specific tool). The community members can select a
specific time period to investigate the activity of the community and explore the spe-
cific member, artefact or tool that draws their attention. The dashboard was designed
and evaluated following a Design-Based Research (DBR) methodology and aimed at
providing better community awareness in the context of learning design communities
[16].
The two working groups also positively evaluated the second case with respect to
the objective, research methodology and overall findings. However, answers to open
questions and the discussion between the groups’ facilitator and members provided
more information for the HCLA approach.
Regarding the objectives, participants mentioned the alignment between the main
goal and final integration but a definition of sub-objectives could lead to the devel-
opment of different/varied visualizations. According to the working groups, the over-
all DBR methodology was well elaborated and implemented but it could be time-
consuming. The development of the visualizations could include more stakeholder
involvement (e.g., teachers with varied profiles and experiences) and focus on priori-
ties according to the end-users’ needs and the overall aim of the visualization.
When discussing the results participants stated that the definition of indicators
would help in this context and a space for discussing the use of the dashboard with
annotations would be meaningful. After elaborating the case, participants mentioned
as advantages of the HCLA methodology, the stakeholder involvement helps to un-
derstand the complexity of the learning situation in a particular context. They also
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
Learning Analytics in times of COVID-19: Opportunity from crisis 87
pointed out that time, effort and access to a number of dedicated stakeholders can be a
drawback.
4 Conclusions
Regarding the workshop activity results there were some interesting answers that
strengthened a number of actual early findings in HCLA and some points that raised
some critics to the approach that could give hints for the future steps of HCLA. One
interesting aspect of the workshop was the discussion on two completely different
HCLA cases that led to similar conclusions with respect to the advantages (e.g.,
stakeholders’ involvement being a positive aspect) and disadvantages (e.g., the highly
time-consuming process being the main drawback) of the HCLA approach. However,
the presentation and elaboration of cases led to contextualized discussion that could
create a database for case specific findings and show more value in participatory ap-
proaches in HCLA research. Some strengthened findings examples are: HCLA looks
for LA solutions (mainly used in the Human Centered Design field) designed for hu-
mans with humans, considering the human perspective. Participants also addressed
that through participation of stakeholders in the design process, the adoption of the
tool may be wider and the innovation in LA may correspond better to the users’ needs
and wishes. On the other hand, and regarding the disadvantageous aspects of the
HCLA process, participants expressed that using techniques from the HCLA compen-
dium requires a lot of resources.
The critical issues that showed up during the discussion are mainly centered around
two points: The first one discusses that the design process may be difficult to under-
stand if that process is not followed properly (taking into account the three main fac-
tors of HCLA: human factors, social factors and technology factors). And linked to
the first point, the second point touches on some doubts linked to the HCLA approach
methodology, especially coming from a non-specialized public. Said doubts address
possible loops that may be created through constant research and the iterative process,
thus not finding the “stopping point” and not having a clear timing on how long the
process would take due to it. This first point may be connected to one of the early
findings in HCLA (LAK’21 workshop) [7] previously introduced: “3. LA design
choices from non-data experts: non-data experts are unlikely to be aware of the im-
plications of LA design choices” which aside from impacting the use of time and re-
sources, gaps or loops in the design process might also be harder to avoid. Another
doubt (or more like a criticism) points out that if the feedback of the stakeholders and
involved parts is not captured in a structured way, it might be excessively chaotic and
out of scope, rendering the approach impractical.
These two critical points may hint at making HCLA approach more accessible and
easier to understand for non-specialized users, which brings us to one of the previous-
ly mentioned findings about teachers and students: they should be considered non-
data experts. Therefore, they should probably be considered non-design experts too,
which raises the need to make HCLA as an approach more transparent and friendly
for the non-specialized users involved in the design process. The series of HCLA
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
88 Learning Analytics in times of COVID-19: Opportunity from crisis
workshops could incorporate this challenge, by focusing on identifying tools and
practices that make the approach easier to grasp by the community. There are some
examples of tools extending on this last aspect, namely the LA tutorial by NYU Learn
and LASI’21 tutorial [17].
Furthermore, these two arguments led to some questions that may be regarded as
critical for the HCLA community, inviting practitioners to reflect about further re-
finement of the approach: How accessible of an approach is the design process of
HCLA? Are the contributions of all the involver parts (i.e., users and stakeholders)
being considered with the same level of importance despite the differences in the level
of expertise? And how does that affect (in regard to time and resources) the design
process itself? Therefore, and as a final reflection, these questions might be a starting
point guiding the future steps on research regarding HCLA as an approach.
Acknowledgements
This work has been partially funded by the EU Regional Development Fund and the
National Research Agency of the Spanish Ministry of Science and Innovation under
project grants RED2018-102725-T, TIN2017-85179-C3-2-R, TIN2017-85179-C3-3-
R, and PID2020-11584RB-C23, PID2020-112584RB-C33. D. Hernández-Leo (Serra
Húnter) acknowledges the support by ICREA under the ICREA Academia program.
P. Santos acknowledges the support under the Ramón y Cajal program.
References
1. Siemens, G. & Long, P. (2011). Penetrating the Fog: Analytics in Learning and Education.
EDUCAUSE Review. 5. 30-32. 10.17471/2499-4324/195.
2. Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piercing the learning analytics puz-
zle: A consolidated model of a field of research and practice. Learning: Research and Prac-
tice, 3(1), 63-78.
3. Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education 18(6),
683-695, https://doi.org/10.1080/13562517.2013.827653
4. Prieto Alvarez, C., Martinez-Maldonado, R., Anderson, T. (2018). Co-designing learning
analytics tools with learners. In Jason M. Lodge, Jared Cooney Horvath, & L. Corrin
(Eds.), Learning analytics in the classroom: translating learning analytics research for
teachers. London: Routledge 10.4324/9781351113038-7.
5. Buckingham Shum, S., Ferguson, R. & Martinez-Maldonado, R. (2019). Human-Centred
Learning Analytics. Journal of Learning Analytics, JLA, 6(2)
https://doi.org/10.18608/jla.2019.62.1
6. Michos, K., Lang, C., Hernández-Leo, D., & Price-Dennis, D. (2020). Involving teachers
in learning analytics design: Lessons learned from two case studies. In Proceedings of the
Tenth international conference on learning analytics & knowledge (pp. 94-99).
7. Martínez-Maldonado, R., et al. (2021). LAK21 First International Workshop on Human-
Centred Learning Analytics (HCLA). https://sites.google.com/view/hcla21
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
Learning Analytics in times of COVID-19: Opportunity from crisis 89
8. https://lasi21.snola.es/
9. https://snola.es/
10. https://ea-tel.eu/ectel2021
11. https://www.solaresearch.org/
12. Maltese, A.V., Harsh, J.A., & Dubravka, S. (2015). Data visualization literacy: Investigat-
ing data interpretation along the novice–expert continuum. Journal of College Science
Teaching, 45(1) 84-90.
13. Tchounikine, Pierre. (2019). Learner’s agency and CSCL technologies: towards an eman-
cipatory perspective. International Journal of Computer-Supported Collaborative Learning,
14(2), 237-250.
14. Dimitriadis, Y., Martínez-Maldonado, R., & Wiley, K. (2021). Human-Centered Design
principles for actionable Learning Analytics. In: Tsiatsos T., Demetriadis S., Mikropoulos
A., Dadgilelis V. (eds) Research on E-Learning and ICT in Educations. Springer, Cham.
277-296, https://doi.org/10.1007/978-3-030-64363-8-15
15. Wiley, K., Dimitriadis, Y., Bradford, A., & Linn, M. (2020). From theory to action: De-
veloping and evaluating Learning Analytics for Learning Design, Learning Analytics and
Knowledge Conference (LAK 2020), pp. 569-578
https://doi.org/10.1145/3375462.3375540
16. Michos, K., & Hernández-Leo, D. (2018). Supporting awareness in communities of learn-
ing design practice. Computers in Human Behaviour, 85, 255-270.
17. Sarmiento, J.P, and Campos, F. (2021). Power to the user. 6 Techniques for participatory
design of learning analytics with students and teachers. Tutorial at LASI 2021.
https://www.solaresearch.org/events/lasi/lasi21/workshops-tutorials/
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)