=Paper=
{{Paper
|id=Vol-2704/paper1
|storemode=property
|title=Applying the LALA Framework for the adoption of a Learning Analytics tool in Latin America: Two case studies in Ecuador
|pdfUrl=https://ceur-ws.org/Vol-2704/paper1.pdf
|volume=Vol-2704
|authors=Miguel Angel Zúñiga-Prieto,Margarita Ortiz,Marlon Ulloa,Alberto Jiménez
|dblpUrl=https://dblp.org/rec/conf/ectel/PrietoOUJ20
}}
==Applying the LALA Framework for the adoption of a Learning Analytics tool in Latin America: Two case studies in Ecuador==
https://ceur-ws.org/Vol-2704/paper1.pdf
Applying the LALA Framework for the adoption
of a Learning Analytics tool in Latin America:
Two case studies in Ecuador
Miguel Angel Zúñiga-Prieto1 , Margarita Ortiz2 , Marlon Ulloa1 , and Alberto
Jiménez2
1
Departamento de Ciencias de la Computación - Universidad de Cuenca, Cuenca,
Ecuador
miguel.zunigap@ucuenca.edu.ec, marlon.ulloa.amaya@gmail.com
2
Escuela Superior Politécnica del Litoral, ESPOL, Information Technology Center,
Campus Gustavo Galindo, Guayaquil, Ecuador
Campus Gustavo Galindo km 30.5 Vı́a Perimetral, P.O. Box 09-01-6863, Guayaquil
{maelorti,albjimen}@espol.edu.ec
Abstract. Worldwide, Higher Education Institutions (HEIs) are recog-
nizing the benefits of using Learning Analytics (LA). Thus, there is more
research on the adoption of LA tools as well as works presenting different
frameworks for implementing LA in HEIs, mostly in Europe. In the case
of Latin America, the LALA Framework was defined, containing detailed
guidelines for LA adoption that take into account policies, ethics, and
development of tools in the Latin American context. However, this frame-
work has not been applied in real scenarios. Thus, this paper presents
the results obtained with the application of the LALA Framework for
the development and adoption of LA tools in two Latin-American HEIs
with different LA contexts. As a result, this work not only shows the
feasibility of this framework to guide the adoption of LA tools but also
shows that different LA context requires the execution of activities ap-
plying different approaches. This work proposes changes to improve the
LALA Framework. Changes mainly related to the inclusion of adoption
alternatives that allow practitioners to select the one suitable for their
specific institutional context.
Keywords: Learning Analytics · Framework · Learning Analytics Adop-
tion
1 Introduction
In recent years, Higher education institutions(HEIs) have begun to explore the
benefits of using Learning Analytics (LA) [13]. Some works such as [14] focus on
Copyright c 2020 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0).
2 M. Zúñiga-Prieto et.al.
the development and implementation of the different LA tools (e.g., dashboards),
whereas other works provide approaches to adopt LA policies or implement
LA tools, mostly in a European context. For instance, the SHEILA Framework
consists of policies that can be used to inform strategic planning and policy
processes for large-scale implementation of LA in a higher education context.
While in [2], a guide to the implementation of LA in HEIs is presented. In the
case of Latin America, where LA is beginning to gain more attention [3], the
project Learning Analytics in Latin America (LALA), provides a guide that
aimed to help HEIs to adopt, adapt and implement LA policies and tools [5].
This guide is known as the LALA Framework [8].
However, studies mentioning how an institution adopts LA policies and tools
are scarce [12]. Hence, this paper presents the experience of two HEIs in Latin
American applying the LALA Framework for the adoption of a LA Tool for
Academic Counseling. This paper’s contribution is the analysis of these two case
studies coming from different LA backgrounds: one with prior experience while
the other one with none. This work is structured as follows. First, the LALA
Framework is explained. After that, the application of the LALA Framework
with the case studies is described in detail. Finally, a discussion and conclusion
section is explained.
2 LALA Framework
The LALA Framework is a set of methodologies and instruments to facilitate
the implementation of LA tools in Higher Education HEIs in Latin America [8].
It consists of four dimensions: institutional, technical, ethical, and communal.
The framework is flexible enough to allow any institution to use it according to
its needs. It was adapted from the SHEILA Framework to the Latin American
context.
The Institutional dimension helps to identify the current state and needs
regarding LA in the institution. It consists of 4 phases. Phase 1 is about per-
forming an institutional diagnosis. This is achieved through the LALA canvas,
a document that analyzes the current state of LA in the institution around six
dimensions: a strategy for change, desired behaviors, internal capabilities, polit-
ical context, influential actors and measurement and evaluation plan. The aim
of phase 2 is to understand the political context and institutional needs. This is
done through a protocol for interviews with institutional leaders, professors and
students about three dimensions: current state of LA adoption, the desired state
of adoption of LA and challenges for LA adoption. Phase 3 helps to identify what
is expected from the use of educational data. The instrument used is an online
questionnaire for students and professors and analyses normative vs. predictive
expectations about privacy and the use of educational data. Finally, phase 4
helps develop a change strategy through the LALA template, a document that
analyses the same six dimensions in the LALA canvas. The difference is that the
latter is based on what is expected to occur in the institution as the result of
adopting a LA tool.
Title Suppressed Due to Excessive Length 3
The Technological dimension aims to obtain the system requirements, iden-
tify the technical considerations for the implementation of a LA tool, and sets
guidelines for evaluation and testing. To obtain system requirements, an instru-
ment named Orchestration of Learning (ORLA) [7] is recommended. It helps to
analyze system requirements from the teacher, researcher, and developer. When
identifying technical considerations, the framework recommends taking into con-
sideration aspects such as hardware, software, personnel, and data sources. Fi-
nally, for the evaluation and testing of the system, the LALA Framework presents
a set of guidelines for considering different types of tests such as usability, system
tests, among others.
The ethical dimension aims at considering the ethical and privacy considera-
tions to take into account when adopting a LA tool. The first phase is to conduct
a review of the literature to search for national and international regulations in
order to make stakeholders aware of their existence and how they could be im-
plemented in their HEIs. The second phase focuses on anticipating students’ and
professor’s expectations by interviewing them and asking them to fill out surveys
related to aspects such as being aware if the institution asks for permission to
use their data. The questions related to this phase are part of the instruments of
phases 2 and 3 of the institutional dimension. The last phase focuses on taking
actions to ensure proper treatment and use of data at an ethical and privacy
level. One of the suggested activities in this phase it to design consent forms.
Finally, the community dimension gives guidelines to join the LALA com-
munity. The LALA Community is aimed to promote a long-term sustainable
cooperation, creating lasting relationships among its members, which contribute
to the replication of the results obtained by the LALA project [1]. This is achieved
by providing a space for research and knowledge exchange in order to develop
local capacity in HEIs in Latin America.
3 Aplication of LALA Framework: Case Studies
In this section, we describe the case studies of two HEIs that applied the LALA
Framework to adopt a LA tool, namely, a counseling system. We put particular
emphasis on describing the application of the technological dimension since the
execution of its activities was the most influenced by each institution context.
The different HEIs’ contexts required to apply different approaches or techniques
during the execution.
3.1 Context
Universidad de Cuenca (U1) and Escuela Superior Politécnica del Litoral (U2)
are Ecuadorian public HEIses. While the former offers a wide variety of knowl-
edge fields such as medicine, engineering, social sciences, the latter is only en-
gineering oriented. It is also different in its population. U1 has approximately
17000 students, while U2 10000 students. However, one main difference lies in
the adoption of LA tools. U1 has no previous experience in implementing any LA
4 M. Zúñiga-Prieto et.al.
tool, nor has policies or processes related to academic counseling. While U2 has
already implemented an academic counseling system since 2014 and the process
of academic counseling has already been institutionalized.
3.2 Institutional Dimension
Both HEIs followed the guidelines of the LALA Framework to identify the cur-
rent state and needs of LA in their institutions. Phase 1 about performing an
institutional diagnosis, was executed through the use of the LALA canvas. The
instrument was filled out by two decision-makers and eight researchers in total in
U1 and U2. Phase 2 about understanding the political context and institutional
needs, was accomplished through interviews with 19 institutional leaders and
focus groups to 31 teachers and 27 students. Phase 3 about identifying what is
expected from the use of educational data, was applied through surveys to 912
students and 166 teachers. Finally, phase 4 about developing a change strategy
was executed through the LALA template. The instrument was filled out by the
same team as in phase 1. After triangulating and analyzing the results with the
involvement of 4 researchers in each HEI, the primary need for students was re-
lated to having quality feedback and timely support. This translated in the case
of U1 in designing a counseling system that could support students in decision
making. Regarding U2, the need came from students, teachers and leaders to
reinforce the current counseling system with more data to make better decisions
when advising students.
3.3 Technical Dimension
Based on the needs identified as a result of executing the institutional dimen-
sion, both HEIs considered that a counseling system would satisfy those needs.
Therefore, the technological dimension aimed at designing, implementing a data
visualization tool to facilitate the dialogue between a student and his/her aca-
demic counselor.
There exist different approaches for the development of data visualization
tools (e.g., design thinking) that are applied depending on specific project re-
quirements or organizational policies. Independently of the applied approach, all
of them include everyday activities (e.g., requirements elicitation, development,
testing), existing other activities (or similar ones) specific of each approach that
defines differences among them. Next, we show how each HEI followed a differ-
ent development approach while still using the guidelines provided by the LALA
Framework.
3.3.1 Requirements Definition Requirements elicitation is a complex ac-
tivity that is supported by a variety of techniques (e.g., brainstorming, focus
groups, interviews, observation, prototyping, document analysis, questionnaires)
that developers use according to the project context. HEI teams executed this
activity once they completed the Institutional Dimension, which produced the
Title Suppressed Due to Excessive Length 5
identification of institutional needs and the main stakeholders. The main stake-
holders’ identification allowed U1 and U2 to direct their efforts in involving peo-
ple who were going to act as project promoters—, facilitate access to resources—,
or who were going to be critical requirement sources during elicitation.
The LALA Framework suggests using ORLA as the instrument for defining
and extracting design requirements; however, U1 and U2 did not find it suitable
for their contexts because the questions in ORLA are related to the design of LA
tools to support teachers at the classroom level [7]. Therefore, being a counseling
system, the chosen tool in U1 and U2, the ORLA questions were not applicable.
Instead, the elicitation process followed by each institution took into account the
activities proposed for the analysis process suggested in the LALA Framework
”Guide for the extraction of Requirements for the design of LA tools” such as
identify the requirements that coincide among the different actors to ensure that
the minimum requirements will be considered during the design.
Due to the lack of experience of U1 in the counseling process (and counseling
tool usage), this HEI executed a combination of focus groups, interviews, and
prototyping techniques for gathering requirements. The prototyping technique
is useful when developing human-computer interfaces, or when the stakeholders
do not know about available solutions [9], [6]. It consists of using existing exam-
ples of similar systems as instruments for requirements elicitation [16]. Therefore,
this technique allowed U1 to use the mock-ups generated by other LALA Project
partners as a starting point to get additional requirements. On the other hand,
unstructured interviews allow obtaining an in-depth knowledge of a domain [15],
which helped U1 to understand the rationale about the information to be visu-
alized.
Regarding U2, according to its context, it was just necessary to ask stakehold-
ers what other data was needed to make better decisions when advising students.
Techniques such as brainstorming or interviews were considered time-consuming
due to a large number of advisors (around 300). Hence, a questionnaire, an ef-
ficient technique to gather requirements from multiple stakeholders quickly by
avoiding redundant and irrelevant data [10], was applied. The questionnaire in-
cluded one open question aimed to collect information about new visualizations.
To refine the initial requirements, U1 used focus groups and interviews, while
U2 used only interviews. Focus groups help to get feedback on prototypes and
to let stakeholders generate new ideas [4], being a suitable technique for refining
and evaluating design artifacts [11]. In both scenarios, a low fidelity prototype
was used.
3.3.2 Tool Development and Implementation Information and interac-
tion are both essential characteristics of visualizations. During the development
and implementation phase, U1 and U2 generated high fidelity prototypes that
allowed users: i) to interact (or simulate interaction) with the designed dash-
boards, and ii) to validate dashboards’ interaction capabilities.
To simulate interactions, U1 used an online mockup creation tool. On the
other hand, the experience in the development and implementation of counseling
6 M. Zúñiga-Prieto et.al.
tools of U2 allowed them to implement an executable high fidelity prototype by
using the technology to be used in the final deployment.
To validate interactions, U1 executed focus groups in which a development
team member used the mockup to simulate the analysis of students’ academic
performance, showing users how visualizations change in response to events (e.g.,
click, mouse over). On the other hand, U2 designed a validation protocol that
included specific tasks and expected execution times, to be carried out by users
by using the executable prototype. To improve the dashboard design, U1 used
the qualitative information obtained as a result of focus groups, whereas U2
used quantitative information from the validation protocol. After validation,
both HEIs built a beta version of their dashboards that are being used in the
piloting phase.
In the tool development and implementation phase, both U1 and U2 applied
the LALA Framework “Guide of technical considerations for the development
and implementation.” However, U1 used it when building the beta version, while
U2 used it when building the executable high fidelity prototype. This fact shows
that the guidelines provided by LALA can be used according to the development
process chosen by each institution.
3.3.3 Tool Evaluation and Testing The evaluation and testing of the de-
veloped dashboards are taking place during the piloting phase. LALA project
members defined a process for this phase, which includes the tests and evalua-
tions suggested in the LALA Framework ”Guide on considerations for the design
of the procedure for evaluation and testing of the tool.” This process includes the
following activities: identifying the current state (baseline) about the counseling
process/system in order to later make comparisons after the piloting/training
takes place, planning and executing training, tool usage reporting, and evalua-
tion. Where, as suggested by the guide, process activities take into account not
only technological aspects (e.g., data quality, usability, performance) but also
ethical considerations (e.g., informed consents).
In the case of U1, there is no baseline because there is no existing counseling
process; therefore, performance or usability reports were not generated. Regard-
ing the training process, it helped U1 to validate data quality of visualizations
as well as identify and implement new requirements. Additionally, introducing
the counseling process as well as the counseling tool as part of the academic
activities is being delayed due to issues related to cultural changes and lack of
policies. For instance, some teachers and administrative staff think it is going to
represent an extra workload. Furthermore, counseling is perceived as a useless
task since students are old enough to make their own academic-related deci-
sions. While from some academic decision-makers’ point of view, results need to
be shown in order to create policies. The above issues make it clear the need for
defining policies that facilitate the adoption of LA by HEIs.
On the other hand, U2 was able to report the impact of introducing changes
to its existing dashboard and had escalated the new dashboard at the institu-
tional level (i.e., all program studies advisors are using the new version). How-
Title Suppressed Due to Excessive Length 7
ever, implementing new requirements that emerged during training and actual
use is not just adding new visualizations but changing existing ones, which means
taking the risk of changing the previous system version source code. Further-
more, this requires strong evidence about the benefits of implementing the new
requirements in order to give confidence to decision-makers to authorize the
modification of the previous system.
3.4 Ethical Dimension
Due to time constraints, both HEIs only applied the second and third phases
proposed in this dimension; anticipate professors’ and students’ expectations and
adapting ethical and privacy considerations for the creation of the institutional
framework on ethics and data privacy, respectively. Results from executing the
second phase showed that in both HEIs, there are no institutional policies re-
garding data privacy and protection; however, some teachers, students and insti-
tutional leaders think those policies exist. Finally, in the third phase, to ensure
good treatment and use of data at ethical and privacy level, informed consents
forms were created. The stakeholders involved in the interviews, focus groups
and surveys were asked to sign these forms.
3.5 Community Dimension
Since both HEIs are founding members of the LALA project, the community
dimension was not used. Nevertheless, if both HEIs had used this dimension,
they would have subscribed to the LALA community. This would have allowed
them the opportunity to receive newsletters about dissemination events where
the experience of designing and implementing a LA tool would have been shared
with other HEIs.
4 Conclusions and Recommendations
This paper presents the application of the LALA Framework for the develop-
ment and adoption of a LA tool (i.e., an academic counseling dashboard) in two
Latin-American HEIs with different LA contexts. For instance, U1 has no previ-
ous experience using LA tools or supporting students trough academic counseling
sessions, whereas U2 has experience with both. When executing the dimensions
proposed by the framework, the execution of the institutional and ethical dimen-
sions showed no differences between both HEIs. Nevertheless, changes appeared
while executing the technological dimension. For instance, when gathering re-
quirements, U1 used a prototyping technique that helped to obtain requirements
from actors without previous experience with LA tools, while U2 used a survey
to collect additional data o visualizations needs from actors requiring improve-
ments in the current counseling system.
Regardless of the different LA context of the HEIs involved in this study,
these institutions were able to apply the LALA Framework, which points out its
8 M. Zúñiga-Prieto et.al.
flexibility and openness. However, during the framework application, both HEIs
had to apply techniques not proposed by it (e.g., requirement elicitation tech-
niques). This because the proposed techniques were not appropriate for the HEIs’
context. Therefore, in order to enrich the framework, the following suggestions
are proposed. First, although the Institutional Dimension allows HEIs to iden-
tify their current state and the desired state concerning policies and strategies
for the incorporation of LA tools, this context should be categorized according
to the level of maturity in using LA (e.g., HEIs without LA experience, HEIs
with LA experience). Furthermore, the framework should also suggest activities
according to this maturity level. For example, although the framework allows
HEIs to execute a dimension in any order, it would be better if practitioners
know under what maturity level to execute a dimension. Second, the framework
should provide more specific guidance to practitioners about the tasks to per-
form or techniques to apply during the execution of the proposed activities, and
it should be connected to the HEI’s maturity level identified in the institutional
dimension. For example, during the evaluation phase in the technological dimen-
sion, is not enough to suggest a set of tests. Practitioners need to know the task
in which those test must be applied. Another example occurs when gathering
the tool’s requirements. The framework should provide practitioners different
techniques to gather system requirements. One limitation of this work is that it
was applied only in a Higher Education setting in a face to face modality with
the members of the LALA project. Thus, there is no information of the applica-
tion of this framework in other contexts. For instance, a high school in on online
modality. The aforementioned limitations help us to define future work: the ap-
plicability of the LALA Framework in other contexts (e.g online education, in
other latin american countries, with a different educational levels), and with a
different tool besides dashboards.
Acknowledgments
Work funded by the LALA project (grant no. 586120-EPP-1-2017-1-ES-EPPKA2-
CBHE-JP). This project has been funded with support from the European Com-
mission.This publication reflects only the views of the authors, and the Commis-
sion cannot be held responsible for any use which may be made of the information
contained therein.
References
1. Lala community cooperation network (2017), hhttps://www.lalaproject.org/be-
part-of-lala/
2. Adejo, O., Connolly, T.: Learning analytics in higher education development: A
roadmap. Journal of Education and Practice 8(15), 156–163 (2017)
3. Dos Santos, H.L., Cechinel, C., Nunes, J.B.C., Ochoa, X.: An initial review of
learning analytics in latin america. In: 2017 Twelfth Latin American Conference
on Learning Technologies (LACLO). pp. 1–9. IEEE (2017)
Title Suppressed Due to Excessive Length 9
4. Hogan, T., Hornecker, E.: Blending the repertory grid technique with focus groups
to reveal rich design relevant insight. In: Proceedings of the 6th International
Conference on Designing Pleasurable Products and Interfaces. pp. 116–125. ACM
(2013)
5. Maldonado-Mahauad, J., Hilliger, I., De Laet, T., Millecamp, M., Verbert, K.,
Ochoa, X., Pérez-Sanagustı́n, M.: The lala project: Building capacity to use learn-
ing analytics to improve higher education in latin america. In: companion pro-
ceedings of the 8th international learning analytics & knowledge conference. pp.
630–637. LAK (2018)
6. Ortiz-Rojas, M., Jimenez, A., Maya, R., Hiliger, I., Chiluiza, K.: A step by step
methodology for software design of a learning analytics tool in latin america: A
case study in ecuador (2019)
7. Prieto, L.P., Rodrı́guez-Triana, M.J., Martı́nez-Maldonado, R., Dimitriadis, Y.,
Gasevic, D.: Orchestrating learning analytics (orla): supporting the adoption of
learning analytics at the practitioner level (2018)
8. Sanagustı́n, M., Hilliger, I., Maldonado, J., Pérez, R., Ramı́rez, L., Muñoz-Merino,
P., Tsai, Y., Ortiz-Rojas, M., B.T., Zuñiga-Prieto, M., Sheihing, E., Whitelock-
Wainright, A.: LALA Framework. https://www.lalaproject.org/deliverables/
(2019)
9. Shams-Ul-Arif, Q.K., Gahyyur, S.: Requirements engineering processes,
tools/technologies, & methodologies. International Journal of Reviews in
Computing 2(6), 41–56 (2009)
10. Sharma, S., Pandey, S.: Revisiting requirements elicitation techniques. Interna-
tional Journal of Computer Applications 75(12), 777–780 (2013)
11. Tremblay, M.C., Hevner, A.R., Berndt, D.J.: Focus groups for artifact refinement
and evaluation in design research. Communications of the Association for Infor-
mation Systems 26(27), 599–618 (2010)
12. Tsai, Y.S., Gasevic, D.: Learning analytics in higher education—challenges and
policies: a review of eight learning analytics policies. In: Proceedings of the seventh
international learning analytics & knowledge conference. pp. 233–242. ACM (2017)
13. Tsai, Y.S., Gašević, D., Whitelock-Wainwright, A., Muñoz-Merino, P.J., Moreno-
Marcos, P.M., Fernández, A.R., Kloos, C.D., Scheffel, M., Jivet, I., Drachsler, H.,
et al.: Sheila: Support higher education to integrate learning analytics (2018)
14. Verbert, K., Govaerts, S., Duval, E., Santos, J.L., Assche, F., Parra, G., Klerkx,
J.: Learning dashboards: an overview and future research opportunities. Personal
and Ubiquitous Computing 18(6), 1499–1514 (2014)
15. Zhang, Y., Wildemuth, B.M.: Unstructured interviews. Applications of social re-
search methods to questions in information and library science pp. 222–231 (2009)
16. Zowghi, D., Coulin, C.: Requirements elicitation: A survey of techniques, ap-
proaches, and tools. In: Engineering and managing software requirements, pp.
19–46. Springer (2005)