Worldwide, Higher Education Institutions (HEIs) are recognizing the bene ts of using Learning Analytics (LA). Thus, there is more research on the adoption of LA tools as well as works presenting di erent frameworks for implementing LA in HEIs, mostly in Europe. In the case of Latin America, the LALA Framework was de ned, containing detailed guidelines for LA adoption that take into account policies, ethics, and development of tools in the Latin American context. However, this framework has not been applied in real scenarios. Thus, this paper presents the results obtained with the application of the LALA Framework for the development and adoption of LA tools in two Latin-American HEIs with di erent LA contexts. As a result, this work not only shows the feasibility of this framework to guide the adoption of LA tools but also shows that di erent LA context requires the execution of activities applying di erent approaches. This work proposes changes to improve the LALA Framework. Changes mainly related to the inclusion of adoption alternatives that allow practitioners to select the one suitable for their speci c institutional context.
In recent years, Higher education institutions(HEIs) have begun to explore the
bene ts of using Learning Analytics (LA) [
However, studies mentioning how an institution adopts LA policies and tools
are scarce [
The LALA Framework is a set of methodologies and instruments to facilitate
the implementation of LA tools in Higher Education HEIs in Latin America [
The Institutional dimension helps to identify the current state and needs regarding LA in the institution. It consists of 4 phases. Phase 1 is about performing an institutional diagnosis. This is achieved through the LALA canvas, a document that analyzes the current state of LA in the institution around six dimensions: a strategy for change, desired behaviors, internal capabilities, political context, in uential actors and measurement and evaluation plan. The aim of phase 2 is to understand the political context and institutional needs. This is done through a protocol for interviews with institutional leaders, professors and students about three dimensions: current state of LA adoption, the desired state of adoption of LA and challenges for LA adoption. Phase 3 helps to identify what is expected from the use of educational data. The instrument used is an online questionnaire for students and professors and analyses normative vs. predictive expectations about privacy and the use of educational data. Finally, phase 4 helps develop a change strategy through the LALA template, a document that analyses the same six dimensions in the LALA canvas. The di erence is that the latter is based on what is expected to occur in the institution as the result of adopting a LA tool.
The Technological dimension aims to obtain the system requirements,
identify the technical considerations for the implementation of a LA tool, and sets
guidelines for evaluation and testing. To obtain system requirements, an
instrument named Orchestration of Learning (ORLA) [
The ethical dimension aims at considering the ethical and privacy considerations to take into account when adopting a LA tool. The rst phase is to conduct a review of the literature to search for national and international regulations in order to make stakeholders aware of their existence and how they could be implemented in their HEIs. The second phase focuses on anticipating students' and professor's expectations by interviewing them and asking them to ll out surveys related to aspects such as being aware if the institution asks for permission to use their data. The questions related to this phase are part of the instruments of phases 2 and 3 of the institutional dimension. The last phase focuses on taking actions to ensure proper treatment and use of data at an ethical and privacy level. One of the suggested activities in this phase it to design consent forms.
Finally, the community dimension gives guidelines to join the LALA
community. The LALA Community is aimed to promote a long-term sustainable
cooperation, creating lasting relationships among its members, which contribute
to the replication of the results obtained by the LALA project [
In this section, we describe the case studies of two HEIs that applied the LALA Framework to adopt a LA tool, namely, a counseling system. We put particular emphasis on describing the application of the technological dimension since the execution of its activities was the most in uenced by each institution context. The di erent HEIs' contexts required to apply di erent approaches or techniques during the execution. 3.1
Universidad de Cuenca (U1) and Escuela Superior Politecnica del Litoral (U2) are Ecuadorian public HEIses. While the former o ers a wide variety of knowledge elds such as medicine, engineering, social sciences, the latter is only engineering oriented. It is also di erent in its population. U1 has approximately 17000 students, while U2 10000 students. However, one main di erence lies in the adoption of LA tools. U1 has no previous experience in implementing any LA tool, nor has policies or processes related to academic counseling. While U2 has already implemented an academic counseling system since 2014 and the process of academic counseling has already been institutionalized.
Both HEIs followed the guidelines of the LALA Framework to identify the current state and needs of LA in their institutions. Phase 1 about performing an institutional diagnosis, was executed through the use of the LALA canvas. The instrument was lled out by two decision-makers and eight researchers in total in U1 and U2. Phase 2 about understanding the political context and institutional needs, was accomplished through interviews with 19 institutional leaders and focus groups to 31 teachers and 27 students. Phase 3 about identifying what is expected from the use of educational data, was applied through surveys to 912 students and 166 teachers. Finally, phase 4 about developing a change strategy was executed through the LALA template. The instrument was lled out by the same team as in phase 1. After triangulating and analyzing the results with the involvement of 4 researchers in each HEI, the primary need for students was related to having quality feedback and timely support. This translated in the case of U1 in designing a counseling system that could support students in decision making. Regarding U2, the need came from students, teachers and leaders to reinforce the current counseling system with more data to make better decisions when advising students. 3.3
Based on the needs identi ed as a result of executing the institutional dimension, both HEIs considered that a counseling system would satisfy those needs. Therefore, the technological dimension aimed at designing, implementing a data visualization tool to facilitate the dialogue between a student and his/her academic counselor.
There exist di erent approaches for the development of data visualization tools (e.g., design thinking) that are applied depending on speci c project requirements or organizational policies. Independently of the applied approach, all of them include everyday activities (e.g., requirements elicitation, development, testing), existing other activities (or similar ones) speci c of each approach that de nes di erences among them. Next, we show how each HEI followed a di erent development approach while still using the guidelines provided by the LALA Framework. 3.3.1 Requirements De nition Requirements elicitation is a complex activity that is supported by a variety of techniques (e.g., brainstorming, focus groups, interviews, observation, prototyping, document analysis, questionnaires) that developers use according to the project context. HEI teams executed this activity once they completed the Institutional Dimension, which produced the identi cation of institutional needs and the main stakeholders. The main stakeholders' identi cation allowed U1 and U2 to direct their e orts in involving people who were going to act as project promoters|, facilitate access to resources|, or who were going to be critical requirement sources during elicitation.
The LALA Framework suggests using ORLA as the instrument for de ning
and extracting design requirements; however, U1 and U2 did not nd it suitable
for their contexts because the questions in ORLA are related to the design of LA
tools to support teachers at the classroom level [
Due to the lack of experience of U1 in the counseling process (and counseling
tool usage), this HEI executed a combination of focus groups, interviews, and
prototyping techniques for gathering requirements. The prototyping technique
is useful when developing human-computer interfaces, or when the stakeholders
do not know about available solutions [
Regarding U2, according to its context, it was just necessary to ask
stakeholders what other data was needed to make better decisions when advising students.
Techniques such as brainstorming or interviews were considered time-consuming
due to a large number of advisors (around 300). Hence, a questionnaire, an
efcient technique to gather requirements from multiple stakeholders quickly by
avoiding redundant and irrelevant data [
To re ne the initial requirements, U1 used focus groups and interviews, while
U2 used only interviews. Focus groups help to get feedback on prototypes and
to let stakeholders generate new ideas [
tion are both essential characteristics of visualizations. During the development and implementation phase, U1 and U2 generated high delity prototypes that allowed users: i) to interact (or simulate interaction) with the designed dashboards, and ii) to validate dashboards' interaction capabilities.
To simulate interactions, U1 used an online mockup creation tool. On the other hand, the experience in the development and implementation of counseling tools of U2 allowed them to implement an executable high delity prototype by using the technology to be used in the nal deployment.
To validate interactions, U1 executed focus groups in which a development team member used the mockup to simulate the analysis of students' academic performance, showing users how visualizations change in response to events (e.g., click, mouse over). On the other hand, U2 designed a validation protocol that included speci c tasks and expected execution times, to be carried out by users by using the executable prototype. To improve the dashboard design, U1 used the qualitative information obtained as a result of focus groups, whereas U2 used quantitative information from the validation protocol. After validation, both HEIs built a beta version of their dashboards that are being used in the piloting phase.
In the tool development and implementation phase, both U1 and U2 applied the LALA Framework \Guide of technical considerations for the development and implementation." However, U1 used it when building the beta version, while U2 used it when building the executable high delity prototype. This fact shows that the guidelines provided by LALA can be used according to the development process chosen by each institution. 3.3.3 Tool Evaluation and Testing The evaluation and testing of the developed dashboards are taking place during the piloting phase. LALA project members de ned a process for this phase, which includes the tests and evaluations suggested in the LALA Framework "Guide on considerations for the design of the procedure for evaluation and testing of the tool." This process includes the following activities: identifying the current state (baseline) about the counseling process/system in order to later make comparisons after the piloting/training takes place, planning and executing training, tool usage reporting, and evaluation. Where, as suggested by the guide, process activities take into account not only technological aspects (e.g., data quality, usability, performance) but also ethical considerations (e.g., informed consents).
In the case of U1, there is no baseline because there is no existing counseling process; therefore, performance or usability reports were not generated. Regarding the training process, it helped U1 to validate data quality of visualizations as well as identify and implement new requirements. Additionally, introducing the counseling process as well as the counseling tool as part of the academic activities is being delayed due to issues related to cultural changes and lack of policies. For instance, some teachers and administrative sta think it is going to represent an extra workload. Furthermore, counseling is perceived as a useless task since students are old enough to make their own academic-related decisions. While from some academic decision-makers' point of view, results need to be shown in order to create policies. The above issues make it clear the need for de ning policies that facilitate the adoption of LA by HEIs.
On the other hand, U2 was able to report the impact of introducing changes to its existing dashboard and had escalated the new dashboard at the institutional level (i.e., all program studies advisors are using the new version). However, implementing new requirements that emerged during training and actual use is not just adding new visualizations but changing existing ones, which means taking the risk of changing the previous system version source code. Furthermore, this requires strong evidence about the bene ts of implementing the new requirements in order to give con dence to decision-makers to authorize the modi cation of the previous system. 3.4
Due to time constraints, both HEIs only applied the second and third phases proposed in this dimension; anticipate professors' and students' expectations and adapting ethical and privacy considerations for the creation of the institutional framework on ethics and data privacy, respectively. Results from executing the second phase showed that in both HEIs, there are no institutional policies regarding data privacy and protection; however, some teachers, students and institutional leaders think those policies exist. Finally, in the third phase, to ensure good treatment and use of data at ethical and privacy level, informed consents forms were created. The stakeholders involved in the interviews, focus groups and surveys were asked to sign these forms. 3.5
Since both HEIs are founding members of the LALA project, the community dimension was not used. Nevertheless, if both HEIs had used this dimension, they would have subscribed to the LALA community. This would have allowed them the opportunity to receive newsletters about dissemination events where the experience of designing and implementing a LA tool would have been shared with other HEIs. 4
This paper presents the application of the LALA Framework for the development and adoption of a LA tool (i.e., an academic counseling dashboard) in two Latin-American HEIs with di erent LA contexts. For instance, U1 has no previous experience using LA tools or supporting students trough academic counseling sessions, whereas U2 has experience with both. When executing the dimensions proposed by the framework, the execution of the institutional and ethical dimensions showed no di erences between both HEIs. Nevertheless, changes appeared while executing the technological dimension. For instance, when gathering requirements, U1 used a prototyping technique that helped to obtain requirements from actors without previous experience with LA tools, while U2 used a survey to collect additional data o visualizations needs from actors requiring improvements in the current counseling system.
Regardless of the di erent LA context of the HEIs involved in this study, these institutions were able to apply the LALA Framework, which points out its
exibility and openness. However, during the framework application, both HEIs had to apply techniques not proposed by it (e.g., requirement elicitation techniques). This because the proposed techniques were not appropriate for the HEIs' context. Therefore, in order to enrich the framework, the following suggestions are proposed. First, although the Institutional Dimension allows HEIs to identify their current state and the desired state concerning policies and strategies for the incorporation of LA tools, this context should be categorized according to the level of maturity in using LA (e.g., HEIs without LA experience, HEIs with LA experience). Furthermore, the framework should also suggest activities according to this maturity level. For example, although the framework allows HEIs to execute a dimension in any order, it would be better if practitioners know under what maturity level to execute a dimension. Second, the framework should provide more speci c guidance to practitioners about the tasks to perform or techniques to apply during the execution of the proposed activities, and it should be connected to the HEI's maturity level identi ed in the institutional dimension. For example, during the evaluation phase in the technological dimension, is not enough to suggest a set of tests. Practitioners need to know the task in which those test must be applied. Another example occurs when gathering the tool's requirements. The framework should provide practitioners di erent techniques to gather system requirements. One limitation of this work is that it was applied only in a Higher Education setting in a face to face modality with the members of the LALA project. Thus, there is no information of the application of this framework in other contexts. For instance, a high school in on online modality. The aforementioned limitations help us to de ne future work: the applicability of the LALA Framework in other contexts (e.g online education, in other latin american countries, with a di erent educational levels), and with a di erent tool besides dashboards.
Work funded by the LALA project (grant no. 586120-EPP-1-2017-1-ES-EPPKA2CBHE-JP). This project has been funded with support from the European Commission.This publication re ects only the views of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.