The development of Learning Analytics (LA) capabilities in a Higher Education institution is challenging. On the one hand, the institution requires of a technological infrastructure for adapting and/or developing LA services. On the other hand, the institution also needs of an organizational structure for designing and implementing new processes for assuring the adoption of these services. There are two different approaches for developing the necessary infrastructure and organizational structure. One consists on following a top-down process, in which the leadership of the LA initiative is mainly driven by institutional managers, who provide the necessary means. Another is bottom-up, where the initiatives are led by ground-level teaching staff without involving institutional managers. This article presents both approaches through two LA initiatives of Engineering School at the Pontificia Universidad Católica de Chile (UCEngineering). We show how these two initiatives emerged and integrated into existing academic processes to improve teaching and learning at an institutional level. The infrastructure and organizational structure resulting from each initiative is presented, as well as the lessons learned. This paper aims at serving as an example for other universities in Latin America interested on developing and incorporating LA capabilities.
Incorporating Learning Analytics (LA) at an institutional level is challenging, especially in Higher Education institutions in Latin America (Latam). Although some efforts have been made in this region to incorporate LA services [1], these initiatives are still immature for moving from experimentation to full institutional integration [2]. In Latam, institutions face two main challenges. On the one hand, they often lack the technological infrastructure for developing and/or adapting LA services [1]. On the other hand, they also lack the organizational structures to integrate the use of educational data into existing institutional processes to inform strategic planning or guiding teaching and learning practices [2].
Regarding the technological infrastructure required for building ne
Regarding the organizational structure, some researchers point out that an enabling
leadership is key to overcome the challenges inherent to the institutional adoption of
Learning Analytics [5]. However, finding the appropriate balance between different
types of leadership for fostering LA adoption is not easy to over
Since experiences on LA capacity building in Latam are scarce, institutions in this
region are forced to take as a reference experiences and cases conducted world-wide,
which usually do not face problems and challenges of similar nature. With this paper,
we aimed at presenting how LA capabilities have been developed at a Latam institution
—specifically, at Engineering School at the Pontificia Universidad Católica de Chile
(UC-Engineering), one of the better ranked institut
This section describes a LA initiative that emerged as a bottom-up process led by a group of researchers at UC-Engineering. This initiative aimed at proposing a LA tool capable of supporting students’ self-regulatory strategies in MOOCs, in order to improve their learning experience and help them better achieve their objectives. This tool is called NoteMyProgress and it has been used in all MOOC courses produced by UCEngineering, besides supporting a blended learning course.
2.1
In 2015, UC-Engineering launched the UC Online Engineering initiative. This initiative aims to create open online courses (MOOCs) for Coursera and digital content to transform traditional teaching-learning practices. Since this initiative began, UC Engineering has 17 MOOCs with more than 400,000 registered students, and several projects have reused MOOCs as a complement to the curriculum courses [6], [7].
As a result of the initiative, UC Engineering began to collect a large amount of data on students from all over the world, from demographics to their interaction with online resources. This large volume of data was seen at the institution as an opportunity to launch research initiatives around LA, aimed at improving student experience in these new digital learning environments.
In this context, a group of UC-Engineering researchers proposed a project to support student study strategies in digital learning environments, in order to improve MOOC learners’ engagement and performance. This project was funded by the National Commission of Science and Technology of Chile (CONICYT) between 2015 and 2018. One of the results of the project was the tool NoteMyProgress (NMP) [8], a LA tool for supporting students’ self-regulation strategies in online environments in an automatic and personalized way. Through interactive visualizations, it provides actionable aggregated information about student activity in the online course and its interaction with its contents. The objective of this tool was to promote students’ reflection on their learning strategies, motivating informed decision-making to improve their performance. 2.2
The development of NMP was conducted by a software company specialized in
visualizations, and coordinated by one of the researches of the team involved in the
CONICYT project. For its development, the researchers followed a design-based
research approach called the Interactive Learning Design Framework (ILD)
Therefore, the infrastructure involved in this initiative was: (1) an external local server to host the first version ; (2) a web server at UC-Engineering for installing the second version, and (3) an account in Google Apps for uploading the last version of the tool and make it available to final users.
2.3
Different stakeholders were involved in the process. • Three researchers participated in the literature review process for defining the requirements of the tool and the final analysis for reporting the impact of the tool. • Two developers from an external company participated in the development of the tool in collaboration with the research team; • Seven teachers and teaching assistants took part of the evaluation process; • The Director of the Engineering Education coordinated the relationship between the research team and the teaching staff involved in the evaluation process; • The Ethical Committee of the University, who validated the consent forms facilitated to the users that downloaded and installed the tool, as well as the agreement for using the data for the analysis.
Since this process emerged from a group of researchers, the first phase of the LA initiative involved mostly grounded level stakeholders, while the Director of Engineering Education, a higher-level stakeholder, was involved when scaling up the usage of the tool. 2.4
During the development and deployment process of NMP tool, the research team encountered two main challenges, especially in the deployment and piloting of the tool. Regarding the technological infrastructure, the implementation of the tool at a university level required of the coordination of the technical team of the university via the Engineering Education leader. This step required meetings for convincing intermediate managers that piloting an innovative initiative could lead to potential benefits.
Regarding the organizational structure, researchers involved in the project were already familiar with the LA service potential. However, making senior managers and teaching staff aware of how the NMP tool could impact students’ performance required empirical evidence the potential of the tool. In fact, the research team continues to work on new tool versions, in order to facilitate their incorporation into institutional processes, such as undergraduate blended teaching. 3
This section describes a LA initiative that emerged as a top-down process leaded by the Office for Undergraduate Studies and the Engineering Education Unit of the UCEngineering School. The aim of this initiative was to propose a LA tool for supporting a continuous improvement process at five programs in UC-Engineering, in order to comply with one criterion of an international accreditation process [10]. This tool is called Curriculum Analytics (CA) tool, and it has been used by 124 teaching staff from in 96 course sections.
3.1
In 2007 and 2011, after an institutional strategic decision, the UC-Engineering decided to comply their programs to the quality standards of the Accreditation Board of Education Technology (ABET). Five out of the eleven engineering degrees were accredited by ABET during this period. In 2015, ABET mandated a continuous improvement process to renew the accreditation of these programs, providing evidences of how competency attainment data has been used to improve curriculum and teaching practices at a program-level.
To facilitate the accreditation process, the Office for Undergraduate Studies and the Engineering Education Unit designed a process for collecting data about student competency attainment. This process consisted on supporting teaching staff as they determine assessment plans to measure competency attainment at a course-level, and analyzing the results of course assessments so as to discuss the students’ competence attainment at a program meeting [10]. As a starting point, the assessment plans and the competency attainment results were stored in Dropbox folders. However, the collection of assessment evidence become overwhelming forteaching staff, and the analysis of the data collected was not readily available for program meetings. To alleviate this process, the Director of the Undergraduate Studies Office decided to invest on a LA tool. This tool was called CA tool, and its design aimed at facilitating the storage of assessment evidence, such assessment plans and competency attainment results, besides providing visualizations of competency attainment for program meetings.
3.2
The CA tool was also developed by a Chilean Company following the Interactive
Learning Design Framework (ILD)
The infrastructure involved in this initiative was: (1) a web-based application of other university that helped to have a preliminary idea of how a Curriculum Analytics tool could help a continuous improvement process, and (2) an internal university server for deploying the piloting tests tool and the final version of the tool.
3.3
Different stakeholders from different teams of the UC-Engineering School were involved in the tool development and evaluation process.
• The Director of the Office for Undergraduate Studies • One manager from the Engineering Education Unit who collected and analyzed data to identify requirements and evaluate how the tool was used by teaching staff during the continuous improvement process, and one academic who collaborated with this manager during the definition of the requirements needed for adapting the tool to the final stakeholders, • Two project managers of an external company who coordinated the development process of the Continuous Improvement Platform; • 125 teaching staff participated in the evaluation process;
Since this process emerged from the Director of the Office for Undergraduate Studies, and it was led by a manager by the Engineering Education Unit, the stakeholders involved in the first phases of the LA initiative were high level, while more groundedlevel stakeholders participated in the design and evaluation process of the initiative. 3.4
During the development and deployment process of the Curriculum Analytics tool main challenges were encountered. Regarding technological infrastructure, the most challenging aspect was to integrate data from different sources into the CA tool. In order to obtain automated reports about students’ competency attainment, the CA tool had to integrate course partial grading with course enrolment, in addition to linking manual parameters that indicated what partial grades indicated learning results for a specific competency. During its implementation, managers had to conduct several validation exercises to compare if the data of the report reflected the competency attainment results that were supposed to be visualized during program meetings.
Regarding organizational structure, the Engineering Education and the office of Undergraduate Studies had to find the mechanisms for integrating the use of the LA tool as part of already existing processes to avoid teaching workload. Since the implementation of the CA tool responded to a top-down initiative, teaching staff were reticent to conduct additional tasks to the ones they already undertake to assess competency attainment and learning results in their courses. For this, managers of the CA unit trained teaching assistants about how to use the CA tool, so they could help teaching staff to upload the required evidence for the international accreditation process. 4
This paper summarizes two of the LA initiatives that were developed at UC Engineering School: (1) NoteMyProgress, an initiative to support students’ self-regulation in MOOCs; and (2) the Curriculum Analytics tool, an initiative to support continuous improvement processes at a program-level based on competency attainment evidence. Both initiatives were successfully deployed, but some challenges were encountered in the way in each of the projects.
The first initiative followed a bottom up approach, where grounded-level researchers were involved in the design and evaluation of the LA tool. The main challenges identified were two: (1) Regarding technical aspects: the lack of intermediate managers to support the incorporation of the tool into an existing institutional process; and (2) Regarding organizational aspects: the lack of awareness of the potential of this tool to support online learners due to the lack of involvement of managers during early phases. The second initiative followed a top-down approach that emerged from senior managers. The main challenges identified were two: (1) Regarding technical aspects, the difficulties to integrate educational data from different sources into an analytical tool developed by an external company, and (2) Regarding organizational aspects, the resistance of teaching staff to undertake additional tools, and the need to involve teaching assistants to support them as they collect evidence for competency attainment.
From the two approaches, the following lessons learned were extracted for future initiatives. First, it is important to combine top-down and bottom-up approaches to facilitate the involvement of varied stakeholders during the design and the implementation of an LA tools. In the bottom-up LA initiative, ground-level staff played a key role during the design process, providing feedback to tool developers regarding their needs and preferences. Whereas, in the top-down initiative, senior managers play a key role during the incorporation of the tool into an existing academic process, managing resources and training for engaging teaching staff. Second, it is important to anticipate the need for servers and data warehouses in order to integrate data from different sources. In the bottom-up LA initiative, UC-Engineering had to move their tool from a company sever to web server managed by the university. In the top-down LA initiative, data integration was crucial, besides the successive validation of automated reports. Third and final, it is important to spread the potential that LA tools could have for addressing institutional needs, besides building the expertise required to organize, clean and manage educational data responsible. We expect that these and other initiatives motivate other universities to take paths as the one taken by UC-Engineering. 5
Work funded by the LALA project (grant no. 586120-EPP-1-2017-1-ES-EPPKA2CBHE-JP). This project has been funded with support from the European Commission. This publication reflects only the views of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein. 6 [1] [2] [3]