Within higher education, different types of analytics have been implemented to tackle painstaking work required to improve learning results at different levels [1], [2]. One of these analytical approaches is Curriculum Analytics (CA),defined as the collection, analysis and visualization of administrative curricular data-such as course enrolment and student grades -to inform and support decision making at a program level [3]. In specific educational contexts, CA has been proposed as a good solution for lightening task workload required to identify courses where the improvement of learning results is crucial to the program success [4]. This workload is particularly high in competencybased curriculums because each course emphasizes specific core competencies within an academic plan [5]. Traditionally, the analysis of this type of curriculum requires to assess the alignment between program and course level competencies, determine whether both type of competencies are attained and formulate action plans to improve teaching and learning [4]. The emergence of CA techniques opens new possibilities to perform these tasks in a timely and cost-effective manner [4]. However, the CA promise of supporting curriculum analysis is far from fulfilled. Several tools have been proposed to identify gateway courses in a curriculum and improve its outcomes [4], but managers and teaching staff still perceive that they lack systematic information for course improvement, such as students' academic results regarding taken courses and core competency attainment [5]. Research about CA adoption is still in an early stage, so most of the tools developed propose solutions to tackle institutional needs are not necessarily related to any existing improvement process [6]. In some cases, using technology to help stakeholders to visualize institutional data available is a good means of supporting data-driven decision-making and promoting transparency in certain educational processes. However, prior work suggests that, in order to leverage the potential of CA tools, an institution needs to be ready to address data-driven changes [6], having already implemented processes to examine what is and is not working at different levels [1].
To illustrate how the use of CA tools could systematically support continuous curriculum improvement for an extended period of time, we present an instrumental case study about a continuous improvement process implemented in an engineering school in Latin America (UC-Engineering). This process was implemented between 2015 and 2017 to comply with the North American Accreditation Board for Engineering and Technology (ABET), and in mid-2016, a CA tool was implemented to support teaching staff. To evaluate the teaching staff's usage of this tool in five competency-based programs, we collected and analysed data before and after tool implementation. The following data collection techniques were used: 1) analysis of documentary evidence reported for the continuous improvement process by 124 teaching staff in 96 course sections, and 2) teaching staff questionnaires applied to 25 out of 63 teaching assistants who interacted with the CA tool. These two types of data sources were triangulated to evaluate tool usage from the teaching staff 's perspective, exploring its implications in terms of tool usefulness and ease-of-use.
Over the past decade, researchers have developed CA tools to address multiple educational challenges from the perspective of different stakeholders. Table 1 summarizes some of the tools that have been documented in recent conference proceedings and journal articles describing the institution in which they were developed, and their distinctive features for different users. Concerning students, tools 1 and 2 were developed with the objective of changing their approach to study, using visualizations of their performance to motivate them to adopt help-seeking behaviours. Regarding teaching staff and managers, tools 3-10 were developed with the objective of providing information to identify students who are facing difficulties in their studies, expecting staff to reach out to their students and provide guidance. Finally, tools 11-14 were developed for students or staff, aiming to help them to identify crucial courses in a curriculum, and anticipate the impact of course-level improvements in competency attainment at a program level.
Most of the tools presented in Table 1 do not provide information about the methodology used to evaluate its impact on any existing institutional processes. By methodology, we mean experimental approaches to determine whether the tool is effectively fulfilling its objective [7]. For instance, little is known about how students perceived the tools developed to promote help-seeking behaviours (such as tool 1) [8]. Regarding tools that aim to identify students at risk (such as tool 9), there is no clear relationship between the use of the tool and student outcomes yet [8]. And, concerning the tools for monitoring course-level improvement and competency attainment, researchers have just started to evaluate their adoption by collecting information about user perceptions [5], without necessarily reporting data about its actual use to drive improvements in program design or academic program delivery [9].
Building upon this latest idea, we present a case study that evaluates teaching staff usage of a CA tool in the context of a continuous improvement process that has already been installed at an institutional level. Our aim is not only to present the results of evaluating the tool from users' perspective, but also to analyse how it was used to support an existing process for curriculum improvement. 3 Methods
According to Zelkowitz [7], instrumental case studies are useful to determine if a technological tool makes it easier to produce something compared to prior scenarios. Since the objective of this study is to evaluate usage of a CA tool throughout a continuous improvement process, we select instrumental case study as the most appropriate methodology. The case study took place at UC-Engineering between 2015 and 2017. In this case, we specifically evaluated whether the CA tool supported teaching staff efforts to collect documentary evidence to account for competency attainment.
In the early 2000s, UC-Engineering decided to accredit five academic programs by ABET, which concentrated 35% of its undergraduate enrolment (1,500 students out of a total of 4,000): 1) Civil Engineering, 2) Electrical Engineering, 3) Computer Engineering, 4) Mechanical Engineering, and 5) Chemical Engineering. In 2015, the managers at the Office for Undergraduate Studies and the Engineering Education Unit designed a continuous improvement process to be implemented for the renewal of the ABET accreditation of these five programs. The continuous improvement process was organized in six semesters between the first semester of 2015 and the second semester of 2017. Every semester, teaching staff were required to undertake the tasks described in Fig. 1. Before the semester started, they had to participate in a workshop to revise their course syllabuses to make sure the competencies declared at a course level were aligned to the ones at assigned program level. At the beginning of the semester, they had to report an assessment plan to account for competency attainment at their courses (two competencies per course in most cases). Once the semester finished, they had to
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) report competency assessment results, which were transformed into percentages of competency attainment to be revised in an end-of-semester curriculum meeting (curriculum discussions). Additionally, they might have been required to submit a sample of assessment methods used to assess the competencies assigned to inform curriculum decision-making (if needed).
At the end of the continuous improvement process, 104 assessment plans (http://bit.ly/2SYxWxc) and spreadsheets of competency assessment results (http://bit.ly/2VOdUKx) were expected to be reported by 124 teaching staff from 96 course sections, analysing the attainment of the 11 competencies proposed by ABET Criterion 3 (http://bit.ly/2SeVzRj). By the end of 2015, 38 assessment plans and competency assessment results were already collected from 29 course sections for an interim report to be sent to ABET in June 2016. These documents were attached and sent in e-mails by teaching staff, and then uploaded to Dropbox folders by UC-Engineering managers. For curriculum discussions, analysts of the Engineering Education Unit had to transform graded assessment results into percentages of competency attainment, so this information was only available at the end of the semester. By automating this analysis, program chairs and teaching staff would have access to competency attainment results whenever they needed them for their decision-making. This motivated UC-Engineering managers to implement a CA tool developed jointly by the University of Sydney and U-Planner (https://www.u-planner.com/en-us/home), a Latin American company that provides technological solutions and research services to higher education institutions. This tool was originally developed to facilitate the alignment between competencies in a graduate profile and the teaching and assessment methods of the different courses within a program (see Fig. 2), and to collect documentary evidence of competency assessment at a
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) course level as attachments. In order to include an automated visualization of competency attainment in the CA tool, U-Planner had to implement an ETL process to integrate course grading results from the LMS, and then create a report that transformed these results into percentages of competency attainment LMS (see Fig. 3). After validating the report for 12 courses during the first semester of 2016, the CA tool was implemented for supporting the continuous improvement process since the second semester of 2016. At the end of 2017, the CA tool was not only used to collect all the documentary evidence required for the accreditation purposes (syllabuses, course descriptions, and assessment plans), but also to visualize competency attainment results in each course for different academic periods. The percentages of competency attainment were calculated by dividing each student grade into the maximum score of an assessment method, and then multiplying this result by 100. Subsequently, each percentage of competency attainment was classified into four performance levels according to different thresholds established by teaching staff. These performance levels were: 1) unsatisfactory, 2) developing, 3) satisfactory, and 4) exemplary (see Fig. 3). Fig. 2. Screenshot of the UC-Engineering CA tool for supporting continuous improvement. Through this tool, teaching staff performed the tasks required for the continuous improvement process since the first semester of 2016.
Fig. 4 depicts the period considered for the case study, indicating the period were the tool was implemented. The tool aimed at supporting the continuous curriculum process by means of facilitating the following activities (see Fig. 1 with the semester tasks):
(1) Filling a course description form to describe broadly the teaching and assessment methods.
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
(2) Indicating the relationship between competencies at both program and course level (Fig. 2). (3) Listing performance indicators for competencies at a program level that could be assessed at a course level. (4) Aligning performance indicators with graded assessment methods at a course level. ( 5) Generating automated reports on percentages of competency attainment (Fig. 3). This functionality is integrated with the institutional LMS to automatically capture the students' grades for align graded assessment methods. ( 6) Uploading the following documentary evidence as attachments: Course syllabus; Assessment plans; Competency assessment results; Sample of assessment methods. Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
Between 2015 and 2017, 124 members of the teaching staff participated in the continuous improvement process implemented at UC-Engineering. These 124-teaching staff include 61 teachers (44 faculty members and 17 part-time instructors) who reported documentary evidence of competency attainment in 96 course sections, and 63 teaching assistants who supported teachers in outcome assessment tasks (see Table 2). The evaluation process was organized into two phases. The first phase consisted in evaluating how the CA tool was used to facilitate the collection of documentary evidence for analysing competency attainment in curriculum discussions. The documentary evidence consisted of any documentation that was reported to account for competency attainment at a course level, such as: course syllabus, course description, competency assessment results, and samples of assessment methods (see Fig. 1). In order to compare the number and the type of documentary evidence generated before and after the CA tool was implemented, three researchers used a coding scheme to classify the evidence reported by each teaching staff member in each course section. This scheme was developed based on a bottom-up coding approach, and each category was defined by examining the files uploaded in both Dropbox and the CA tool.).From this bottom-up approach, four categories emerged, so the three researchers used these categories to assigned scores from 0 to 1 to account for the type of documentary evidence reported every semester (see Table 3): 1) reported assessment plans, 2) reported a sample of assessment methods, 3) reported competency attainment results, 4) reported course syllabus, 5) included a course description, and 6) used the automated report of the CA tool. Then, the scores assigned ranged from 0 to 6 in each course section, in which a score equal to 0 indicates a minimum amount and variety of evidence reported (and a score equal to indicates 6 a maximum amount and variety of evidence).
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) Table 3. Coding Scheme to analyse documentary reported throughout the continuous improvement process implemented at UC-Engineering (before and after tool implementation)
Category description Reported assessment plans (0)
The teaching staff did not report an assessment plan informing how course assessment methods were used to measure competency attainment. Reported assessment plans (1) The teaching staff reported an assessment plan to inform how course assessment methods were used to measure competency attainment at a course section level. Reported a sample of assessment methods (0)
The teaching staff did not report a sample of assessment methods to account for different levels of competency attainment among different students. Reported a sample of assessment methods (1) The teaching staff reported a sample of assessment methods to account for different levels of competency attainment, such as developing or satisfactory.
The teaching staff did not report competency attainment results at a course section level.
The course reported competency attainment results based on graded assessments at a course level. Reported course syllabus (0)
The teaching staff did not report the course syllabus to complement the evidence items for the accreditation process. Reported course syllabus (1) The teaching staff reported the course syllabus as a complement to other evidence items reported for the accreditation process. Included a course description (0) The teaching staff did not include a course description among the evidence items uploaded in the CA tool. Included a course description (1) The teaching staff included a course description among the evidence items uploaded in the CA tool. Reported percentages of competency attainment (0)
The teaching staff did not reported percentages of competency attainment.
The teaching staff reported percentages of competency attainment at a program level.
The second phase consisted in exploring the perceived usefulness and ease-ofuse of the CA tool from the viewpoint of its users. For this purpose, researchers have recently proposed instruments to evaluate the impact of analytical tools on educational practices [2], these instruments have still been implemented at a course-level in controlled environments. Therefore, we decided to develop a paper-based questionnaire based on the prior work of Ali et al [12], considering that his objective was also exploring teaching staff's perspectives in a real-life context. Our questionnaire consisted of a closed-ended and an open-ended question section (http://bit.ly/2Jh3VVG). The closedended section consisted of a 5-point Likert scale to determine the level of staff's agreement on different items related to perceived usefulness and perceived ease-of-use, while the open-ended section included the following questions to understand usability and ease-of-use implications from an exploratory approach:
• What use would you give to the CA tool after interacting with it?
• What kind of information would you expect from this tool?
• What do you think the CA tool lacks in terms of information and functionality? In order to gather information from the user experience once the CA tool implementation process was well advanced, we applied the questionnaire during a workshop for teaching assistants held at the beginning of the second semester of 2017 (see semester tasks in Fig. 1). A total number of 25 teaching assistants who attended this workshop Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) responded to the questionnaire voluntarily (representing 25 out of the 63 teaching assistants who supported outcome assessment tasks since the second semester of 2016). After transcribing their responses, we estimated the percentage of respondents who agreed with these items by counting the number of respondents whose scores were equal or higher than 4, and then dividing them by the number of total respondents (see Fig. 6). Fig. 5. Percentage of teaching assistants who agreed with the questionnaire items related to the CA tool usefulness and ease of use (N=25, see questionnaire in: http://bit.ly/2Jh3VVG).
Once all the evidence items and questionnaire were analysed, we triangulated questionnaire and document analysis results from phase 1 (see Fig. 5 and 6). This process consisted of contrasting the amount and variety of documentary reported by teaching staff before and after tool implementation, in addition to analysing the perspectives of questionnaire respondents on tool usage for curriculum analysis based on evidence of competency attainment.
The main findings of the case study are summarized in Table 5. Firstly, the CA tool helped teaching staff collect a greater number and variety of evidence, providing visualizations of competency attainment at a program level (Finding 1 in Table 5). Document analysis results show that the number and the variety of evidence reported per course section increased from two to five after the CA tool was implemented (see Fig. 2). In most cases, these three additional items included course syllabuses, course descriptions, and the percentages of competency attainment. Before the CA tool was implemented, teaching staff delegated the transformation of graded assessment results into percentages of competency attainment to professionals in the Engineering Education Unit. However, once the CA tool was implemented, 89% of the course sections relied on the automated report provided by the tool to account for competency
The CA tool allows me to obtain more information about courses than…
The CA tool allows me to obtain information easily.
The purpose of using CA tool is clear and understandable.
It is easy to learn how to use the CA tool.
In general, the CA tool seems useful for institutional management.
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
attainment at a course level. This effort to collect a greater number and variety of evidence is not due to greater administrative pressure, since UC-Engineering managers presented a report to ABET in the first semester of 2016, and all subsequent work was done solely with the motivation to continuously improve the curriculum. Additionally, the results of the questionnaire show that 92% of respondents agreed with the item 'In general, the CA tool seems useful for institutional management' (see Fig. 4). In the open-ended questions, teaching assistants mentioned that the CA tool facilitated the use of evidence to account for the implementation of a competency-based curriculum throughout the accreditation process. They also mentioned its potential use to provide students with information about course methods and its alignment with competencies from the graduate profile.
Secondly, teaching staff identified usability and functionality issues that affect the use of evidence to inform course redesign and students' understanding of performance (Finding 2 in Table 5). The results of teaching staff questionnaires show that 56% agreed with the item 'The CA tool allows me to obtain more information about courses than other tools (such as the institutional LMS and a web application to search for course information).' In the open-ended sections, respondents claimed that the CA tool had usability issues. For example, respondents indicated that the tool views had too many tabs and fields, which hindered loading of information. Respondents also mentioned functionality issues, such as the lack of feedback about the quality of course information uploaded, and documentary evidence reported. Also, the tool lacks actionable information on student performance to provide timely and specific feedback throughout the semester.
Furthermore, questionnaire respondents indicated that, although they were able to upload the documentary evidence to account competency-attainment, they perceived the CA tool site was not intuitive enough. The fields to be completed, as well as the files to be uploaded were not clearly explained. This may partly explain why only 16% of course sections reported samples of assessment tools. These results suggest that the CA tool could be improved by including 'help messages' or indications related to the process within the platform. Table 4. Main findings about the usefulness and the ease-of-use of the CA tool Findings Document analysis and questionnaire results Supporting data 1. The CA tool helped teaching staff collect a greater number and variety of evidence, providing visualizations of competency attainment at a program level.
Course sections reported 3 additional evidence items for the accreditation process after the CA tool was implemented. Document analysis results (Fig. 6) 89% of course sections used the automated reporting feature of the CA tool to account for competency attainment at a course level (40 out of 45).
Screenshot of automated report of competency attainment (Fig. 3) 92% of teaching staff agreed with the questionnaire item 'In general, the CA tool seems useful for institutional management' (23 out of 25).
Teaching staff questionnaire (Fig. 5)
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) Findings Document analysis and questionnaire results Supporting data 2. Teaching staff identified usability and functionality issues in the CA tool that affect the use of evidence to inform course redesign and students' understanding of performance. 56% of teaching staff agree with the questionnaire item 'The CA tool allows me to obtain more information about courses than other tools.' (14 out of 25)
Teaching staff questionnaire (see Fig. 5) 16% of course sections reported samples of assessment methods after the CA tool was implemented (7 out of 45). Document analysis results (Fig. 6) Fig. 6. Average number of evidence items per course submitted each semester.
This case study showed the results of the usage of a CA tool in a long-term continuous improvement process. The main finding shows that the use of the CA tool helped teaching staff collect a greater number and variety of documentary evidence of competency attainment at a program level, allowing staff to upload documents that were previously shared or stored somewhere else, such as course syllabus. Furthermore, the CA tool provided staff with a visualization of competency attainment during the whole continuous improvement, information that was previously shown and discussed only in curriculum meetings at the end of the semester. This finding not only reflects that CA tools could be a good solution to reduce task workload for curriculum analysis [2], but also that this type of solutions provides visualizations of student performance that is typically hidden in institutional processes. This is particularly valuable for competency-based curriculums because it provides an overview of the competencies attained by the students at a program level. Teaching staff usually face difficulties in competency assessment because they tend to be abstract and complicated at a course level, while managers deal with the complexity of understanding learning results in a hierarchical structure of courses [3]. With this new tool, managers and teaching staff could have more actionable information for making better decisions to reinforce the required competencies at a program level.
Although this work has illustrated that a CA tool can support curriculum analysis in the context of an accreditation process, there are still needs of teaching staff that have not been met by the tool under study. Finding 2 in Table 5 indicates that there are usability and functionality issues that prevent teaching staff from using the tool to obtain information to redesign courses and to understand students' performance at an individual level. These are staff needs that have already motivated the development of tools [8]. In order to solve these issues, we could specify redesign requirements for the CA tool by using information we have already collected to evaluate usage. Thus, lessons learned from this case study imply that institutions could adopt and adapt existing CA tools to support curriculum analysis, investing not only in tool development and redesign, but also in capacity building for evidence-informed continuous improvement [1].
Along these lines, one of the main contributions of this paper is that it evaluates the use of a CA tool in an existing institutional process for an extended period of time, going beyond current evaluation strategies that mostly employ self-reported data without relying on a technology validation methodology [5]. The document analysis considered evidence items of the whole 3-year period in which the continuous improvement process was implemented. This long-term period assured collecting enough information to determine whether the CA tool facilitated teaching staff efforts to collect evidence of competency attainment before and after its implementation [7].
Yet, this study has its limitations. Questionnaire results only represented a small sample of teaching staff members who interacted with the CA tool. To understand the contribution of the CA tool for a larger group of teachers, future work would have to explore the type of information most staff need to inform course or curriculum redesign. We anticipate this work might require monitoring learning results for an extended period of time as we did it in this case study [5], so we can expand the current knowledge on the impact of CA tools on curriculum improvement even further.
This case study illustrates the usage of a CA tool to support a continuous improvement process for a competency-based curriculum in a university setting. Findings show that CA tools support curriculum analysis when they are implemented to help teaching staff to cope with tasks they are already performing for an existing institutional process, such as course planning and competency assessment. In this study, the CA tool not only facilitated the collection of documentary evidence to account for competency attainment at a program level, but also provided visualizations of competency attainment results that are usually not available for staff. Despite staff feeling comfortable when using the tool to upload evidence of competency attainment, usability and functionalities issues should still be improved in future versions of this tool. Future work on CA adoption should not only focus on tool development, but also on the evaluation of its impact in the formulation of curriculum improvement actions.
This work was funded by the EU LALA project (grant no. 586120-EPP-1-2017-1-ES-EPPKA2-CBHE-JP). The authors would like to thank Camila Aguirre from U-Planner for collaborating with this study, and the reviewers for their constructive suggestions.