Curriculum analytics (CA) consists of using analytical tools to collect and analyse educational data, such as program structure and course grading, to improve curriculum development and program quality. This paper presents an instrumental case study to illustrate the usage of a CA tool to help teaching staff collect evidence of competency attainment in an engineering school in Latin America. The CA tool was implemented during a 3-year continuous improvement process, in the context of an international accreditation. We collected and analysed data before and after tool implementation to evaluate its use by 124 teaching staff from 96 course sections. Data collection techniques included: analysis of documentary evidence collected for the continuous improvement process and teaching staff questionnaires. Findings show that the tool supported staff tasks related to the assessment of competency attainment at a program level. However, usability and functionality issues would have to be addressed to also support course redesign, providing actionable information about students' performance at an individual level. Lessons learned imply that institutions could adapt and adopt existing CA tools to support curriculum analysis by not only investing in tool development, but also in capacity building for its use for continuous improvement processes.
Within higher education, different types of analytics have been implemented to tackle painstaking work required to improve learning results at different levels[1], [2]. One of these analytical approaches is Curriculum Analytics (CA),defined as the collection, analysis and visualization of administrative curricular data— such as course enrolment and student grades —to inform and support decision making at a program level [3]. In specific educational contexts, CA has been proposed as a good solution for lightening task workload required to identify courses where the improvement of learning results is crucial to the program success [4]. This workload is particularly high in competencybased curriculums because each course emphasizes specific core competencies within an academic plan [5]. Traditionally, the analysis of this type of curriculum requires to assess the alignment between program and course level competencies, determine whether both type of competencies are attained and formulate action plans to improve teaching and learning [4]. The emergence of CA techniques opens new possibilities to perform these tasks in a timely and cost-effective manner [4].
However, the CA promise of supporting curriculum analysis is far from fulfilled. Several tools have been proposed to identify gateway courses in a curriculum and improve its outcomes [4], but managers and teaching staff still perceive that they lack systematic information for course improvement, such as students’ academic results regarding taken courses and core competency attainment [5]. Research about CA adoption is still in an early stage, so most of the tools developed propose solutions to tackle institutional needs are not necessarily related to any existing improvement process [6]. In some cases, using technology to help stakeholders to visualize institutional data available is a good means of supporting data-driven decision-making and promoting transparency in certain educational processes. However, prior work suggests that, in order to leverage the potential of CA tools, an institution needs to be ready to address data-driven changes [6], having already implemented processes to examine what is and is not working at different levels [1].
To illustrate how the use of CA tools could systematically support continuous
curriculum improvement for an extended period of time, we present an instrumental case
study about a continuous improvement process implemented in an engineering school
in Latin America (UC-Engineering). This process was implemented between 2015 and
2017 to comply with the North American Accreditation Board for Engineering and
Tech
Over the past decade, researchers have developed CA tools to address multiple educational challenges from the perspective of different stakeholders. Table 1 summarizes some of the tools that have been documented in recent conference proceedings and journal articles describing the institution in which they were developed, and their distinctive features for different users. Concerning students, tools 1 and 2 were developed with the objective of changing their approach to study, using visualizations of their performance to motivate them to adopt help-seeking behaviours. Regarding teaching staff and managers, tools 3-10 were developed with the objective of providing information to identify students who are facing difficulties in their studies, expecting staff to reach out to their students and provide guidance. Finally, tools 11-14 were developed for students or staff, aiming to help them to identify crucial courses in a curriculum, and anticipate the impact of course-level improvements in competency attainment at a program level.
Most of the tools presented in Table 1 do not provide information about the methodology used to evaluate its impact on any existing institutional processes. By methodology, we mean experimental approaches to determine whether the tool is effectively fulfilling its objective [7]. For instance, little is known about how students perceived the tools developed to promote help-seeking behaviours (such as tool 1) [8]. Regarding tools that aim to identify students at risk (such as tool 9), there is no clear relationship between the use of the tool and student outcomes yet [8]. And, concerning the tools for monitoring course-level improvement and competency attainment, researchers have just started to evaluate their adoption by collecting information about user perceptions [5], without necessarily reporting data about its actual use to drive improvements in program design or academic program delivery [9].
Building upon this latest idea, we present a case study that evaluates teaching staff usage of a CA tool in the context of a continuous improvement process that has already been installed at an institutional level. Our aim is not only to present the results of evaluating the tool from users’ perspective, but also to analyse how it was used to support an existing process for curriculum improvement. Tool name and user 9. Course Signals (focused on teaching staff) 10. Student Flow Diagrams (focused on managers and teaching staff) 11. Curricular Analytics (focused on managers, teaching staff) 12. Visualized Analytics of Core Competencies-VACCs (focused on students) 13. Competency Analytics Tool-CAT (focused on managers and staff) 14. Course University Study Portal-CUSP (focused on managers and teaching staff)
Developer Purdue University, USA (licensed to Ellucian) University of New USA
Mexico,
Distinctive features [References] A set of risk indicators to classify students in subgroups [1], [8] Sankey diagram to understand students’ academic trajectories [1] University of New USA
Mexico,
Interactive graphical representation of the curriculum [1] Yuan Ze University, Taiwan (currently used in Yuan Ze University) Singapore Management University, Singapore University of Sydney, Australia
Visualization of competency attainment in radar charts regarding grades, credit hours and peer performance [5] Curriculum progression statistics based on competency map and course information [4] Web-based application to model competency development in 5 maturity levels [11] 3 3.1
According to Zelkowitz [7], instrumental case studies are useful to determine if a technological tool makes it easier to produce something compared to prior scenarios. Since the objective of this study is to evaluate usage of a CA tool throughout a continuous improvement process, we select instrumental case study as the most appropriate methodology. The case study took place at UC-Engineering between 2015 and 2017. In this case, we specifically evaluated whether the CA tool supported teaching staff efforts to collect documentary evidence to account for competency attainment. 3.2
In the early 2000s, UC-Engineering decided to accredit five academic programs by
ABET, which concentrated 35% of its undergraduate enrolment (1,500 students out of
a total of 4,000): 1) Civil Engineering, 2) Electrical Engineering, 3) Computer
Engineering, 4) Mechanical Engineering, and 5) Chemical Engineering. In 2015, the
managers at the Office for Undergraduate Studies and the Engineering Education Unit
designed a continuous improvement process to be implemented for the renewal of the
ABET accreditation of these five programs. The continuous improvement process was
organized in six semesters between the first semester of 2015 and the second se
At the end of the continuous improvement process, 104 assessment plans (http://bit.ly/2SYxWxc) and spreadsheets of competency assessment results (http://bit.ly/2VOdUKx) were expected to be reported by 124 teaching staff from 96 course sections, analysing the attainment of the 11 competencies proposed by ABET Criterion 3 (http://bit.ly/2SeVzRj).
Fig. 1. Semester tasks that teaching staff had to undertake for the continuous improvement process implemented at UC-Engineering between 2015 and 2017.
By the end of 2015, 38 assessment plans and competency assessment results were
already collected from 29 course sections for an interim report to be se
Fig. 2. Screenshot of the UC-Engineering CA tool for supporting continuous
improvement. Through this tool, teaching staff performed the tasks required for the
continuous improvement process si
Fig. 4 depicts the period considered for the case study, indicating the period were the tool was implemented. The tool aimed at supporting the continuous curriculum process by means of facilitating the following activities (see Fig. 1 with the semester tasks): (1) Filling a course description form to describe broadly the teaching and assessment methods. (2) Indicating the relationship between competencies at both program and course level (Fig. 2). (3) Listing performance indicators for competencies at a program level that could be assessed at a course level. (4) Aligning performance indicators with graded assessment methods at a course level. (5) Generating automated reports on percentages of competency attainment (Fig. 3). This functionality is integrated with the institutional LMS to automatically capture the students’ grades for align graded assessment methods. (6) Uploading the following documentary evidence as attachments: Course syllabus; Assessment plans; Competency assessment results; Sample of assessment methods.
Fig. 3. Screenshot of automated report of percentages of competency attainment at a course level generated by the CA tool regarding the performance indicator ’Communicates constructively with other classmates’ for the effective communication competency.
Fig. 4. Summary of the continuous improvement process implemented at UCEngineering. Light grey dots indicate the semesters where the teaching staff tasks were not supported by the CA tool, and the dark grey dots the periods where the tool was implemented as part of the process. 3.3
Between 2015 and 2017, 124 members of the teaching staff participated in the continuous improvement process implemented at UC-Engineering. These 124-teaching staff include 61 teachers (44 faculty members and 17 part-time instructors) who reported documentary evidence of competency attainment in 96 course sections, and 63 teaching assistants who supported teachers in outcome assessment tasks (see Table 2). (*) Total number of staff members who reported documentary evidence between 2015 and 2017 (**) Number of staff members who were involved as CA use after tool implementation (42 faculty members and part-time instructors, and 63 teaching assistants) The evaluation process was organized into two phases. The first phase consisted in evaluating how the CA tool was used to facilitate the collection of documentary evidence for analysing competency attainment in curriculum discussions. The documentary evidence consisted of any documentation that was reported to account for competency attainment at a course level, such as: course syllabus, course description, competency assessment results, and samples of assessment methods (see Fig. 1). In order to compare the number and the type of documentary evidence generated before and after the CA tool was implemented, three researchers used a coding scheme to classify the evidence reported by each teaching staff member in each course section. This scheme was developed based on a bottom-up coding approach, and each category was defined by examining the files uploaded in both Dropbox and the CA tool.).From this bottom-up approach, four categories emerged, so the three researchers used these categories to assigned scores from 0 to 1 to account for the type of documentary evidence reported every semester (see Table 3): 1) reported assessment plans, 2) reported a sample of assessment methods, 3) reported competency attainment results, 4) reported course syllabus, 5) included a course description, and 6) used the automated report of the CA tool. Then, the scores assigned ranged from 0 to 6 in each course section, in which a score equal to 0 indicates a minimum amount and variety of evidence reported (and a score equal to indicates 6 a maximum amount and variety of evidence).
Reported competency attainment The teaching staff did not report competency attainment results at results (0) a course section level.
Reported competency attainment The course reported competency attainment results based on results (1) graded assessments at a course level.
Reported course syllabus (0) The teaching staff did not report the course syllabus to complement the evidence items for the accreditation process.
Reported course syllabus (1) The teaching staff reported the course syllabus as a complement to other evidence items reported for the accreditation process.
Included a course description (0) The teaching staff did not include a course description among the evidence items uploaded in the CA tool.
Included a course description (1) The teaching staff included a course description among the evidence items uploaded in the CA tool.
Reported percentages of compe- The teaching staff did not reported percentages of competency attency attainment (0) tainment.
Reported percentages of compe- The teaching staff reported percentages of competency attainment tency attainment (1) at a program level.
The second phase consisted in exploring the perceived usefulness and
ease-ofuse of the CA tool from the viewpoint of its users. For this purpose, researchers have
recently proposed instruments to evaluate the impact of analytical tools on educational
practices [2], these instruments have still been implemented at a course-level in
controlled environments. Therefore, we decided to develop a paper-based questionnaire
based on the prior work of Ali et al [12], considering that his objective was also
exploring teaching staff’s perspectives in a real-life context. Our questionnaire consisted of a
closed-ended and an open-ended question section (http://bit.ly/2Jh3VVG). The
closedended section consisted of a 5-point Likert scale to determine the level of staff’s
agreement on different items related to perceived usefulness and perceived ease-of-use, while
the open-ended section included the following questions to understand usability and
ease-of-use implications from an exploratory approach:
• What use would you give to the CA tool after interacting with it?
• What kind of information would you expect from this tool?
• What do you think the CA tool lacks in terms of information and functionality?
In order to gather information from the user experience once the CA tool
implementation process was well advanced, we applied the questionnaire during a workshop for
teaching assistants held at the beginning of the second se
60% 56%
92% 84% 80%
for institutional management.
information easily.
The CA tool allows me to obtain more information about courses than… 0% 20% 40% 60% 80% 100%
Once all the evidence items and questionnaire were analysed, we triangulated questionnaire and document analysis results from phase 1 (see Fig. 5 and 6). This process consisted of contrasting the amount and variety of documentary reported by teaching staff before and after tool implementation, in addition to analysing the perspectives of questionnaire respondents on tool usage for curriculum analysis based on evidence of competency attainment. 4
The main findings of the case study are summarized in Table 5. Firstly, the CA tool
helped teaching staff collect a greater number and variety of evidence, providing
visualizations of competency attainment at a program level (Finding 1 in Table 5).
Document analysis results show that the number and the variety of evidence reported
per course section increased from two to five after the CA tool was implemented (see
Fig. 2). In most cases, these three additional items included course syllabuses, course
descriptions, and the percentages of competency attainment. Before the CA tool was
implemented, teaching staff delegated the transformation of graded assessment results
into percentages of competency attainment to professionals in the Engineering
Education Unit. However, once the CA tool was implemented, 89% of the course sections
relied on the automated report provided by the tool to account for competency
attainment at a course level. This effort to collect a greater number and variety of
evidence is not due to greater administrative pressure, since UC-Engineering managers
presented a report to ABET i
Additionally, the results of the questionnaire show that 92% of respondents agreed with the item ‘In general, the CA tool seems useful for institutional management’ (see Fig. 4). In the open-ended questions, teaching assistants mentioned that the CA tool facilitated the use of evidence to account for the implementation of a competency-based curriculum throughout the accreditation process. They also mentioned its potential use to provide students with information about course methods and its alignment with competencies from the graduate profile.
Secondly, teaching staff identified usability and functionality issues that affect the use of evidence to inform course redesign and students’ understanding of performance (Finding 2 in Table 5). The results of teaching staff questionnaires show that 56% agreed with the item ‘The CA tool allows me to obtain more information about courses than other tools (such as the institutional LMS and a web application to search for course information).’ In the open-ended sections, respondents claimed that the CA tool had usability issues. For example, respondents indicated that the tool views had too many tabs and fields, which hindered loading of information. Respondents also mentioned functionality issues, such as the lack of feedback about the quality of course information uploaded, and documentary evidence reported. Also, the tool lacks actionable information on student performance to provide timely and specific feedback throughout the semester.
Furthermore, questionnaire respondents indicated that, although they were able to upload the documentary evidence to account competency-attainment, they perceived the CA tool site was not intuitive enough. The fields to be completed, as well as the files to be uploaded were not clearly explained. This may partly explain why only 16% of course sections reported samples of assessment tools. These results suggest that the CA tool could be improved by including ‘help messages’ or indications related to the process within the platform.
Document analysis and questionnaire results Course sections reported 3 additional evidence items for the accreditation process after the CA tool was implemented. 89% of course sections used the automated reporting feature of the CA tool to account for competency attainment at a course level (40 out of 45). 92% of teaching staff agreed with the questionnaire item ‘In general, the CA tool seems useful for institutional management’ (23 out of 25).
Supporting data Document analysis results (Fig. 6) Screenshot of automated report of competency attainment (Fig. 3) Teaching staff questionnaire (Fig. 5) Findings 2. Teaching staff identified usability and functionality issues in the CA tool that affect the use of evidence to inform course redesign and students’ understanding of performance.
Document analysis and questionnaire results Supporting data 56% of teaching staff agree with the question- Teaching staff quesnaire item ‘The CA tool allows me to obtain tionnaire (see Fig. 5) more information about courses than other tools.’ (14 out of 25) 16% of course sections reported samples of Document analysis assessment methods after the CA tool was results (Fig. 6) implemented (7 out of 45).
Dropbox
Curriculum Analytics Tool This case study showed the results of the usage of a CA tool in a long-term continuous improvement process. The main finding shows that the use of the CA tool helped teaching staff collect a greater number and variety of documentary evidence of competency attainment at a program level, allowing staff to upload documents that were previously shared or stored somewhere else, such as course syllabus.
Furthermore, the CA tool provided staff with a visualization of competency attainment during the whole continuous improvement, information that was previously shown and discussed only in curriculum meetings at the end of the semester. This finding not only reflects that CA tools could be a good solution to reduce task workload for curriculum analysis [2], but also that this type of solutions provides visualizations of student performance that is typically hidden in institutional processes. This is particularly valuable for competency-based curriculums because it provides an overview of the competencies attained by the students at a program level. Teaching staff usually face difficulties in competency assessment because they tend to be abstract and complicated at a course level, while managers deal with the complexity of understanding learning results in a hierarchical structure of courses [3]. With this new tool, managers and teaching staff could have more actionable information for making better decisions to reinforce the required competencies at a program level.
Although this work has illustrated that a CA tool can support curriculum analysis in the context of an accreditation process, there are still needs of teaching staff that have not been met by the tool under study. Finding 2 in Table 5 indicates that there are usability and functionality issues that prevent teaching staff from using the tool to obtain information to redesign courses and to understand students’ performance at an individual level. These are staff needs that have already motivated the development of tools [8]. In order to solve these issues, we could specify redesign requirements for the CA tool by using information we have already collected to evaluate usage. Thus, lessons learned from this case study imply that institutions could adopt and adapt existing CA tools to support curriculum analysis, investing not only in tool development and redesign, but also in capacity building for evidence-informed continuous improvement [1].
Along these lines, one of the main contributions of this paper is that it evaluates the use of a CA tool in an existing institutional process for an extended period of time, going beyond current evaluation strategies that mostly employ self-reported data without relying on a technology validation methodology [5]. The document analysis considered evidence items of the whole 3-year period in which the continuous improvement process was implemented. This long-term period assured collecting enough information to determine whether the CA tool facilitated teaching staff efforts to collect evidence of competency attainment before and after its implementation [7].
Yet, this study has its limitations. Questionnaire results only represented a small sample of teaching staff members who interacted with the CA tool. To understand the contribution of the CA tool for a larger group of teachers, future work would have to explore the type of information most staff need to inform course or curriculum redesign. We anticipate this work might require monitoring learning results for an extended period of time as we did it in this case study [5], so we can expand the current knowledge on the impact of CA tools on curriculum improvement even further. 6
This case study illustrates the usage of a CA tool to support a continuous improvement process for a competency-based curriculum in a university setting. Findings show that CA tools support curriculum analysis when they are implemented to help teaching staff to cope with tasks they are already performing for an existing institutional process, such as course planning and competency assessment. In this study, the CA tool not only facilitated the collection of documentary evidence to account for competency attainment at a program level, but also provided visualizations of competency attainment results that are usually not available for staff. Despite staff feeling comfortable when using the tool to upload evidence of competency attainment, usability and functionalities issues should still be improved in future versions of this tool. Future work on CA adoption should not only focus on tool development, but also on the evaluation of its impact in the formulation of curriculum improvement actions.
This work was funded by the EU LALA project (grant no. 586120-EPP-1-2017-1-ESEPPKA2-CBHE-JP). The authors would like to thank Camila Aguirre from U-Planner for collaborating with this study, and the reviewers for their constructive suggestions.