Learning analytics, with a risk management approach, provides relevant and actionable information to teaching and administrative staff to make evidence-based decisions in curriculum and program quality improvement. This paper outlines the development and pilot implementation of a risk management model with an online feedback system in a research-intensive Australian university. Providing teachers and executives with the opportunity, facilitated by the essential IT infrastructure, to contextualise data and to document their response to the identified risks is a proactive approach to empower staff to make enhancements to their teaching practices, and to influence academic management. In addition, the opportunity for individual teaching staff to examine the progress of their own courses is a fundamental step in curriculum and program quality improvement. Positive feedback has been received in terms of the ease of access and opportunity provided to contextualise the risk. Future development will incorporate dynamic data from different sources, such as student participation in the learning management system, to build a holistic risk management framework in teaching and learning. • Social and professional topics→Professional topics→ Management of computing and information systems→Project and people management→Systems analysis and design
Risk management; analytics; teaching; curriculum; quality assurance.
In the current highly competitive environment, new modes of
governance that emphasise performance, quality and accountability
of student learning and experience have become common practice
in higher education institutions (HEIs) [
This paper outlines the development of a risk management framework in the revised Curriculum and Teaching Quality Appraisal (CTQA) process at a research-intensive Australian university, which will be fully implemented for the academic year 2016. The pilot phase of implementation concluded in January 2016. The paper also discusses how a risk management model better facilitates data-driven decision making, and curriculum and program quality improvement, compared with the traditional performance management framework. Alongside with the risk management framework, a series of interactive reports and dashboards for University Executives, Program Convenors, Course Coordinators and teaching staff are also developed. This is an attempt to provide comprehensive, relevant, and actionable information to key stakeholders to encourage the use of evidencebased practices, as well as to assist individual teaching staff to examine the success of a course which is fundamental to curriculum and program quality improvement. Last, but not least, an online feedback system also acts as an effective means to close the loop of the risk management process. Staff are provided with the opportunity to document their response to the data provided via the online feedback system. Risk management with active participation from staff empowers the University community to make datadriven decisions in considering student learning and experience.
The CTQA is a key component of this University’s overall quality assurance process in teaching and learning. It is undertaken on an annual basis, and involves an evidence-based consideration of the overall quality of its teaching programs. The previous CTQA process was established in 2008 and was based on a performance management model, which identified programs that did not meet the specified performance indicators. Since 2008, there have been changes in both the external and internal higher education environment. In order to align the University’s teaching and learning quality assurance process to the national agenda, and to maximise the internal benefits of this quality assurance process, a decision was made to revise the CTQA process.
The principle of the revised CTQA process is to collect relevant
data, and undertake critical and diagnostic data analyses which
focus on trends, issues, actions taken and outcomes to support
ongoing curriculum and program quality improvement. The
rationale of selecting a risk management framework, instead of
using a performance management framework, is based on the
concept that through identification and management of risk, it can
impact performance. A performance management framework
focuses on the measurement of the actual results and their deviation
from the targets [
The first step in developing the revised process is key stakeholder
consultation to ensure that relevant and actionable information is
provided to teaching staff and University executives. A broad
consultation was conducted with the Associate Deans (Academic)
in each Faculty, Chairs of Teaching and Learning Committees of
each School, Heads of Schools, Program Convenors and Course
Coordinators. Through committee meetings, presentations and
individual discussions, a community of teaching and administrative
staff was encouraged to engage in making evidence-based
decisions to improve student learning. Based on the outcomes of
the consultation, in alignment to the TEQSA risk assessment
framework [
The set of risk indicators for programs and the rationale, based on
the TEQSA risk assessment framework [
Overall Position (OP) 1-5 (OP ranges from 1 – the highest to 25 – the lowest): This indicator shows the ability of a program at this University to attract students with high academic achievements in comparison to its competitors. A significant decrease may signal a decline in the quality or value of the program offered. However, recruitment strategies and employment in a profession need to be considered when interpreting this indicator. 2. 3. 4. 5. 6. 7. 8.
student load could potentially impact on the quality of student experience. Conversely, an unplanned significant and continuing decrease may signal a decline in the quality of the programs offered as perceived by prospective students.
that there are potential quality issues in the process of student admission, teaching and learning, and the overall student experience. Prompt actions to address early attrition are critical to minimise the compound effect on attrition in the later years of the program.
employment rate could indicate that students may not be well-equipped with the necessary graduate attributes for successful transition to the next stage of their chosen profession. However, volatility in the labour market needs to be factored in when interpreting this indicator.
of the academic environment. When the pass rate is at very high/low levels, it may suggest that there are potential quality issues in student teaching and learning, and/or the overall student experience.
Completion Times: This indicator represents one dimension of the effectiveness of the delivery of educational services. Number of students in different study mode (full-time or part-time) need to be factored in when interpreting the results. Prompt actions to identify at-risk students, at an early stage, who are not being able to complete a program and to provide them with appropriate support are essential to minimise the possibility of reaching the stage of non-completion.
The set of risk indicators for courses and the rationale, based on the
TEQSA risk assessment framework [
enrolments could potentially impact on the quality of student experience. Conversely, an unplanned significant and continuing decrease may signal a decline in quality in courses offered as perceived by prospective students.
of the academic environment. When the pass rate is at very high/low levels, it may suggest that there are potential quality issues in student teaching and learning, and/or the overall student experience.
Response Rate: This is one of the indicators to reflect student engagement with the course in providing feedback. However, strategies implemented and timing at which the SECaT was administered need to be considered when interpreting this indicator.
11. Average SECaT Score for Q8: Overall, how would you rate this course? For indicators 4 to 11, these are core quality indicators to provide a guide as to whether a course met student expectations. Prompt actions to address low student satisfaction scores in specific areas will assist in identifying the issues and implementing appropriate strategies to minimise student attrition and increase overall student satisfaction over time.
Using separate sets of risk indicators for courses and programs enable individual Course Coordinators and teaching staff to examine the success of the courses that they have taught in a semester. This is an obvious progression from the former CTQA, as previously only faculty- and school-level data were available with limited individual course/program information. Nevertheless, individual courses are the building blocks of the curriculum and program. The provision of course-level data will further engage teaching staff in the curriculum and program quality improvement. Most importantly, the key feature of this risk management model is the opportunity provided for teaching and administrative staff to contextualise and mitigate the identified risk, to make a decision on whether the identified risk should be closely managed, or the risk is expected and actions have been in place to minimise its impact. Staff can also document their feedback to the data provided via an online feedback system which will be further discussed in Section 5. This active engagement from teaching and administrative staff in the revised CTQA process encourages them to reflect on the relevant student learning data and adopt a continuous improvement approach to teaching and learning. Staff are able to review individual program data on an annual basis, and individual course data on a semester basis. By using trend data of each program and course, teaching and administrative staff are proactively managing risks rather than reactively managing performance. The revised process not only identifies the at-risk courses and programs, but also the minimal-, neutral-, increasing-risk courses and programs. The opportunity to explore the risk indicators, which contribute to a heightened risk for increasing-risk courses and programs, as well as those result in a lesser risk for neutral- and minimal-risk courses and programs, allows staff to adopt a proactive approach in managing risks. For example, course staff are able to modify their teaching practices, such as the use of a flipped classroom model to allow more interactive sessions with students, in anticipation of an increasing trend of student enrolments. Unlike the reactive management approach, staff only formulate a solution after an increase in student enrolments is evident. The revised CTQA is an annual process that focuses on data-driven decision making through contextualising and mitigating risks, evidence-based action planning, and revisiting and evaluating proposed actions in subsequent annual reviews.
This section outlined the development of the revised CTQA process. The next section will focus on how to create visualisations that encourage a community of teaching and administrative staff to engage in making evidence-based decisions to improve student learning at both course- and program-levels.
The ultimate goal of data visualisation is to provide clear and useful
information to the targeted audience. However, it is an iterative
process to find the best way to visually present data to meet the
needs of the stakeholders [
The third level is the detailed course/program report for an individual course/program (see Figure 3). Previously, Course Coordinators or individual teaching staff were required to collate and compile their own reports from the available and relevant teaching and learning data about a course/program. The new reports consolidate all the required data and provide the stakeholders with an integrated report for each course/program. Staff, who have access to these modified detailed course/program level reports, are already actively using them to explore the strengths and limitations of their courses/programs. They have also provided positive feedback about the reports and process. This unified approach reduces a considerable amount of administrative time in collating data. As a result, they can use the time to engage in data-rich conversations focused on improving curriculum and pedagogical practices, reflection and decision-making as to how to improve student learning in their course/program.
In addition, these three levels of reports and dashboards are interrelated, which provide the opportunity for key stakeholders to either drill down to the details of the strengths and limitations of a course/program, or zoom out to look at the relationship of a particular course/program to the relevant group of courses/programs. These three levels of data visualisation aim to generate conversations, initially, between individual teaching staff, and gradually expand the conversations with the Course Coordinators and Program Convenors, and collaborate to make evidence-based decisions to improve teaching practices, curriculum and program quality.
Apart from the three levels of data visualisation, it is essential that reasonable requests of teaching and learning data from individual teaching staff are adequately addressed. Nevertheless, courses are the building blocks in a curriculum and program. Providing individual teaching staff with customised reports could, in fact, extend their engagement in the curriculum and program quality improvement process. The additional data that an individual teacher requests may also be beneficial to other courses/programs. Hence, consideration should be made to incorporate those in the new iteration of the reports and dashboards. An example is the request of analysing the distribution of assessment types (that is, examinations, presentations, essay writing) in the compulsory courses of a program. These relevant and actionable data about assessment allows teaching staff and Program Convenors to have a holistic view of student learning and assessment experience in a program. When data revealed that a large percentage of assessment was examinations, one would expect that investigation into the rationale of the existing assessment regime is conducted and changes will be made to provide students with the opportunity to demonstrate their knowledge and skills via different modes of assessment. This process is the start of a continuous improvement approach to teaching and learning, in which assessment is a core component, and should be encouraged in other Faculties/Schools. The first iteration of data visualisation for the revised CTQA process only includes static and historical data about student learning. In the second iteration of data, the aim is to create interactive reports and dashboards with automatic drill-down functions to reveal dynamic data, such as student access patterns to online resources and assessment, and student and teacher engagement patterns with the Learning Management System (LMS). As part of the curriculum and program quality improvement, these additional data about student interactions with online resources and technologies would provide insight into the optimal structure of a course/program that will engage and motivate students to learn.
The continuous process of reviewing, reflecting and proposing new solutions is a core part of the quality improvement process. One of the strategies to engage a community of teaching staff in curriculum and program quality improvement is to empower them to complete the revised CTQA process loop via an online feedback system (see Figure 4). The purpose of this online feedback system is to provide an opportunity for staff, firstly, to provide contextualised information around selected courses/programs, such as those identified as increasing- or at-risk. Secondly, to confirm or disconfirm the identified risk and determine the residual risk for relevant courses/programs as minimal-, neutral-, increasing- or atrisk. Finally, to document proposed actions that will be undertaken to address the confirmed risks. The documentation of feedback is pivotal in the continual cycle of curriculum and program quality improvement, as the feedback collected from academic staff, Course Coordinators/Program Convenors, and Faculty Executives establish the basis for the required actions to address the risks. All key stakeholders can review their feedback and document progress in comparison to the previous release of data. The program reports and dashboards are updated on an annual basis, whereas the course reports and dashboards are released after the conclusion of a semester. Once these reports are available, each Faculty and School will have the autonomy to decide which group/s of courses or programs to focus on in order to enhance their delivery, and the approach they use in response to the data provided. This autonomy provides opportunities to generate conversations among staff to develop a Faculty/School-wide response to the issues identified and raised during the review process and the ability to apply the learnings of best practice to other courses or programs requiring intervention and/or reward. In summary, this online feedback system is developed to enable collection and consolidation of feedback and proposed actions to address risk.
The purpose of this pilot was to ascertain the effectiveness of the new process and associated communication strategy. The information gathered provided an opportunity for the Learning Analytics and Evaluations teams to mitigate risks associated with a University-wide implementation, and facilitate resolutions to any identified issues prior to the formal rollout of the new process across the University.
Feedback from the participants was positive. They appreciated the integrated course/program reports which provide all the relevant data for a particular course/program. This unified approach reduces a considerable amount of administrative time in collating the data from different sources. In addition, the Faculty/School reports provided an overview of the minimal-, neutral-, increasing-, and atrisk courses/programs in a Faculty/School, which assists in directing attention, resources or recognition to particular groups of courses/programs. The identified courses/programs risk dashboard appeared to have face validity based on the participants’ knowledge and experience. Participants also acknowledged that the revised process provides them with the opportunity to contextualise and mitigate the identified risk, to make a decision of whether the risk should be closely managed, or the risk was expected and actions have been in place to minimise its impact via the online feedback system.
This paper presents how learning analytics methodologies play a pivotal role in developing understanding, optimising and transforming courses/programs, using a risk management framework with an online feedback system. The two major challenges encountered in the development of the revised CTQA process are the institutional culture change from a performance management to a risk management framework, and collaboration with the business intelligence and IT departments. The lessons learnt in developing and implementing the pilot revised CTQA process revealed that effective communication, with the support from the University senior executives, is the best strategy in dealing with these challenges. Although a cultural shift in an institutionalwide system can take up to a few years, consistent communication and clear expectations from all key stakeholders involved are the important incremental steps in shifting the culture from a performance to a risk management model. In terms of collaboration with business intelligence and IT departments, the message needs to be focused on the value-adding role of learning analytics to the current business intelligence and IT functions, instead of being perceived as a threat to their operation.
The development of the risk management framework, and its associated reports and dashboards and online feedback system, is still evolving. Continual support to the teaching and administrative staff in terms of understanding the data, as well as possible pedagogical enhancement that they could implement in their courses/programs, is required to sustain their engagement with the data to make evidence-based decisions in the curriculum and program improvement process. Future development will incorporate dynamic data from additional sources, such as student participation in the LMS, to build a holistic risk management framework in teaching and learning in higher education.