In this paper, we present some of our recent experiences with a data visualization tool and offer some use cases where the visualization tool can potentially drive programmatic change in universities. The Ribbon Tool provides an interactive visualization of student flows through academic programs, progressing over time to either successful completion (graduation) or attrition. Through effective use of the Ribbon Tool by those who can effect curriculum change, their ability to generate persuasive arguments for change are enhanced. This paper presents some use cases and commentary on actual usage of the Ribbon Tool to call for programmatic change across a university.
Academics pride themselves on evidence-informed decision making, but when it comes to making changes in their teaching practices, curricula, or academic programs, data and evidence seem to hold little sway. Perhaps this stems from the belief that as an expert in a subject area, one is automatically an expert in how, where, and what of the subject area should be taught. Perhaps this stems from the outdated “mini-me” assumption that students are either faculty in training or destined for attrition. Or, or perhaps it stems from the discipline-based belief that teaching practices, curricula and academic programs were carved in stone tablets by the ancestors and never meant to change.
Instigating change in university programs is difficult, in part because it is easy to throw sand in the wheels of change, but also in part because the agents of change and the influencers are rarely the same people. Sadly, evidence-informed arguments to justify changes in teaching or curriculum often have no more persuasive effect, or perhaps even less effect, than anecdotal stories about “in my day”, or “my son or daughter experienced”. While skepticism can be healthy when evaluating evidence gathered from others’ observations and statistical analysis, it can also be used to stonewall or stymie change.
Confronting academics and administrators with cold facts, such as “One third of your students from certain diversity groups are leaving your program within the first two years” or “One quarter of your students are failing their required first mathematics course” are met with retorts like “Tell me something I haven’t heard before!” or “Bring me some evidence that is actionable!”.
A data visualization tool called the “Ribbon Tool” has been
developed at UC Davis (http://t4eba.com/ribbon/) building upon
the Sankey Diagram functionality with the Data-Driven
Documents (D3) data visualization library [
In the Ribbon Tool, a “mouse-over” in the diagram will reveal a text box showing the number of students in a particular ribbon. The textboxes in Figure 2 show that of the 351 students who began in Engineering in the fall of 2011, some 240 were still enrolled in the fall of 2014. Another 33 students had transferred in from Arts and Science. Some students had transferred out of Engineering, to Arts and Science or another faculty. Some had dropped out of the University, and a few were on a “stop-out”. By fall 2015 (after 4 years), one can see that 88 students had graduated with an Engineering degree. A few others had degrees in other faculties and 177 were still enrolled for their 5th year. Note that a substantial number of Engineering students complete a one-year paid internship, which naturally extends the degree to a minimum 5-year duration.
The vertical bars represent a hierarchy of temporal information. In the above example, the top level of the hierarchy represents whether students were enrolled, had been granted a degree, or had left the institution. In the next level, we show the college or school in which they had been enrolled (or had granted them a degree or from which they left or stopped-out). If one were to drill down to a third level, the data shows the department (Electrical, Mechanical, Civil, etc.) where the student is enrolled or awarded a degree. Expanding or collapsing the hierarchy gives a more or less refined view. The interactive visualization allows the user to isolate a particular group in the hierarchy (for example students who were enrolled in Mechanical Engineering in 2014) and project backward to see where they came from and forward to see where they went next. Moving through the hierarchy and isolating views allows the user to focus in on areas of potential interest.
Along with the visualization, the user is provided a set of filters. For example, if one were interested in examining gender differences in student flows through Engineering, one could filter to obtain separate diagrams for female and male students. These can be quickly visually compared to see if proportions of degrees granted, attrition, time to graduation, departmental breakdowns are impacted by gender. Other filters based on any set of categorical demographic or academic characteristics can be added. For example, the program flow-through for female students entering Engineering directly from high school with SAT scores in the top decile can be examined with a few mouse clicks. This flexible and powerful visualization tool has been used extensively at UC Davis and is now being disseminated to other universities through the “Tools for Evidence-Based Action (TEA) Community” [3], funded in part by the Helmsley Foundation. The Ribbon Tool has been greeted with great enthusiasm by deans and other administrators at our University as a tool to augment their other data analysis efforts and as a means to explore elements of their academic programs.
The Ribbon Tool requires two chunks of data, a set of filter values and a data hierarchy. There can be an arbitrary number of filter variables, each with a label and a set of nominal values. The data hierarchy can have an arbitrary depth and at each level of the hierarchy a value must exist for each student. The branching factor at each level of the hierarchy must be fixed in terms of its subcategory options. Each student represented in the visualization must have a full set of values corresponding to the filter variables. Further each student must have a value for each level in the hierarchy. Data can be imported into the Ribbon Tool from either a pair of csv files or from a JSON file.
In the datasets we have prepared for our institution, students are not individually identified, other than by a sequential index. As a result, the data held in Ribbon, although hosted in the Amazon Cloud, has low risk of abuse. Nevertheless, efforts are underway to offer a local data storage option for some universities hesitant to store even de-identified student data off-site.
We have been using the Ribbon Tool at the University of Saskatchewan for only a few months now. During that time the tool has been further enhanced in its capabilities and features and improving in its reliability and robustness, thanks to the development team at UC Davis. Below are some use cases that have proven useful in our experience to date.
Timely degree completion is a key component of enrolment management. For example, university funding is often associated with 6-year completion rates in undergraduate programs. Students stuck in a program for an extended time can reduce the number of available spaces in critical courses, and can face compounded delays due to rigid, prerequisite-bound course sequences.
Using the Ribbon Tool it is easy to see degree completion times and to determine the number of students in a cohort who are completing degrees within 6 years or who are embarking on a 7th or 8th year. Furthermore, it is possible, using filters to see if the students failing to complete within 6 years have had stop outs, academic probation actions, internships, etc. It is possible to differentiate completion rates for students with different demographic factors, such as first-in-family (first-generation) students, international students, under-represented minority students, etc. It is possible to display student GPAs within the hierarchy to determine if students slow to graduate have or have not reached certain academic achievement levels.
The combination of filters and hierarchy refinements has permitted our Engineering School to discover some new insights and bottlenecks regarding time to graduation.
Analyzing student attrition and retention factors is an interest in some parts of every university. Universities focused on broad access in Arts and Sciences are often faced with retention challenges. Students unprepared for the change in culture of university life and those with academic deficiencies are not the only students who sometimes leave the institution. Established retention risk factors such as lower socio-economic status, being a first-generation student, being a member of an under-represented minority all need to be considered. But when comparing different academic programs, such as Engineering and the Humanities, there may be different factors leading to attrition. For example, belief in the benefit of completing a university degree may be a factor in some areas whereas the rigor of completing the degree may be a factor in others [4].
We have used the Ribbon Tool to track attrition and to differentiate students moving to a different program versus stopouts versus drop-outs. Furthermore in areas where there are various entry points into programs, it is possible to examine retention factors for students who have entered through different paths. In doing a retention analysis with Ribbon, demographic filter variables corresponding to expected causes of retention can be quickly examined to see which factors or combinations of factors seem to make a difference. Being able to isolate a particular collection of students (e.g. those who drop out after sophomore year in a program) to further investigate their demographic makeup and their pathways has proven useful. Ribbon can also be used to determine whether the flow of students through academic programs has been affected by changes in demographics of entering students, whether as a result of changes to the feeder system or changes in admission policies. The Ribbon Tool has enabled our Engineering School as well as our Faculty of Arts and Sciences to study retention issues (in STEM and elsewhere) more closely and to get a better understanding of attrition patterns, particularly of underrepresented minority students.
As academic programs evolve and as new learner supports are introduced there is a need for ongoing program monitoring and evaluation. The Ribbon Tool provides a mechanism for supporting the early phase of program evaluation through its rapid means of detecting differences across cohorts of students. For example, it is easy to compare student flows before and after the implementation of some program change. It is also possible to differentiate with a filter those students who were selected for participation in a pilot program and further to filter those who did or did not engage.
We have begun to explore the impact of changes to our academic advising processes, the introduction of a freshman learning communities program, and the impact of increased academic support services in mathematics and writing. In such programs, where the macro effects may take many years to be realized, where effects may be differential across the different student demographics, and where levels of participation and engagement are vital indicators, the Ribbon Tool is helping us to us develop and refine program-impact hypotheses that can then be tested statistically.
Of course, all of these kinds of comparisons and descriptions presented in the use-cases above can be achieved with a comprehensive set of reports, bar charts or tables or with the facile use of a statistics package. The difference with the Ribbon Tool is the speed with which one can mock up a scenario and try different filters and breakdowns to get an impression of where problems may be lurking or where impact may be seen. Furthermore, with the Ribbon Tool, an Associate Dean or Department Chair can take the reins and drive the visualization tool to explore exactly what is interesting - to follow a hunch or to confirm or deny a commonly held view.
Putting a powerful visualization tool in the hands of agents of change can empower them to make more persuasive cases for change with their colleagues. We have seen how visualizations that show scenarios with no perceptible difference, when conventional wisdom would predict a difference, does help people to confront and question their biases. These are precisely the kinds of evidence that can change minds, and actionable decisions arise from changed minds.
Our experiences with the Ribbon Tool confirm that visualizations of student progression can be highly informative and powerfully persuasive in moving administrative staff to action. Uncovering the factors affiliated with undesired outcomes and discovering those connected with positive outcomes sets the stage for change.
The Ribbon Tool is one tool that can help with moving people to action, but like any tool it has its limitations. It is best suited for analyzing historical patterns and flows and is not well suited for forecasting or modeling the future effects of potential changes. It is also a tool that readily looks over relatively longer time scales we have not yet produced data to explore a more granular time scale. Finally, like any other tool, it can be mis-used to oversimplify relationships or to mis-represent realities. Just as with any power tool, much persuasive power is placed in the hands of the tool operator. [3] Tools for Evidence-Based Action (TEA) Community (http://t4eba.com).