In this paper, we present some of our ongoing, as well as more recent work designing, implementing, and improving three tools to help university instructors and department leaders make evidence-based improvements to instruction. The first tool, Know Your Students, is a very early prototype that helps instructors tailor their instruction based on characteristics of the students they would not otherwise be aware of in their courses. The other two tools, the Departmental Diagnostic Dashboard and Ribbon Tool, help department chairs, curricular chairs, and/or advisors identify and make sense of student patterns they may be trying to minimize or enhance within individual courses, course series, and/or throughout their entire program. This paper illustrates examples of these tools and some of the actions they have inspired as a means of improving student outcomes.
University faculty members, staff and administrators often pride themselves on making decisions in a collaborative manner that takes multiple viewpoints into consideration. They view their decision-making as highly informed by evidence though the evidence is often more opinion-based than quantitative in nature. When data is brought into the equation it is often insufficient, not timely, nor tailored to questions that have the potential to impact student outcomes. Class size, student grades, and time to degree sliced and diced by gender, ethnicity and class standing are often the primary measures and offer outcome information after the term. Student evaluations of general course satisfaction are also universally applied and tend to be the only “actionable” information provided to instructors, albeit at the end of the course. There is a fundamental need to better understand our students, and the patterns that exist within our instructional system at the course, department and institution levels, to bring about effective instructional improvement.
Often, when instructors, curricular chairs and others involved in student instruction are asked how to improve student outcomes, a few common replies are heard: “we need to be more selective”, “students were better in the past”, “our students are not prepared”, “when I was a student I was expected to work so much harder”, “we need to fail more students and increase our rigor”, and the list goes on. There is a tendency to blame the students while reality
When instructors receive their class roster at many universities they are usually presented with student names, identification numbers and codes corresponding to students’ field of study (4 letter major codes at our institution). The assumption is often made that the students have met all preparatory requirements and are ready to effectively engage in the course content. A student’s failure to grasp course content adequately is usually assumed to be due to lack of time on task, lack of interest, lack of capability, and/or poor preparation. Very few instructors bother to take the time to check on one or more of these assumptions due to time pressure to “cover” all needed material, especially when teaching courses of 50 or more students over a 10 week course period (our standard “quarter”).
We posit that there is actionable data that can be shared with the instructor prior to course start that can lead to a better instructional experience and increased student learning. This data is gathered by our Center for Educational Effectiveness, a division of undergraduate education, and presented in aggregate fashion. This aggregation helps alleviate potential privacy concerns were the data to be presented is by individual student. Some of the information that can be considered, and sample actions that have been attempted, include: Basic demographics This can contain aggregate information on gender, year in school, international status, primary language spoken, need for English remediation, first generation status, socioeconomic status, extent of testing accommodations for learning challenges and more. Such information can shape the form of writing assignments and grading rubrics, the types of examples brought to the classroom, the types and form of classroom activities or assignments chosen.
Preparation Background expertise brought into the course from prior course experiences at the university or at prior institutions is important. This not only includes grades received but also time gaps with material, course repetition, learning objective achievement (only available where measured as part of standard course practice – currently only available for one department’s introductory courses), pre-requisite completion, motivational survey data and more. Such information can greatly influence course content and emphasis as well as examples chosen. Motivation and load Students’ needs and interest in a specific course is based on a variety of factors such as course of study being pursued, credit load in quarter, number of STEM courses currently enrolled in, course difficulty load and more. Awareness of student motivation can greatly influence course examples chosen, course workload and expectations.
These are the three areas that are currently being pilot tested with multiple first and second year courses that vary from 70 to 600 students. Data are currently aggregated manually into a multiple page report shared with the instructor. The information and analyses are being created using Tableau, SPSS and R, as most convenient. The envisioned final product would be an automatically generated analysis with suggested actions for student success and links to a Shiny dashboard providing more detailed information.
One instructor currently prototyping Know Your Students information dashboard teaches a first course in organic chemistry for physical science majors. Based on the data she has discovered that a substantial number of students had not met pre-requisites or had performed very poorly on prior introductory chemistry courses, most of her students were first generation which usually is indicative of lack of knowledge of support structures within the university. Over 40% of her students had not had chemistry for over a year with some having had as long as a 2-year break. She also received information pointing out the level of mastery of her class on 27 learning objectives covered in the introductory chemistry year (3 courses; Chemistry 2A,B,C). Some of her information can be seen in Figure 1. Based on this information, she has altered various course sessions as well as expanded the range of information she would like to see from the product in the next iteration.
The Departmental Instructional Dashboard was initially conceived as a unified place where each of our 100 plus programs of study could be better understood by department administration and advising staff. The dashboard provides information on a quarterly basis rather than the traditional once every seven year program review cycle. With an ever-growing pressure to improve graduation rates, this prototype tool was created in Fall 2015 and now contains student, course, and graduation information for all undergraduate programs at our university since 2000. Once one or more programs are selected, the following types of information are currently available in the prototype: Student information This information shows numbers of enrolled students by quarter that can be separated by multiple different demographic variables. In Figure 2 you can view the growth of our computer science major since 2006 as well as gauge the numbers of under-represented minorities in the program and whether they entered as freshmen or as third year transfer students.
Course Information This area shows information related to course popularity at the various stages of a student’s timeline on our campus, and can point out courses that are particularly difficult or challenging for students in a particular course of study. In Figure 3 you can see which courses were taken and when they were particularly troublesome for the computer science graduates that started as freshmen and completed their degree within 4 years.
Graduation Information These tabs contain information about an initial cohort of students who enrolled in a given major and the numbers and percentages of those students that were able to graduate, graduate within four years, or almost graduate within four years. Additionally there is information about “forgetters” – those that graduated one term after 4 years but took no units in their last term, and “almosters” – those that finished one term past four years. This information is presented via simple bar chart outlining numbers of students and courses taken.
The dataset used by the dashboard is a cleaned dataset supplied by our registrar and enhanced with multiple additional fields and substantial calculations, analyses and visualizations completed in the R programming language and delivered via a Shiny dashboard interface. The tool was initiated as a summer project for one of our Master’s students in statistics and is now slated to be delivered via password-protected Shiny interface to all of our program administrators and lead advising staff.
We have developed a data visualization tool called the
“Ribbon Tool” (http://t4eba.com/ribbon/) building upon the
Sankey Diagram functionality with the Data-Driven Documents
(D3) data visualization library [
Vertical bars within the tool indicate the status of students in a particular year and term of an academic program. The ribbons that flow from bar to bar correspond to the number of students moving from state to state. For example, in Figure 4, the engineering discipline, a set of multiple degree programs, is compared to MTHPS or math and physical science programs. Note that while the two programs are roughly equivalent in size at the start, 670 for engineering and 500 for MTHPS. Four years later 200 engineers have graduated and 228 are still enrolled with the other 252 gone from engineering (52 dismissed in the first year alone) while in MTHPS 133 graduated and 71 are still enrolled four years later with almost three hundred going mostly to other disciplines.
Within the Ribbon Tool, hovering the cursor over a ribbon in the diagram will reveal a text box showing the number of students it represents. A right click will allow the ribbon to be further subdivided as dictated by the available variables. In this case, further subdivision can reveal individual majors, gender, international status and more with the ability to rearrange the order of splitting to highlight different comparisons.
The vertical bars represent any form of milestone, in our case
term dates, but really anything deemed to be worthy of
demarcation, such as passing a course, a set of courses or other
criteria can be utilized. The data format, available as both JSON
or multiple CSV files can allow any form of information to be
visualized. Progression in a course timing (when course A then
course B then C are taken and patterns based on course of study),
course series and grade progressions (getting a specific grade in
course 1 then leads to specific grades in course 2 and so on),
progressions based on term attended (allows multiple cohort years
to be overlapped) are just some of the other types of data
progressions that have been visualized. Additional examples of
the Ribbon tool in use can be read in greater detail in Greer’s
paper [
This flexible, easy to use and powerful flow visualization tool
has been used extensively at our institution and is disseminated to
other universities through the “Tools for Evidence-Based Action
(TEA) Community” [
All of the ideas and visualizations presented in this paper can be created via specific requests to institutional research staff, the Registrar, Admissions and/or department analysts – the difference is that the request, turn around time, and likely need for multiple iterations can be mostly bypassed once useful datasets are obtained or created. The tools presented allow for a great deal of local exploration and generation of powerful visuals that can help communicate ideas to others in position to make positive changes. The dashboard approach taken in the Departmental Dashboard and Know Your Students suites of analyses allows for easy central updating, secure use, and quick iterative improvement. The common language across departments that is gained from using the same tools also facilitates discussions across departments and colleges allowing the potential for decisive actions to occur on a faster timescale. Additionally, the collection of tools fosters informed discussions and decisions at the scale of the individual instructor, the department, the school/college, all the way to the institution- or system-wide level.
Our extensive experience with the Ribbon Tool, growing use of the Departmental Diagnostic Dashboard and beginning work with the Know Your Students collection of data are setting the stage for widespread use of data to improve instructional outcomes for all students. As with all tools, there is still much room for improvement as well as ongoing opportunities for misinterpretation. Ribbon Tool and the Departmental Diagnostic Dashboard are best for looking for patterns and helping uncover areas for potential improvement while the Know Your Students tool can help break the patterns from the start and guide, hopefully, useful interventions. It is yet to be seen how many types of data and visualizations thereof can inform effective actions. Showing a trend does not clarify where and how action should be taken. Still, ongoing experimentation with these tools continues to uncover new ways for their use and inspires more thinking about what instructors, administrators and staff can do to create the most effective educational experiences for our students.