One of the most important aspects to consider during the development of a PhD is the students' progress, both for their advisors and the students themselves. However, several achievements of different natures are involved during a PhD (research stays, publications, seminars, research plans, etc.). For these reasons, we propose a set of data visualizations to support decision-making processes in a PhD program. A preliminary requirement elicitation process was carried out to obtain a design basis for the implementation and integration of these tools in the PhD portal. Once the visualizations were implemented, a usability study was performed to measure the perceived usability of the newly added PhD portal functionalities. This paper presents the design process and usability study outcomes of applying data visualizations to the learning outcomes of the PhD Programme in Education in the Knowledge Society at the University of Salamanca.
Data visualizations are crucial tools to understand and exploit Learning Analytics and educational outputs. They allow the transformation of raw data into valuable knowledge that could lead stakeholders into better decision-making processes.
It is very important to lead decisions using data, especially in the educational domain, because data and evidence-based policymaking provide the means to improve learning, quality, and the relationships between every involved actor within this context [1-3].
However, data is not always presented using the best visualizations (or not even presented using visualizations at all), which could result in weaker informed decisionmaking processes because data is not fully exploited. In fact, several roles and actors must be considered within the educational context, which also adds more complexity to the data exploitation process, because each actor have his or her own information goals and requirements.
For all these reasons, the design of data visualizations for Learning Analytics and educational outputs is not a trivial task, and it needs to be tackled using a user-centered approach [4].
This work describes the integration of data visualizations in a PhD portal to follow the progress of students and advisors during the development of their PhD thesis. The purpose of this integration is to provide more context to the PhD Program managers and academic committee regarding the workload and achievements of the PhD advisors and students, as well as to improve the engagement of the platform by presenting the milestones and current situation within the PhD to each student.
These data were already being collected through the PhD portal, but the statistics about them were not accessible through straightforward methods. The addition of visualizations is set to exploit these data and offer an accessible tool to fully understand the milestones, achievements and related metrics derived from the PhD program.
The rest of this paper is organized as follows. Section 2 outlines the Doctoral Programme in Education in the Knowledge Society, which is the PhD Program in which we integrated the information visualizations. Section 3 describes the methodology followed to collect the information requirements, as well as the user study that we carried out to validate the modified portal. Section 4 details the visualizations included, while section 5 presents the results of the user study. Finally, section 6 discusses the results and section 7 concludes the results with the conclusions derived from this work. 2
The PhD Programme, “Education in the Knowledge Society”, was established and launched in the academic year 2013-2014 at the University of Salamanca (Spain), following the Spanish Royal Decree 99/2011 [5, 6]. The Programme is based on four cornerstones.
Firstly, multidisciplinarity and interdisciplinarity. In this sense, the Programme is based on the foundations of the knowledge society, i.e., on technology and learning. The problems and challenges of this society are so complex that they cannot be tackled from a single perspective, hence the need for multidisciplinarity. However, many of them require the application of approaches from various disciplines; therefore, interdisciplinary interventions are sought and recommended, although the longer-term objective is to achieve true transdisciplinarity.
Secondly, this Programme is aligned with the University’s R&D&I strategy, the regional and national R&D&I strategy and the objectives of the European H2020 Programme and its continuation Horizon Europe 2021-2027. The interdisciplinary fusion of engineering, medicine, communication, information and education is a global objective that has been reflected in other doctoral proposals at top international institutions such as Harvard.
Finally, the commitment to the quality of research and its dissemination and scientific outreach. Besides, the Education in the Knowledge Society PhD Programme has been aligned with the Open Access / Open Knowledge / Open Science movement [714], promoting that all the educational and research resources will be available in open access on the PhD portal [15, 16] or on the institutional repositories [17-19].
The web portal enables knowledge management inside and outside the PhD Programme (https://knowledgesociety.usal.es). This tool provides an environment in which students can manage all the knowledge they generate throughout their doctoral studies. Likewise, the PhD portal gives visibility and disseminates this knowledge, so the work carried out by junior researchers has a greater impact at a national and international level.
In addition, the PhD portal allows online monitoring of the doctoral students’ progress, enabling the Academic Committee, the Quality Committee and PhD advisors to carry out periodic monitoring tasks. The doctoral students share their evidence, such as the research plan, annual reports, pre-doctoral visits, grants, publications, conferences, and another kind of activities related to their doctoral studies. 3 3.1
Before defining the design of the data visualizations to be included in the PhD portal, it is necessary to understand the requirements of the involved roles. As described in section 2, three main roles arise within the PhD portal users: doctoral student, PhD advisor and Manager/Academic Committee member.
A requirement elicitation process was carried out to capture important information requirements that must be considered during the design of the PhD portal’s information visualizations.
The detailed results and the prototypical design of the PhD portal’s visualizations can be consulted in [20]. To summarize, the main information requirements were the following: - PhD advisors: information regarding their doctoral students, including publications, conference attendance/participation, and research profiles. - Academic committee members: information related to the progress of all the doctoral students and their deadlines, as well as the number of doctoral students associated with each research group/advisor. - Doctoral students: remaining activities, distribution of activities/milestones (publications, seminars, etc.) by type, status of each milestone, deadlines, enrollment dates, comparisons with other doctoral students, etc. 3.2
The selected tool for this preliminary usability study of the integration of data visualizations into the PhD portal was the System Usability Scale (SUS) questionnaire [21]. Due to the fact that the majority of people involved in the PhD in Education in the Knowledge Society are Spanish-speaking, we employed the Spanish version of the scale [22].
The SUS questionnaire consists of 10 items rated on a 1 to 5 Likert scale (from “strongly disagree” to “strongly agree, respectively). The items are positive and negative alternated statements (to avoid response biases).
This questionnaire provides an effective, valid and reliable [23, 24] manner to rate a system’s usability. It is also an efficient test, due to the short quantity of items required to score the usability (10 items), and it can be applied over a wide range of systems [25]. In addition to the 10 items of the SUS questionnaire, we also collected a set of demographic variables, including: • Age range • Birthplace • Gender • PhD Programme role • Enrollment year • Frequency of use of the PhD portal • PhD advisors’ situation (current students being advised)
Besides these demographic variables, two open fields were provided when at the end of the survey to allow users to remark any positive and negative aspects of the visualizations. These open fields enabled us to collect qualitative feedback in addition to the quantitative measures of the SUS.
We implemented the SUS questionnaire using a customized version of LimeSurvey (https://www.limesurvey.org), an Open Source online statistical survey web application. 3.3
We sent the implemented SUS questionnaire to the PhD Programme participants (including students, PhD advisors, and managers). A total of 35 persons answered the questionnaire, which, according to the literature, provide fairly reliable results (the SUS is reliable with a minimum sample size of 12 participants [24]).
Table 1 presents an overview of the participants that took part in the usability test.
On the other hand, regarding the frequency of use of the PhD portal, most participants remarked that they use the portal weekly or monthly. These data are detailed in Table 2. The outcomes of the instrument were analyzed using the Python Pandas [26] library. The individual SUS score of each participant was computed, to finally obtain the mean of all scores (i.e., the usability score of the PhD portal visualizations).
The interpretation of the results is based on previous System Usability Scale studies and benchmarks [27, 28], which allow meaningful SUS score comparisons to provide significant insights into the study outcomes.
All the source code developed for this analysis is available at https://github.com/AndVazquez/phd-visualizations-sus. 4
The PhD portal helps to monitor the doctoral student’s activities, but even so, when the student has a lot of activity, it is very difficult to know if he/she is meeting all the necessary milestones. Based on the requirements elicitation, the portal needs to improve this monitoring support. In particular, the PhD advisors and the Academic Committee need information regarding the doctoral students’ progress. Moreover, doctoral students requested information about the distribution of milestones and remaining activities.
For this reason, a set of tools has been implemented in the portal to facilitate the monitoring of doctoral students. 4.1
Firstly, a set of tools that help doctoral students, their advisors, and the Academic Committee to know the status of the main milestones to be reached by a doctoral student. This is accomplished through a timeline displayed in the doctoral student’s profile, and it is only visible if the student is logged in the portal (Fig. 1). This timeline is complemented by a detailed view that allows students to find reports, research plans, and other milestones (Fig. 2) quickly and easily. Furthermore, the requirement elicitation process also identified the need for information about the workload of PhD advisors, including the number of doctoral students associated with each advisor.
Therefore, a set of static visualizations has been implemented to provide statistics on the advisors’ workload inside the PhD Programme. This tool is not only for the advisors but also for the Academic and Quality Committees. Although the information was already available at the PhD portal, it was not easy to get and analyze. In particular, a new tab was included in the advisors’ profile, which is only visible for themselves and the Academic and Quality Committees.
The charts show the number of theses supervised and the status of the theses (in progress, completed, failed, and doctoral students who have dropped out). Besides, there is a chart that summarizes the results in the completed theses. In addition, there are two charts related to quality indicators. Firstly, the number of doctoral grants held by the advisor’s doctoral students. Secondly, the number of seminars or workshops delivered as part of the activities organized by the PhD Programme. Fig. 3 and 4 show a real example of an advisor with the heaviest workload in the PhD Programme.
Although the SUS is set to provide a single usability score [23], subsequent research found a two-dimensional nature of this scale [29], which allows the calculation of two complementary measures: the system’s learnability score, and the system’s usability score. From the items that are part of the SUS questionnaire, items 4 and 10 can be employed to remark the learnability of the system, while the remaining items are used to obtain its perceived usability [29].
Considering this, we analyzed the results to obtain every result, including the learnability. First, the overall SUS score (considering every item), and the learnability and usability scores (taking into account items 4 and 10, and the remaining items, respectively).
These scores were transformed into a scale from 0 to 100, as in the original SUS scoring method, allowing researchers to perform comparisons.
The calculations yielded the following outcomes: • The average perceived usability of the PhD portal visualizations is 75.36, which can be considered as a good SUS score, as it is above the average SUS score (68.00) and falls around the 75th percentile (interpretation based on the studies done in [27, 28]). • On the other hand, the perceived learnability is 74.29, a slightly lower score than the usability (75.63), both being acceptable and good scores following the adjective scale of the SUS [27]. The outcomes of the usability study showed that the information visualizations included in the PhD portal were appreciated by the students as well as by the PhD advisors and members of the Quality Committee.
Specifically, from the 35 answers collected using the SUS questionnaire, we obtained an average perceived usability of 75.36, which is a “Good” score according to the SUS adjective ratings [27]. Learnability and Usability scored are interpreted in the same way.
The positive and negative feedback collected through open text fields shed more light on these results and complemented them to provide a more detailed picture of the included data visualizations’ implications. Most of the positive comments remarked on the ease-of-use of the data visualizations as well as their design, versatility, and engagement that these visual tools provide.
One specific comment pointed out that “although it can be obvious that each person knows their situation [in the PhD program], providing a timeline where each [PhD] milestone is visually displayed allows the organization and planning of the remaining time within the PhD program”. This was one of the main motivations to include data visualizations in the PhD portal; visualizations are powerful tools to understand data at first sight, which fit perfectly as tools to tackle the challenge of providing a big picture of the achievements and learning outcomes during a PhD.
In fact, as introduced before, these visualizations were not only designed for students, but also for managers. These roles also rated high the new functionality of the PhD portal. However, there are some limitations to this approach. One member of the Quality Committee commented that “statistics and graphs are based on data that students upload themselves, and this upload task is not always carried out properly”.
The effectiveness of the data visualizations is tightly coupled with the quality of the data they display. If a doctoral student does not update his or her progress, then the visualizations will not be useful at all, because there will not be meaningful information to show. However, another reason to include this functionality in the PhD portal was to improve the engagement of doctoral students with the portal, and to motivate them to keep their progress up to date. We also plan to research the impact of the data visualizations on the students’ initiative to upload their achievements into the portal.
To sum up, the application of the SUS test provided very useful insights about the usability of data visualizations to follow the doctoral students’ progress and PhD advisors’ workload. The SUS placed the system in a good range of usability, with room for improvement based on qualitative feedback. 7
A set of data visualizations have been included in the PhD portal of the Doctoral Programme in Education in the Knowledge Society. These data visualizations have several goals: to provide a big picture of the different reached milestones and achievements of doctoral students and to visualize the workload of PhD advisors in a straightforward way for better decision-making. Moreover, these visualizations also aim at improving the engagement of the involved actors with the PhD portal.
The System Usability Score was carried out to obtain insights about the usability of the data visualizations. It is important to remark that this test is not diagnostic; it gives an overview of the usability of a system. The outcomes of the study provided an average score of 75.36, which is a score above the average (68) and considered a good result.
Our implementation of the SUS also included open fields in the questionnaire. These comments provided valuable qualitative feedback from the participants, which allows us to mark our path to further improve the visualizations. Also, it provided feedback about the general thoughts regarding the newly added functionality, being most of them positive.
Future works will be focused on solving some technical problems detected by some participants during the usability test, as well as carrying out more studies to keep improving the visualization components. On the other hand, we also plan to add more functionalities (filters, advanced interactivity, etc.) with the goal of improving decisionmaking processes related with the PhD Program management. Acknowledgements. This research work has been carried out within the University of Salamanca PhD Programme on Education in the Knowledge Society scope (http://knowledgesociety.usal.es) and was supported by the Spanish Ministry of Education, Culture and Sport under a FPU fellowship (FPU014/04783). 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25.