Data Storytelling for Feedback Analytics

Abstract

Feedback is an essential process of learning in higher education. Yet, capturing students' interactions with feedback is challenging, which makes it dificult to evaluate its impact. Learning Analytics (LA) is a potential solution to address this issue as it is capable of capturing and analysing learners' activities in a technology-enabled learning environment. LA often use dashboards to deliver insights derived from educational data, yet questions remain on how to most efectively communicate key insights to students. Data Storytelling (DS) is a promising technique to address this challenge by combining data, visuals and narrative to convey key insights. Co-design can facilitate the crafting of visualisations and data stories that best aligns with goals of the students. This study presents the preliminary findings from a design sprint conducted with students to co-design a prototype for a dashboard of an LA solution PolyFeed - that captures and analyses students' interactions with feedback. In developing the dashboards, students used DS principles - Explanatory titles, Annotations, Highlighting important data points, and Decluttering - to improve the selected visualisations. The results show that the student groups perceived visualising strengths and weaknesses identified in feedback, action plans based on feedback, and trends in their performance as key aspects to include in FA dashboard. However, they primarily used two DS principles: explanatory titles and highlighting key data points to improve visualisations because the dataset was pre-dominantly qualitative. Therefore, the efective use of DS to support qualitative data should be further explored.

Keywords

• ability

-

CEUR ceur-ws.org

1. Introduction & Background

Feedback is an important process of learning in higher education. Yet, monitoring how students engage with, interact with and act on feedback (traceability) is challenging [ 1 ]. Learning Analytics (LA) is a potential solution to enhance the efectiveness of feedback by closing the feedback loop through capturing and analysing students’ interactions with feedback. LA derives insights from educational data and delivers them to learners through student-facing LA dashboards to facilitate the feedback process [ 2 ]. However, developing efective LA dashboards is often constrained by students’ lack of expertise with data and visualisation literacy to appropriately interpret the insights from the dashboard [ 3, 4 ]. Based on this challenge, Winstone [ 1 ] emphasised that the use of simple analytics in student-facing analytics dashboards is a safer choice to better influence students’ actions based on feedback. But, the use of simple visualisation may not capture the full impact of feedback due to its limitation in tracking students’ consistent strengths, repeated mistakes, or areas of confusion.

Data Storytelling (DS) is a promising technique that can assist students with varying levels of data literacy in their interpretation of visualisations by combining data, visual, and narrative elements in order to guide non-data experts through a data story that conveys the main insights [ 5 ]. As noted by Bach et al. [ 6 ], there are many ways that visual and narrative elements can be applied in order to tell an efective data story. For instance, a title that clarifies the main insights of the visualisation (Explanatory titles) [ 7 ] and the addition of textual elements to explain particular data points (Annotations) can improve the reader’s understanding of the visualisation by adding context [ 5 ]. Moreover, the use of pre-attentive attributes (e.g., colour and contrast) can also direct the attention of the audience towards particular data points or regions of the visualisation (Highlighting important data points) [ 5 ]. Lastly, removing superfluous information or visual noise (e.g., grid lines and labels) that distracts from the main data story can improve the clarity of the visualisation (Decluttering) [ 7 ].

In designing an LA dashboard supported with DS principles, it is important to co-design with students to capture how they want to view the dashboard in a way that feels authentic to them [ 1 ]. In LA literature, the co-design approach has been implemented to engage and support educational stakeholders by ensuring that the presentation of LA aligns with their goals [ 8 ]. By doing so, it enables us as LA designers to identify ways to communicate insights from data efectively [ 9 ] to diferent groups of students. DS is goal oriented in the sense that data stories and visualisations need to be crafted with a specific purpose [ 10 ]. Therefore, the co-design approach is appropriate as it informs designers as to the DS features and chart types that align best with the goals of the students.

This study presents the preliminary findings of designing a DS-enhanced dashboard with students for an LA solution, PolyFeed, which designed and developed to capture and analyse students’ interactions with feedback. PolyFeed was designed to enhance feedback process through annotate feedback, create action plans and visualise interactions with feedback functionalities. We defined Feedback Analytics (FA) as the capture, analysis, and data-based presentations of students’ interactions with feedback. This preliminary study aims to build upon existing LA literature by exploring how students from diferent educational disciplines: i) perceive efective visualisations depicting FA; and ii) choose DS principles to best support efective interpretation of student-facing FA dashboards. Thus, this study is based on the following research questions. • RQ1: What are the efective ways to visualise FA from students’ points of view? • RQ2: How did students use DS principles to design an FA dashboard?

2. Methodology

This study is inspired by the design sprint methodology [ 11 ] and was conducted via a two-hour design sprint covering five key elements – Understand, Diverge, Decide, Prototype, and Test. Thirteen higher education students from seven diferent faculties in a large Australian university volunteered for this design session. They were grouped into three groups based on faculty, degree program, and current year of study. The participants were introduced to the features of PolyFeed and the data collected through the tool. The dataset included quantitative and qualitative data. The quantitative data included assessment marks for each assignment, and students’ ratings on received feedback. The qualitative data comprised labeled feedback extracts (‘Strengths,’ ‘Weaknesses,’ ‘Action Points,’ ‘Confusion,’ and ‘Other.’), personal notes, to-do lists with deadlines, and further clarifications obtained using generative AI agent (e.g., ChatGPT). Using the provided dataset, students brainstormed questions they wanted to answer using the FA dashboard (Understand). Students individually sketched data visualisations they perceived as suitable for the selected questions (Diverge) referring to two key resources (e.g., Financial Times Visual Vocabulary1, and Power BI Visual Vocabulary2) and collectively prioritised through group discussion and voting (Decide). In the prototype phase, students created a paper-based prototype that included these prioritised visualisations. Each group then further improved their initial design by incorporating data storytelling principles -Explanatory Title, Annotation, Highlighting important data points, and Decluttering- [ 5 ] introduced during the workshop (Prototype). Each of these principles were introduced with an explanation and a scenario based example. Lastly, each group presented their dashboard design to other groups and collected peer feedback (Test).

This paper presents the preliminary findings of the prototype phase. The co-design sessions was recorded and transcribed according to each step: Ideation, Exploration, Initial Design, DS-Enhanced Design, and Retrospective. The transcripts were then analysed alongside the artefacts created by students. In this preliminary analysis we analysed the dashboard design with and without DS elements. In finding answers for RQ1 we conducted a group discussion where we inductively identified the topics and diferent types of visualisations students included into their dashboards. Subsequently, in finding answers for RQ2 we compared the visualisations in the initial design with DS incorporated design to identify the DS principles students used in designing their dashboard.

3. Preliminary Results

3.1. RQ1: What are the efective ways to visualise FA?

Students primarily wanted to include four key aspects of FA in the dashboard: (1) an overview of strengths and weaknesses identified in their feedback, (2) an overview of to-do lists based on feedback, (3) mark comparisons with the class, and (4) individual performance within the course. In visualising these aspects, the groups utilised a variety of diferent data visualisations. For instance, group one used a pie chart while group two used a word cloud (Figure 1 - a) to visualise the overview of strengths and weaknesses (unit-specific and across units or assignments). In addition, to visualise the overview of to-do lists, groups one and three used a non-traditional data visualisation such as a checklist that supports prioritisation and records the completion of the to-do list (Figure 1 - b). In visualising the mark comparisons, group two used a Gaussian 1https://on.ft.com/3vN38YC 2https://bit.ly/48IWXDH curve (showing mark distribution), and group three used a bar graph with distribution bars (Figure 1 - c). Lastly, in visualising individual performance, group two used line graphs (showing trends across assignments) (Figure 1 - d), and group three used a radar graph.

3.2. RQ2: How did students use DS principles to design an FA dashboard?

The most common DS principle that student groups incorporated into their DS-enhanced design was highlighting important data points. Specifically, students would leverage pre-attentive attributes such as colour or size to emphasise what they believed to be salient information in their dashboard. As shown in Figure 1, group two used green for strengths and red for weakness in their word cloud (Figure 1 - f), while group one included red and green data points in the line graphs to highlight highs and lows in the individual performance (Figure 1 - i). Explanatory titles were also leveraged by groups two and three to their mark distribution graph (e.g., Gaussian curve, bar graph with distribution bars) (Figure 1 - h). Annotations and decluttering were less commonly observed in the final designs. Group three designed a ‘decluttered’ prototype by removing the bars showing individual students’ marks and instead including a tick into the mark distribution to indicate a particular students’ position in the class (Figure 1 - h).

4. Discussion and Conclusion

This study sought to examine how to efectively tell data stories relating to feedback by giving the opportunity to higher education students to co-design a FA dashboard with DS features.

In response to RQ1, What are the efective ways to visualise FA from students’ points of view? , we found that students expected to see an overview of strengths and weaknesses and action plans (i.e., to-do lists), mark distribution across the class, and the trends in their performance. In visualising these aspects, students selected both common (e.g., pie charts and line charts) and uncommon data visualisations (e.g., radar graphs and custom visualisations). It was suggested by Knaflic [ 5 ] that a preference for familiar data visualisations may result from a perception that less conventional visualisations are too complicated. However, as students also selected uncommon data visualisations from the supplied reference sheet, this indicates that students may be receptive to the presence of a wider range of visualisations within FA dashboards, as long as they are appropriate to communicate the insights [ 1 ]. Therefore, in designing efective visualisations of FA, LA designers should be conscious of the diverse range of visualisations that can be employed to reach a wider audience while also ensuring that the intended message of the visualisations is being clearly conveyed. Doing so, we can ensure that we appropriately balance the presentations of the FA dashboard meets its’ objective while catering to the needs of diferent student groups (e.g., disciplines, and performance). Moreover, we can provide diferent types of visualisations for one aspect (e.g., presenting common strengths and weaknesses through word clouds, lollipop graphs, and grouped bars) and students can select the most efective visualisation for them.

In response to RQ2, How did students use DS principles to design an FA dashboard?, we found that students primarily used two DS principles: Highlighting important data points and Explanatory titles. Some groups used these principles individually to improve the graphs, while others combined them. Knaflic [ 5 ] emphasised that incorporating pre-attentive attributes (e.g., colour and contrast) sparingly to textual content can improve the text. In this study, the students leveraged colour to emphasise the key point within their explanatory title. Students also combined multiple pre-attentive elements when applying the DS principle of highlighting important data points. For instance, they used colour and size (Figure 1-f) or colours and shapes (Figure 1-i) together. However, as the purpose of pre-attentive attributes are to assist in processing salient information in milliseconds [ 5 ], the inclusion of multiple pre-attentive attributes on the dashboard interface may obscure each other and make it dificult to distinguish the key insights within a short time. In addition, we observed that the students rarely used the DS principles Annotation and Decluttering. A possible reason for this could be because of the qualitative nature of the dataset (i.e., the PolyFeed dataset contains feedback extractions, labels, to-do lists and explanations from Generative AI agent) and the lesser complexity of the quantitative data. There are a wide range of data visualisations for quantitative data – yet, utilising them to visualise qualitative data is challenging because they are used for a purpose other than the original intention of the visualisation. In addition, the DS principles used in this study are useful to emphasise the key points of quantitative data. However, in the context of qualitative data, we believe that the efective use of DS should be further explored.

In conclusion, the co-design approach enabled further understanding into what students believe are efective ways to visualise FA using DS principles. The findings of this study have provided insights into how Polyfeed designers can successfully incorporate DS principles in the next phase of the FA dashboard’s development. However, as DS principles are generally applicable to quantitative data, further investigations are necessary to see how DS can best support FA given the predominantly qualitative nature of FA datasets.

Acknowledgments

This research is funded by Monash University educational innovation funding (The Pitch) and Advancing Women’s Success Grant. We would also like to thank the participants for generously contributing their time and valuable insights.

References

[1] N. Winstone , Facilitating Students' Use of Feedback: Capturing and Tracking Impact Using Digital Tools , in: The Impact of Feedback in Higher Education , Springer International Publishing, Cham, 2019 , pp. 225 - 242 .

[2] W. Matcha , N. A. Uzir , D. Gasevic , A. Pardo , A Systematic Review of Empirical Studies on Learning Analytics Dashboards: A Self-Regulated Learning Perspective , IEEE Transactions on Learning Technologies 13 ( 2020 ) 226 - 245 .

[3] V. Echeverria , R. Martinez-Maldonado , S. Buckingham Shum , K. Chiluiza , R. Granda , C. Conati , Exploratory versus explanatory visual learning analytics: Driving teachers' attention through educational data storytelling , Journal of Learning Analytics 5 ( 2018 ) 73 - 97 .

[4] S. Pozdniakov , R. Martinez-Maldonado , Y.-S. Tsai , N. Srivastava , Y. Liu , D. Gasevic , Single or multi-page learning analytics dashboards? relationships between teachers' cognitive load and visualisation literacy , in: O. Viberg , I. Jivet , P. Muñoz-Merino , M. Perifanou , T. Papathoma (Eds.), Responsive and Sustainable Educational Futures , Springer Nature Switzerland, Cham, 2023 , pp. 339 - 355 .

[5] C. N. Knaflic , Storytelling with Data: A Data Visualization Guide for Business Professionals , John Wiley & Sons, Inc., Hoboken , New Jersey, 2015 .

[6] B. Bach , D. Stefaner , J. Boy , S. Drucker , L. Bartram , J. Wood , P. Ciuccarelli , B. T. Yuri Engelhardt , U. Köppen, ” Narrative Design Patterns for Data-Driven Storytelling” , CRC Press (Taylor & Francis), New York, 2018 , pp. 107 - 133 .

[7] D. Zdanovic , T. J. Lembcke , T. Bogers, The influence of data storytelling on the ability to recall information , in: Proceedings of the 2022 Conference on Human Information Interaction and Retrieval , CHIIR '22, Association for Computing Machinery, New York, NY, USA, 2022 , p. 67 - 77 .

[8] D. Gašević , Y.-S. Tsai , H. Drachsler , Learning analytics in higher education - stakeholders, strategy and scale , The Internet and Higher Education 52 ( 2022 ) 100833 . URL: https: //www.sciencedirect.com/science/article/pii/S1096751621000427. doi:https://doi.org/ 10.1016/j.iheduc. 2021 . 100833 .

[9] Carlos G. Prieto-Alvarez , Roberto Martinez-Maldonado, Theresa Dirndorfer Anderson, Codesigning learning analytics tools with learners, in: Learning Analytics in the Classroom, 1st edition ed ., 2018 , pp. 93 - 110 .

[10] R. Martinez-Maldonado , V. Echeverria , G. Fernandez, S. Buckingham Shum , From data to insights: A layered storytelling approach for multimodal learning analytics , 2020 , pp. 1 - 15 . doi: 10 .1145/3313831.3376148.

[11] C. M. Mendonça de Sá Araújo , I. Miranda Santos , E. Dias Canedo , A. P. Favacho de Araújo , Design Thinking Versus Design Sprint: A Comparative Study , 2019 , pp. 291 - 306 .