Implementation and Evaluation of a Trusted Learning Analytics Dashboard

Abstract

The research described in this article covers the user-facing components of a learning analytics environment which is developed with the premise of trust as an essential factor for the adoption of learning analytics. It investigates the influence of privacy settings and personalization on the acceptance and adoption of learning analytics. By ensuring compliance with data protection legislation, and by providing transparency in the means and results of data collection, we aim to reduce doubts and fears in the learning analytics process. By respecting the needs of individuals, we hope to create an environment where learning analytics is perceived as something positive.

Introduction

Trust, or rather the lack of trust in analytics processes is a common topic in data analytics [ 1 ]. For the field of Learning Analytics (LA), trust is a critical factor for the acceptance by the stakeholders [ 2 ]. Building of trust has many layers, from the integrity and quality of the data sources, over the secure storage and processing, to the e↵ectiveness of the analytics results, the stakeholders have to trust that what happens is in their best interest. In addition to this moral and ethical viewpoint, the requirement to be transparent to users is further substantiated by the European General Data Protection Regulation (GDPR), which grants users far-reaching rights with regard to data analytics processes

At our institution, a large German university with primarily live lectures, there is currently no LA environment in place, and the task of our group is to establish it. Our assumption is that LA can only e↵ectively assist in the learning process when the learner actually accepts and utilizes the o↵erings. We therefore decided to follow an approach to LA that we call ”Trusted Learning Analytics” (TLA). With TLA, we focus strongly on the users and give them the information and the tools to control what, how and why their data is collected and analyzed.

As the interface between the learner and the LA approach is often a Learning Analytics Dashboard (LAD), this is also the location where the trust-building measures will be implemented and evaluated. The research question that I will attempt to answer in my PhD project is: ”How can we design and evaluate a trusted learning analytics dashboards?”

Fields of Research

From the gathering of the data, through the storage and processing, to the presentation of the results, there are multiple aspects where trust of the learners has to be won. In my PhD, the focus is on the user-facing side, while the other parts (e.g. storage and infrastructure) are handled by other members within the same research group. My research will depend on and interact with those other parts. 2.1

Establishing a Learning Analytics Environment

Our working group was established at the same time that I started my PhD, in November 2017. Thus, there was no LA environment in place, and we have to create it from the start.

We see this as a big opportunity, as we can choose which approaches to implement in our environment. While trying to get an overview of the field, we noticed that there are very di↵erent approaches to LA and it was not clear which ones would fit best for our case. There is literature that gives an overview of the LA landscape in general (e.g. [ 3 ]), and especially on the topic of LADs (e.g. [ 4 ] and [ 5 ]), yet the literature only provided pointers to what approaches exist, but only in very general categories and without information on where to find the results.

We conducted a literature review on the approaches that are researched in the field of LA. We identified a heterogeneous combinations of various components that form those LA approaches and found that the nature of the results warrants more than simply listing them. In order to make the results of the review continually explorable, we created a web-based tool, which we call the Learning Analytics Indicator Repository (LAIR). In the LAIR, the approaches and the respective papers are listed, and their components (subjects, platforms, activities, metrics, indicators) are visualized in directed graphs. With the LAIR, we hope to achieve two goals with regard to TLA: 1) Being able to quickly find approaches and the related literature for specific combinations of components. 2) A visualization of LA approaches which plan to use in the LADs to explain the learners how their data is gathered and for what purpose. 2.2

Privacy, Transparency and GDPR Compliance

Ethics and Privacy as factors for the acceptance and adoption of LA have been the subject of numerous publications. There are several policy policy papers (e.g. [ 6 ] ,[ 7 ]), and user studies which show that stakeholders have high expectations for privacy and are not willing to consent to indiscriminate sharing of their data [ 8 ].

This need for privacy is underpinned by the GDPR regulation [ 9 ]. For GDPR compliance, the state of research is theoretical in nature (e.g. [ 10 ]), and there is little precedence in how the GDPR can be applied in LA practice (there exist some commercial variants, e.g. the privacy dashboard by Microsoft 1). The combination of the self-imposed ethical and an external legal demand for privacy brings requirements for the user interface components. Especially the learner-facing LADs should provide an interface that gives the users easy and understandable control over their data. In this area, I expect my research to align more with areas such as User Interface and Privacy research than with TEL.

We have currently mapped the GDPR user rights to functionalities and respective user interface components. I am currently implementing those in a LAD. Using a design-based research methodology [ 11 ], we will roll out a prototypical LAD first in a small seminar where we plan to receive feedback using usability studies (Surveys, Eyetracking, Think-Aloud). From this initial feedback, we will improve on the LAD and use it in larger live lectures (100-200 students), where we will further research how the users interact with the privacy settings by gathering usage statistics. 2.3

Personalization

Interviews of learners have also shown that they expect adaptive and personalizable LADs [ 8 ] [ 12 ]. In combination with the privacy expectations and the varying degrees of willingness to share data between individuals, we derive that LA should o↵er a high degree of transparency and personlization to gain more widespread acceptance.

Dashboards in the TLA environment therefore require widgets that can be configured by the learners to their individual needs. One particular issue that we see is the di↵erence in perception with regard to peer referenced LA [ 8 ] [ 12 ]. Widgets in the TLA environment should therefore ideally support both those who are motivated by the aspect of peer reference, and those who see this as a rather demotivating factor. We aim to create widgets which allows individuals to opt-out of peer reference. After opting-out, they would only see their own data and not those of others and others would no longer see the opted-out individuals, except in aggregated form.

In a first iteration, we plan to create an essay-writing widget to be apply this concept in smaller seminars. There, the learners can a) plan their intended progress on the various parts of an essay b) track their total progress. In addition to their own values, they will also see the planning and progress of the other learners in the course that have opted-in to being peer-referenced. We want to investigate how and why learners choose to opt-out of this and how they progress in comparison to those who leave the peer-referenced aspects turned on. 3

Current Progress

I have completed the first half year of the PhD program which is currently set to three years. At the time of this writing, I have completed the literature 1 https://account.microsoft.com/account/privacy review on the LA approaches and implemented the web-based repository. For the TLA dashboard, several tasks are completed and a privacy interface has been implemented on the frontend, which needs to be connected with the rest of the infrastructure. The first usability experiments are set for the fall this year.

References

[1] Analytics , K. I. D. \.: Building Trust in Analytics. Company Whitepaper. KPMG International Data & Analytics.

[2] Greller , W. and Drachsler , H.: “Translating Learning into Numbers: A Generic Framework for Learning Analytics” . In: Educational Technology & Society 15 ( 2012 ), pp. 42 - 57 .

[3] Moissa , B. , Gasparini , I. , and Kemczinski , A.: “A Systematic Mapping on the Learning Analytics Field and Its Analysis in the Massive Open Online Courses Context” . In: Int. J. Distance Educ. Technol. 13.3 (July 2015 ), pp. 1 - 24 .

[4] Schwendimann , B. A. , Rodriguez-Triana , M. J. , Vozniuk , A. , Prieto , L. P. , Boroujeni , M. S. , Holzer , A. , Gillet , D. , and Dillenbourg , P.: “ Perceiving Learning at a Glance: A Systematic Literature Review of Learning Dashboard Research” . In: IEEE Transactions on Learning Technologies 10.1 ( 2017 ), pp. 30 - 41 .

[5] Bodily , R. and Verbert , K. : “Review of Research on Student-Facing Learning Analytics Dashboards and Educational Recommender Systems” . In: IEEE Transactions on Learning Technologies 10.4 ( 2017 ), pp. 405 - 418 .

[6] Drachsler , H. and Greller , W.: Privacy and analytics: it's a DELICATE issue a checklist for trusted learning analytics . In: ACM Press, 2016 , pp. 89 - 98 .

[7] Sclater , N.: “Developing a code of practice for learning analytics” . In: Journal of Learning Analytics (Apr . 2016 ), pp. 16 - 42 .

[8] Ifenthaler , D. and Schumacher , C.: “Student perceptions of privacy principles for learning analytics” . In: Educational Technology Research and Development 64 .5 ( Oct . 2016 ), pp. 923 - 938 .

[9] European Parliament , C. o. t. E. U.: “Regulation (EU) 2016 / 679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data , and repealing Directive 95 /46/EC ( General Data Protection Regulation)” . In: Ocial Journal of the European Union L119 (May 2016 ), pp. 1 - 88 .

[10] Hoel , T. , Griths , D. , and Chen , W.: The influence of data protection and privacy frameworks on the design of learning analytics systems . In: ACM Press, 2017 , pp. 243 - 252 .

[11] Wang , F. and Hannafin , M. J.: “ Design-based research and technologyenhanced learning environments” . In: Educational Technology Research and Development 53 .4 ( Dec . 2005 ), pp. 5 - 23 .

[12] Tan , J. P.-L. , Koh , E. , Jonathan , C. , and Yang , S.: “ Learner dashboards a double-edged sword? Students' sense-making of a collaborative critical reading and learning analytics environment for fostering 21st century literacies” . In: Journal of Learning Analytics 4.1 ( 2017 ), pp. 117 - 140 .