=Paper=
{{Paper
|id=Vol-1841/R06_117
|storemode=property
|title=How Educators Value Data Analytics about their MOOCs
|pdfUrl=https://ceur-ws.org/Vol-1841/R06_117.pdf
|volume=Vol-1841
|authors=Konstantinos Michos,Davinia Hernández-Leo,Manel Jiménez
|dblpUrl=https://dblp.org/rec/conf/emoocs/MichosHJ17
}}
==How Educators Value Data Analytics about their MOOCs==
https://ceur-ws.org/Vol-1841/R06_117.pdf
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
How educators value data analytics about their MOOCs
Konstantinos Michos1, Davinia Hernández-Leo1, Manel Jiménez2
ICT Department1, Communication Department2, Universitat Pompeu Fabra, Barcelona, Spain
{kostas.michos,davinia.hernandez-leo,manel.jimenez}@upf.edu
Abstract. A range of data analytics is provided to educators about the profile,
behavior and satisfaction of students participating in a Massive Open Online
Course (MOOC). However, limited research has been conducted on how this
informs the redesign of next MOOC editions. This work-in-progress paper pre-
sents a study of 4 MOOC educators from Universitat Pompeu Fabra regarding 3
MOOCs offered on the FutureLearn platform. The objective was to evaluate the
usefulness and understandability of different types of data analytics of the
courses they have offered with respect to specific monitoring goals. Preliminary
results show that educators perceived the same information sources and data
visualizations differently, satisfaction surveys and comments in the forum were
among the most useful information but it was difficult to associate data analyt-
ics with the monitoring goals. Further studies for the alignment of educators´
monitoring needs for redesign purposes and the development of appropriate
support tools are suggested.
Keywords: educators in MOOCs, data analytics, redesign, monitoring goals
1 Introduction
Recent research in the use of data analytics in Education proposes the presentation of
valuable information to students and educators for making informed decisions in the
design and use of digital learning environments. For instance, by being aware of their
actions, students can have a better control over their own learning and they can asso-
ciate the presented data to achieve learning goals. In the context of Massive Open
Online Courses (MOOCs), educators are in the position to reflect on their course with
a variety of data sources. This implies to monitor the learning process, to identify
difficulties and problems and to improve the learning environment [1]. However, the
lack of face to face interactions between students and teachers, as opposed to conven-
tional courses, hinders their communication and it becomes difficult to assume in
design time how the massive amount of students will respond to certain learning ac-
tivities. One approach is to analyse the learning design of the MOOC so that the pro-
vided data about students can be interpreted towards course redesigning purposes [2].
Common information sources in MOOCs are students´ profiles and their previous
experiences, clickstreams with the videos, patterns of students´ activities, and com-
ments in forums and students´ surveys [3]. Recent paradigms of MOOC dashboards
aim to display this information to teams of educational practitioners who are involved
77
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
in the development of the MOOC towards supporting their evidence based decision
making [4]. However, in the specific case in which the MOOC has finished, less re-
search addresses the usefulness of the provided data analytics for the redesign purpos-
es of courses. Studies in distance and online learning indicated that educators value
data insights about the performance of students and their common misconceptions,
the material viewed according to the schedule and the forum behavior [5-6]. One
survey study evaluated the opinion of 92 MOOC educators for the usefulness of
common information sources and their association with specific monitoring goals [7].
The results indicate that the most useful information sources were the discussion fo-
rums but quantitative measures were insufficient to identify problems and potential
improvements in the MOOC. A limitation of this study is that educators were not
equipped with the real data about their students but with synthetic data visualizations
that could potentially inform the development of information display tools for educa-
tors.
In this paper, we focus on the FutureLearn platform which is based on a social
constructivist approach which aimed at promoting social learning within MOOCs.
Special attention was given towards the role of educators and how they can improve.
For the facilitation of course evaluation, MOOC educators were provided with three
summary reports before and after their course including: 1) a pre-course survey which
includes data about the previous experiences and motivations of students to partici-
pate in the course; 2) a course report which shows the cumulative growth of enrol-
ments and panel data analysis for the activity patterns of students, their comments,
responses in quizzes and tests, and 3) a post-course survey summary that presents the
satisfaction of students with the course, educators and the platform. The two surveys´
summaries were visualized in the form of bar charts complemented with the related
data tables. The course report included a variety of visualizations (see Figure 1).
Figure 1. Sample visualizations in the FutureLearn course-report showing en-
rolment cumulative growth and analysis of activity, comments and quizzes and
tests during the course.
78
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
The aim of this paper is to evaluate educators’ opinions about the specific summary
reports provided by FutureLearn and to understand their usefulness to identify ideas
in an informed redesign of the course. That end, we analyzed the usefulness of this
range of data for specific monitoring goals [7] like diagnosing problems with tasks-
activities. Our research questions are:
RQ1: Which information sources and visualizations (from FutureLearn reports)
are most useful for MOOC educators?
RQ2: What information sources (from FutureLearn reports) help MOOC educators
to identify problems and potential improvements in a redesign of the
course?
2 Methodology
A survey study was carried out to evaluate the usefulness and understandability of
these three summary reports from MOOC educators with respect to the redesign of
next MOOC editions. The survey was based on the study of Stephens-Martinez,
Hearst & Fox [7] who evaluated the opinions of 92 MOOC educators. The authors
formulated certain monitoring goals as common quantitative or qualitative assessment
of educators in a MOOC which becomes relevant in our case for course redesign pur-
poses. The monitoring goals were related to the learning activities, the material and its
appropriate presentation including: 1) Problems with the activities-tasks, 2) With what
students struggle, 3) Appropriateness of course difficulty, 4) Most difficult part of the
course, 5) Improving the presentation of a topic, the students´ engagement including:
6) Engaging content for students, 7) Least interesting content and the grading tasks in
MOOCs including: 8) Difficulty of the grading activities.
The procedure which we followed consisted of two parts. At first, we searched for
educators who offered MOOCs at Universitat Pompeu Fabra and we selected the
courses whose first editions were already completed and all data report summaries
were available. We sent an e-mail to the MOOC educators involved in these courses
with an explanation of our study, the three summary reports about their own MOOC
and a questionnaire. The questionnaire, based on the approach followed by [7], in-
cluded a first part with demographic information about the educators and their previ-
ous teaching experience in MOOCs. The second part included questions about the
usefulness and understandability of the information sources and the basic visualiza-
tions provided in the three reports followed by open-ended questions. In the third part,
the educators were asked to repeatedly respond on the usefulness of the information
sources for the 8 monitoring goals followed by open-ended questions. To simplify the
three summary reports and to present some visualizations in the questionnaires we
divided them into the following segments with specific focus on the course summary
report: Pre-course survey: a) previous experiences of students; Course report: b) en-
rolment cumulative growth, c) activity patterns by step, d) comments by step, e) quiz-
zes and tests; Post-course survey: f) students’ satisfaction.
79
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
3 Results
All MOOC educators (N=4) had experiences in the development of a MOOC as they
have already finished two editions of their own course. The titles of the three MOOCs
were the following: “3D Graphics for Web Developers”, “Why the European Union?
A brief history of European integration”, “Introduction to Catalan Sign Language:
Speaking with your hands and hearing with your eyes”.
Educators indicated that students´ satisfaction was the most useful information
source, although all the information sources were appreciated with ratings over medi-
um or somewhat useful. In some cases like (a) the previous experiences of students
and (e) the analysis of quizzes and tests, the opinion of MOOC educators was differ-
ent ranging between low vs. high usefulness or uncertainty for their usefulness. The
easiest to understand and interpret were the (a, f) surveys´ summaries while some
difficulties were present for the provided visualization in the (c) activity patterns by
step and the (e) quizzes and test. Further, the responses of the 4 MOOC educators
differed (e.g. different responses for the understandability of visualizations for the
activity patterns of students).
With respect to the monitoring goals, the results show limited application of the
information sources as in each case the 34% to 60% of them were not applicable and
the 22% to 28% of the responses were neutral meaning they were not sure how to
associate them with monitoring goals (see Figure 2 pattern fill and light color). Figure
2 shows that (f) student’s satisfaction and (d) comments by step were perceived more
useful information for the monitoring goals (dark color) compared to the other
sources as they were strongly appreciated in 41% of the monitoring goals.
Usefulness of the information sources for all the monitoring
goals
22 41 34
Information sources
d. Quizzes and tests 28 28 38
22 41 34
c. Activity by step 28 19 44
22 0 59
a. Previous Experiences of students 22 25 47
0% 20% 40% 60% 80% 100%
Percent of educators responses (n=32)
Neutral Agree Strongly agree Not applicable
Figure 2. Usefulness of each of the information sources for all the monitoring goals.
Figure 3 shows further the 8 monitoring goals (numbers from 1-8) and the useful-
ness of all the information sources. It was more difficult for educators to associate the
provided data analytics with (3) the difficulty of grading activities and (8) the least
interesting content as all the information sources were not applicable for more than
80
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
55%. The information sources were strongly appreciated in about 35% in issues like
(4) the appropriateness of the course difficultly, (6) the most difficult part of the
course, (2) with what students struggle and (5) the most engaging content of the
students.
Usefulness of the information sources for the monitoring
goals
25 13 63
Monitoring goals
7. Improving the presentation of a topic 25 21 38
29 33 33
5. Engaging content for students 29 29 29
25 38 29
3. Difficulty of the grading activities 25 58
29 33 29
1. Problems with the activities-tasks 29 17 42
0% 20% 40% 60% 80% 100%
Percent of responses (n= 24)
Neutral Agree Strongly agree Not applicable
Figure 3. Usefulness of all the information sources for each particular monitoring goal.
In the open-ended questions, participants mentioned that all the graphs which in-
cluded time as a variable were particularly useful and the set of data tables provided
in combination with the visualizations were appreciated. Second, visualizations about
the active learners were considered in the development of the next edition of the
MOOC as well as the attempts to correct answers in the quizzes helped to re-design
the quizzes for the next edition. The participants pointed out that all the information
sources were considered in the next edition of the MOOC but as complements. The
final decisions for re-designing of their MOOC were taken after discussions carried
out with the team of educators involved in the production of the MOOC and Future-
Learn platform. According to two educators, some changes in the next editions based
on the provided data were to simplify the content of the MOOC and to intervene in
specific parts of the course which were determined as important from the educators´
team. Finally, one of the educators proposed, further visualizations might help them in
their final decisions. For instance, interaction analysis of participants based on their
comments and the specific words in the provided sentiment analysis from Future-
Learn will provide more insights.
4 Conclusions and Future Work
Results indicated that educators perceive specific information sources useful, with
some divergences in some of their opinions. Further studies need to evaluate larger
samples of educators and how their background and the different contexts (topics,
learning design, and audience characteristics) of their courses influence the interpreta-
81
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
tion of the provided data analytics. Second, preliminary results shows that educators
were struggling to associate the application of the information sources to the monitor-
ing goals which might shows a gap between the educators´ needed information and
the provided data analytics reports. The most applicable information sources for the
presented monitoring goals were students´ satisfactions and comments in the forums,
which is aligned with the results in [7]. Future studies should also consider monitor-
ing goals defined by the educators before the collection and presentation of data. Fi-
nally, educators reckoned that information sources and visualizations were useful as a
complement for determining the changes in the next edition as they took more into
account the observations and conclusions by the team of MOOC educators in charge
of the MOOC and the advices from the MOOC platform. Limitations of this study
include a less number of MOOC educators and relatively a low number of participants
in the MOOC survey reports. However, this study overcame a limitation of the previ-
ous study in [7] by presenting to MOOC educators analytics of real data, from learn-
ers in their own MOOCs. Further research should also study the specific actions in the
redesign of a MOOC that educators perform as a result of the awareness provided by
the data analytics of past cohorts of MOOC students.
Acknowledgments. This research is partly funded by RecerCaixa and the Spanish Ministry
of Economy and Competitiveness under RESET (TIN2014-53199-C3-3-R) and the Maria de
Maeztu Units of Excellence Programme (MDM-2015-0502). DHL is a Serra Hunter Fellow at
UPF. Authors want to also thank Ishari Amarasinghe and all the educators who participated in
this study.
References
1. Dyckhoff, A. L., Lukarov, V., Muslim, A., Chatti, M. A., & Schroeder, U.: Supporting ac-
tion research with learning analytics. In Proceedings of the Third International Conference
on Learning Analytics and Knowledge (pp. 220-229). ACM (2013).
2. Ferguson, R., Clow, D., Beale, R., Cooper, A. J., Morris, N., Bayne, S., & Woodgate, A.:
Moving through MOOCS: Pedagogy, learning design and patterns of engagement. In De-
sign for teaching and learning in a networked world (pp. 70-84). Springer International
Publishing (2015).
3. Shi, C., Fu, S., Chen, Q., & Qu, H.: VisMOOC: Visualizing video clickstream data from
massive open online courses. In Visualization Symposium (PacificVis), 2015 IEEE Pacific
(pp. 159-166). IEEE (2015).
4. León, M., Cobos, R., Dickens, K., White, S., & Davis, H.: Visualising the MOOC experi-
ence: a dynamic MOOC dashboard built through institutional collaboration. In Proceed-
ings of the European MOOC Stakeholder Summit 2016 Research Track, 461 (2016).
5. Mazza, R., & Dimitrova, V.: Informing the design of a course data visualisator: an empiri-
cal study. In 5th International Conference on New Educational Environments (ICNEE
2003) (pp. 215-220) (2003).
6. Zinn, C., & Scheuer, O.: Getting to know your student in distance learning contexts. In Eu-
ropean Conference on Technology Enhanced Learning (pp. 437-451). Springer Berlin Hei-
delberg (2006).
7. Stephens-Martinez, K., Hearst, M. A., & Fox, A.: Monitoring MOOCs: Which information
sources do instructors value?. In Proceedings of the first ACM conference on Learning@
scale conference (pp. 79-88). ACM (2014).
82