We describe an assessment strategy that helped to ensure high participation levels together with a relevant certification in a MOOC Mathematics course. We report on the first running of the Markov Matrices course making emphasis on the preparation of exercises with random parameters for graded weekly tests. A brief description is given of forum interactions during self-assessment moments followed by a discussion how quizzes and assessment tests positively influenced the MOOC completion rates.
Assessment moments, in particular for students running for a free certification in online
courses, play a key role in the participation rates of the audience, e.g. of Computer
Science MOOCs [
Online electronic exercises with random parameters and automatic feedback started
to be created and developed from 2000 by one of the authors, but have been intensively
applied with formative purposes from 2009 to encourage on campus students to practice
feedback with teachers and peers along the full semester [
The MOOC courses of Técnico Lisboa3 are open online courses running on a
customized platform built on top of Open edX. Starting October 2016 three courses were
released: Markov Matrices, Energy Services and Experimental Physics, which
approach several topics on engineering basic sciences and technology, organized in
subtopics to be delivered sequentially over four/five consecutive weeks, according to
wellstructured guidelines (see [
There were more than 420 enrolled participants in the first edition of the Markov Matrices course, all aged between 14 and 58 years old (average age of 32), mostly from Portugal (87%) employees (57%), and many of them with a Master academic degree (42%) and without a link with Técnico Lisboa (48%). The course assessment activities fostered an active participation which ended with 25% of participants earning the free certificate. 2
We wanted to apply our winner approach of on-campus assessment strategies [
The Markov Matrices course (shortly designated by mmX) was the first course to be designed and produced within the MOOC Técnico directives and delivered at the platform. This course was designed to serve two goals: as a complement of an initiation Linear Algebra4 course, and a course aimed to a broader audience with no specific previous knowledge on Linear Algebra. The overall structure was designed in four weekly topics, which then were partitioned in several subtopics, each one of them identified with a video module, within an increasing of complexity sequence of contents and corresponding assessment moments.
Working on the design and curriculum development of the course, several choices
[
To strengthen the discussion and internalize the learnings, practice exercises equal to all students were made available in the course. However, for evaluation purposes in the mmX online course, we wanted to avoid student cheating and be able to recognize for individual work. For that, similarly to the previous online exercises, we developed a system that generates the same type of exercise over several iterations. The exercises with random parameters specially created for the mmX course were constructed with help of Mathematica programming language5, and were conceived to be aligned with the learning content video units, using similar graphics (see Fig. 2) for the main illustrated models and evaluating all mathematical concepts, theorems included. Most of 4 Linear Algebra is an undergraduate first year course at Técnico Lisboa. 5 https://www.wolfram.com/mathematica the created exercises include graphs, sophisticated model descriptions and hypotheses of theorems, and therefore could not simply be randomized using the python-based exercises produced directly in open edX, which only supports randomization for numerical data 6.
The developed system takes advantage of Open edX libraries and their import/export functionalities7, that enables exercise storage in the platform for later usage in a course (see Fig. 1). The first step consists in identifying what is the exact knowledge to be assessed by the exercise, “exercise identification”, and what parameters, figures and/or statements of the exercise can be randomized. After that, a notebook necessarily containing the Mathematica code that will generate the body of the exercise as well as a randomized figure, part of a model of a statement, must be created.
To produce randomly generated answers, the code that generates these answers must also be present within the same notebook, for all correct and incorrect answer hypothesis. Presently, three types of exercises can be generated: multiple choice exercises, checkbox questions and numerical input questions (see Fig. 2). Once a notebook containing the exercise has been created, a zipped folder with a format capable of being imported to a library in the edX platform must be generated as indicated in Fig. 1. This 6 http://edx.readthedocs.io/projects/open-edx-building-and-running-a-course/en/latest/exercises_tools/numerical_input.html 7 http://edx.readthedocs.io/projects/open-edx-building-and-running-a-course/en/latest/exercises_tools/randomized_content_blocks.html is achieved using “LEIA”, a separate notebook developed at Técnico that runs the notebooks with the exercise parameters coding and returns a zipped folder containing all the instances of the respective exercise in the edX compatible format. When the zipped file is uploaded to the platform, the library can be chosen as an evaluation exercise, and the platform will automatically assign one of its exercises to a participant the first time he/she visits that element of the course. After a submission of a given answer, the correction is revealed to the student, and the teacher receives an automatically generated gradebook with the score of each student. 3
The first edition of Markov Matrices course started at October 19 and ended by November 23, 2016. Additionally, to the four week consecutive topics, it included an introductory half week with two videos, which together with a short quiz (diagnostic test) helped participants become acquainted with the organization of the contents, the type and timing of assessment moments, and the communication channels of the course.
PARTICIPANT 1 PARTICIPANT 2 STAFF
The steady-state vectors in the signal transmission model: p 1-p 1-p p always converge to this matrix? 0.5 0.5 0.5 0.5
I had also reached that conclusion, but I’m not sure if it is valid!
Yes, it is true. After many steps, the message has only a 50% probability of staying the same as the original one…
Of course, in real transmission lines the p coefficient is extremely high, and thus, error dissipation is very slow (it only becomes noticeable after many transmission steps).
From day one the discussion forum played the scene of lively exchanges of questions and answers (380 posts total) with vivid and in-depth discussions. In contrast with the parametrized exercises, the practice exercises (18 quizzes in total) were identical for all students, so these could not only serve as preparation material for the evaluation unit but also as a beacon for student interaction and cooperation. This discussion was encouraged on the MOOC Técnico forums, by pinning a discussion thread to all practice exercises. The forum was also the channel of communication chosen for contact between students and course staff for solving issues related with assessment and clarification of the contents when doubts arose manly related with exercises (see, e.g. Table 1).
At the end of Topic 1, participants realized for the first time that exercises in assessment tests (5 assessment tests), unlike practice exercises in self-assessment quizzes, were not the same to each of them (see an example in Fig. 2). Issued posts in the forums were then solved in a one-by-one basis, thus not violating the secrecy of the contacting student(s) unique problem.
The evaluation of participant’s performance during the course was executed through five assessment tests in the course: one at the end of each four topics and one final test. The first assessment test, Test 1, was released at the same time as the contents of Topic 1 (October 21) and had a deadline for submission of one week (October 28). After that, the remaining weekly tests, Test 2, Test 3 and Test 4, were released simultaneously with the corresponding topics, but had due dates schedule for a little longer than a week.
During this process, participants could check on the course progress bar their
individual results. Each weekly topic evaluation moment, counting 80% to the final grade,
was highly participated when compared to standard numbers for MOOCs. The final
assessment test allowed to evaluate the extent to which participants understand all the
fundamental contents taught during the course (20% of the final grade). We can
emphasize once more that the 25% of participants earning a free honor certificate upon
successfully achieving at least 60% of the course graded activities, which compared
with the 7.7% median certificate rate of courses offering free certification referred in
the Report [
In addition to percentages, from the circa 50 answers to the questionnaire applied at the end of the course, we can conclude that the participants of mmX consider that assessment tests were adequate to the contents addressed, reasonable simple and comprehensible in terms of complexity, and more easier than demanding. They also added comments as: “This model of assessment tests facilitates fast and successful learning of all concepts”, and “…the existence of weekly assessment tests made me dedicate a little more of my time to the contents”. Counting on the success of the present experience with parametrized exercises, we expect to make generalized use of these exercises in future online courses to be produced at MOOC Técnico platform.