=Paper=
{{Paper
|id=Vol-1850/TEFA2016_invited_paper_1
|storemode=property
|title=Formative Assessment supported by Analytics Techniques: a Case Study on Teacher and Students Perceptions
|pdfUrl=https://ceur-ws.org/Vol-1850/TEFA2016_invited_paper_1.pdf
|volume=Vol-1850
|authors=Alana Morais,Danielle Medeiros,Davide Taibi,Ivana Marenzi
}}
==Formative Assessment supported by Analytics Techniques: a Case Study on Teacher and Students Perceptions==
https://ceur-ws.org/Vol-1850/TEFA2016_invited_paper_1.pdf
Formative Assessment supported by Analytics
Techniques: a case study on teacher and students
perceptions
Alana Morais1, Danielle Medeiros1, Davide Taibi2, Ivana Marenzi3
1
Universidade Federal de Campina Grande, 58429-900 Campina Grande-PB, Brazil
{alanamorais, daniellemedeiros}@copin.ufcg.edu.br
2
Consiglio Nazionale delle Ricerche, Istituto per le Tecnologie Didattiche, Palermo, Italy
{davide.taibi}@itd.cnr.it
3
L3S Research Center, Hannover, Germany
{marenzi}@L3S.de
Abstract. This article aims to compare the teacher and students’ perceptions on
the formative assessment process carried out in a blended course in Brazil. To
achieve this goal, the authors used analytics techniques to analyse the online
activities which were supported by the LearnWeb collaborative platform. The
performed analysis investigated metrics related to the interactive student profile
considering learning activities such as: creating groups of resources, selecting,
uploading and commenting on learning resources. The results show that
clustering and classification techniques may be used to investigate students
profile and can be useful in preventing students’ dropout. The authors
conducted a survey to draw an analogy between users’ opinions about the
course and their behaviour on the platform. The questionnaires results
highlighted the positive impressions, awareness, and motivation of the teacher
and of the students during the formative assessment process.
1 Introduction
In the last few years, analytics had a huge impact in different fields. With respect to
the educational sphere, the research fields of Educational Data Mining (EDM) and
Learning Analytics (LA) have studied the effects of applying analytics techniques to
the analysis of educational data. Even though the two fields overlap in several aspects
[1], there are relevant differences that make complementary the approaches adopted
by the two communities. The differences and similarities between the two
communities led to a debate [2]. A very concise distinction is: LA gives more
emphasis on creating procedure to support human judgment regarding the learning
progress, while EDM is more focused on creating models to support automated
processes in intelligent tutor systems.
In our work we have adopted a mixed approach to support formative assessment
processes in a blended course. LA techniques have been adopted during the course to
monitor students’ engagement and provide teachers with tools to promptly intervene.
EDM approaches have been applied at the end of the course to analyze and report the
overall learning experiences of the students in order to evaluate course progress based
on evidences supported by data and to obtain useful insights to facilitate the design of
future editions of the course.
The work was led by the following research question: What are the teacher and
learners perceptions about formative assessment outcomes in a Brazilian blended
course using the LearnWeb collaborative platform?
Our goal was to better understand how the main actors of the teaching-learning
process perceive formative assessment informed by an EDM approach. For this
purpose, a case study was conducted over a period of eight weeks of a blended course
in Computer Science. During the course we applied Learning Analytics techniques to
support formative assessment: teacher used the LearnWeb Formative Assessment
Module to monitor students’ engagement (Section 3). After the course, we performed
EDM on the online activity logs, and collected two questionnaires to draw an analogy
between users’ opinions about the course and their behavior on the platform (Section
4). On the one hand, we investigated students’ perception about the learning activities
and the interaction in the online collaborative platform. On the other hand, we
investigated to what extent formative assessment outcomes (supported by
visualization and evaluation tools) were perceived as informative and efficient by the
teacher.
In the remaining of the paper we provide an overview on related work (Section 2)
and a description of the LearnWeb, the platform adopted to support the learning
activities and to carry out formative assessment (Section 3). In Section 4, we describe
the research phases in this study, and in Section 5 we analyze and discuss the results.
2 Background overview
In the last few years, Learning Analytics (LA) emerged as a research field with the
aim of collecting and analysing learners’ traces for a better understanding of the
learning experiences [3].
In online and blended courses, collaborative environments are often used as data
sources for analytic tools, in order to extract meaningful information about the
students' specific needs. The most common analytic technique used for this purpose is
Data Mining (DM). This occurs both by the popularization of Educational Data
Mining (EDM) approaches in recent studies [4] and by the large amount of data
generated during these courses, which turns manually processing of this data an
impractical activity to be executed by the instructor without the help of learning
analytics tools [5].
Through Learning Analytics techniques, students’ logs are visualized by teachers
to find out relevant hints about students’ interactions within the online learning
system. The analysis of quantitative data related, for example, to students’ access to
learning content, or to students’ participation in online learning forum, can be
leveraged to detect students' engagement and promptly intervene to sort out at-risk
situations, such as early dropout [6].
Learning Analytics techniques have been also successfully adopted to support
formative assessment practices. For instance, Taibi and colleagues [7] applied a
backwards stepwise regression model to infer the relationship between the students’
interactions within the platform and their learning performance.
Learning assessment of students’ knowledge acquisition is one of the major
concerns of the teaching-learning process. In educational settings, performance
assessment strategies are used with the purpose of analyzing and evaluating learners
understanding of the topics discussed throughout a course. Whereas in summative
assessment mainly the final outcome is evaluated, the objective of formative
assessments is to provide feedback to students rather than to evaluate them to assign
course grades [8].
In online learning activities, students’ traces play a key role to support the
formative assessment process. Activity logs of the online e-learning system can be
taken into account and analysed to obtain relevant insights on students’ engagement
in general, and students’ learning behavior in particular. The feedback received during
the formative assessment process facilitates learning, assists students in reflecting on
their learning and revise their misconceptions, and improves their motivation
[9,10,11]. Formative assessment techniques provide insightful feedback that help to
guide and shape the learning process during instruction [12]. Furthermore, they
provide teachers with opportunities to monitor students’ progress and their learning
curve along the course, and to adjust the teaching strategies accordingly [13].
This scenario is often observed in the e-learning context, where monitoring the
learner's progress is usually an essential part of the learning process, since students’
retention and limitation of the dropout rates are important factors for a course success
[14]. Therefore, formative assessment can offer effective solutions for identifying and
dealing with potential dropouts.
In blended learning scenarios, traditional performance indicators based on
outcomes-centric analytics considering learners’ performance on predefined tasks are
not enough to evaluate the learning process. The focus has to shift from what the
learner can achieve individually, to a more participatory perspective where students
learn from the interaction with their peers [15].
Social Learning Analytics has been defined by Shum and Ferguson [15] as “a
distinctive subset of learning analytics that draws on the substantial body of work
demonstrating that new skills and ideas are not solely individual achievements, but
are developed, carried forward, and passed on through interaction and collaboration”.
Social learning has emerged as a significant phenomenon and it is characterized by
the involvement of learners in social activities in which students learn from others
interacting with their peers [16]. In this scenario, the evaluation of learning
achievements requires specific tools. Shum and Fergusson [15] identified five
categories of analytics in relation to online social learning. In particular, the authors
distinguished between inherently social analytics, that can be developed in collective
contexts, and socialised analytics, that have been initially developed to analyse
personal learning activities, but that have specific characteristics in collective context.
In general, Social Learning Analytics techniques and tools can be leveraged to
identify disconnected students as well as students playing the role of information hub.
In addition, they are relevant to identify patterns of communication exchange in
students network and to reveal the structure of learners’ communities.
In our experimentation, data mining techniques have been adopted to support
formative assessment practices focused on the analysis of social processes that take
place during learning. Clustering algorithms have been applied with the aim of
evaluating students’ engagement in social learning activities. Furthermore, the
feedback provided by the algorithms have been analysed by the teacher in order to
support a promptly intervention aimed at reducing students’ dropout.
3 Learning scenario and methodology
The context of the case study was a blended course in Logic in Computer Science
which took place at the Universidade Estadual da Paraíba, in Brazil, spanning over a
period of two months (eight weeks) from February 2016 to April 2016.
The participants were 35 students, aged between 16 and 25, of which nearly 70%
were male. Their mother tongue was Portuguese and the course was designed in
Portuguese. Almost 69% of the students had never interacted within a Virtual
Learning Environment before. The participants were informed about the scope of the
case study, but they did not know how and when formative assessment would happen
during the course.
Four distinct collaborative assignments were carried out as described in the Table 2
(section 4.1), using the LearnWeb environment. Each assignment was evaluated by
the instructor, based on direct assessment of student individual and group
performance throughout the course. The assignments were planned by the teacher and
one technician in order to promote a collaborative dynamic between students, where
all functionalities offered by the LearnWeb environment would be used at the same
time.
3.1 LearnWeb as a Formative Assessment tool
LearnWeb is a learning and competence development environment, which allows
users to share and collaboratively work on resources collected from the web or user-
generated [17]. It provides users with a search interface for resource discovery and
sharing across various Web 2.0 services such as YouTube, Flickr, and Slideshare,
including LearnWeb itself, and offers a Personal Learning Space. In order to support
collaborative searching, LearnWeb provides automatic resource annotation; resources
in LearnWeb can be bookmarked, tagged, rated, and discussed by all users who are
allowed to access them. Comments on particular learning resources can be used to
enrich the description. Users can create folders to bundle resources that belong to the
same learning context. Hence, the LearnWeb community can collaboratively identify
the best learning resources for specific learning domains. A discussion of the full
potentialities and affordances of LearnWeb as a collaborative platform is beyond the
scope of this paper, but can be found in a series of published studies [17, 18].
The present study focuses in particular on the use of the Formative Assessment
Module and on the analysis of the potentialities to support the learning process.
3.1.1 The LearnWeb Formative Assessment Module
The Formative Assessment Module in LearnWeb aims to support both the teacher and
the learners, increasing students’ achievement. Learners need to know: the learning
target, where they are positioned in regards to the learning target, and what they can
do to close the gap. The LearnWeb Formative Assessment Module provides students
with clear learning targets, examples and models of strong and weak work, regular
descriptive feedback, and the ability to self-assess, track learning, and set goals.
Furthermore, the Formative Assessment Module allows a visual analysis from the
virtual classroom to the teacher. In order to address the needs of different scenarios
the LearnWeb Formative Assessment Module was designed from four main
perspectives:
• A course perspective, where the teacher has an overview of a specific course and
can make comparisons between classes (Fig. 1).
• A class perspective, where the teacher can monitor and compare the activities of
small student groups within the same class. This view can be shared with each
workgroup to raise awareness concerning the activities carried out by other group
members and increase motivation and competitively between groups (Fig. 2).
• A personal perspective, where the teacher can visualize information about a
specific student (Fig. 3).
• A user group perspective, where the teacher can visualize his/her class in groups
(clusters) according to the student interactive profile. This view shows the most
participative students in the same group according to their interactions on the platform
(Fig. 4).
All perspectives provide to the teacher the possibility to send a feedback message
and generate three different types of chart (i.e. bars, line and pie chart). The teacher's
target can be the class, groups in the courses, specific student or the cluster of students
identified by the LearnWeb Formative Assessment Module. This module supports
formative assessment practices by allowing teachers to interact with students in order
to provide prompt intervention when critical situation or at risk conditions have been
identified.
Fig. 1. Formative Assessment Module in the LearnWeb
Regarding the student group functionality, a useful analysis carried out by the
teacher was the comparison between groups during the course. The groups
comparison allowed the teacher to understand students behavior in groups according
to different learning approaches. The class perspective (Fig. 2) visualized by the
teacher during the Assignment 2 shows two different groups: “Atividade II - {Pinky
and the Brain}” and “Atividade II - {Steff&Paiva}”. In this case, the teacher noticed a
similar behaviour between the students groups regarding the use of LearnWeb
functionalities. However, the level of searching and adding resources in both groups
was lower than the teacher expectations. Based on this information, the teacher
provided feedback messages to adjust the unexpected behavior, encouraging students
to increase searching and uploading. Sometimes the teacher needed a more precise
overview about the personal interactions of specific students in the group.
Fig. 2. Class perspective analysis
In such cases, the teacher explored the Personal Perspective view in the LearnWeb
Formative Assessment Module. This information helped the teacher to plan corrective
feedback and ad-hoc advices. Fig. 3 exemplifies a user profile. The visualization
confirmed the low rates of searching activity and, consequently suggested the need to
monitor the student behavior and act to solve some user learning mistakes.
Fig. 3. Personal Perspective Analysis
Finally, the LearnWeb Formative Assessment Module provides the user groups
analysis. In this perspective, the teacher can visualise his/her class in groups (clusters)
according to the student interactive profile. The ultimate goal is to allow the teacher to
get a better understanding of the classroom dynamics. Fig. 4 shows a visualization
which helps the teacher identify less-active students (for example, student 10 and 11
in cluster 3) and encourage them to participate more in the online interactions,
according to the user interactions in the course (it was based on specific variables as
reported in Table 1). When the clusters are generated, the teacher can visualize
additional information in the clusters charts built by the Formative Assessment
Module.
Fig. 4. User groups analysis
Table 1. Selected user variables in each log record.
Variable Variable Description
Resource Tagging Number of tags added by a user
Comment Number of comments created by a user
Group Joining Number of times that a user joined in a group -
Group Leaving Number of times that a user left a group
Resource Opening Number of times that a user opened a resource
Downloading Number of times that a user downloaded a resource
Searching Number of times that a user performed a search
Resource Deleting Number of times that a user deleted a resource
Resource Edition Number of times that a user edited a resource
Resource Addition Number of times that a user added a resource
Group Creation Number of times that a student created a group
Comment Deleting Number of times that a student removed a comment
For example, (i) it is easy to detect which student belongs to which group because
the Student ID is visualized inside each circle (see orange circles in Fig. 4), (ii) it is
possible to get details about students interactions when the mouse cursor is over the
student name in the chart, and (iii) the size of the circle radius represents the
participation level of a student (according to the variables described above) until the
moment when the analysis was done. The radius function was defined in Phase 1
(course design) of this process and it involves four main student variable sets:
a) s1: Number of comments, resource addition (uploading), resource tagging, and
group creation;
b) s2: Number of searching, resource editing, and resource opening;
c) s3: Number of downloading, group joining, and resource deleting;
d) s4: Number of comment deleting, and group leaving.
The participation level (P) function is a weighted arithmetic mean of those
students’ variable sets. The weight of each variable sets was defined by the teacher
and the technician in Phase 1, but the construction of charts using this metric
happened in the second phase. The set s1 is the most relevant for the teacher and
contributes more than anything else in calculating the mean P. This happens because
the targets in the course guided the teacher to explore comments, adding resources,
tagging resources, and creating groups as the most essential functionalities during the
course. For that reason the weight w1, related to the s1 variable is set to 4 and the
other weights (w2, w3 and w4) are set to 2:
P = (w1*s1 + w2*s2 + w3*s3 +w4*s4) / w; where w1=2, w2=2, w3=2, and w4= 4
As a result of this phase, the clusters charts support the formative assessment
process. In our scenario, the teacher decided to investigate three main student groups
after some tests carried out during the first week of the course.
4 Research Phases
In order to investigate the efficacy of the formative assessment approach, and
compare it to the users’ perceptions, we divided the specific learning scenario in four
phases and analyzed each of them. Fig. 5 presents the sequence of the four phases
which are: (i) Course Design, (ii) Course delivery, (iii) Questionnaires and
Classification analysis, and (iv) Final analysis.
The following sections describes the actions, techniques and tools which were used
in order to analyse each phase of the study. The actual discussion of the results is
presented in Section 5.
4.1 Phase 1 - Course Design
Before the beginning of the course, the teacher planned, scheduled, reviewed and
tested all activities, while the technician supported the teacher on technical issues.
It was decided to adopt a blended format for the course. Blended learning is an
instructional model that combines teaching methods from both face-to-face and online
learning. It can be implemented in a variety of ways, ranging from models in which
curriculum is fully online with face-to-face interaction, to models in which face-to-
face classroom instruction is integrated with online components that extend learning
beyond the classroom or school day. In blended learning scenarios, teachers can use
various types of technology to offer countless different learning pathways to students,
to provide an integrated learning experience. The advantages of using an online
platform in blended learning environments are not only related to the functionalities it
provides to support students’ learning activities, but also to the opportunity for
teachers to evaluate students’ work throughout their learning pathway.
Blended learning brings about an important shift in formal education, making the
teaching/learning pathways much more flexible: students can collaborate both in class
and online, they can organize the learning activities with a different time schedule,
their role and engagement in the learning activities can change due to the use of
technology. Such flexibility increases the need for formative assessment, which can
help monitor student progress and then inform the choice of pathway forward.
This phase took into account the design of the students’ activities and assessments,
the period in which the activities were carried out, the feedback strategies adopted by
the teacher, as well as the students’ evaluation for each activity. The outcome of this
phase is a document that describes all activities and the assessment approaches which
were implemented in the course (see Table 2).
Fig. 5. Phases Sequence in the current study case
Table 2. Assignments proposed in the Brazilian course
Assignments Description Duration LearnWeb
Functionalities
1 The main idea of this assignment was to share Week #1 to questionnaire -
Resource information, by means of sharing learning Week #8 perception about the
Sharing materials. After creating a group of resources, formative assessment
each student had to search and upload one or process
more learning resources related to the topic
Logic in Computer Science. Students could
also exchange ideas and opinions through
comments.
2 This assignment aimed at improving Week #2 to Students groups
Resource learner's knowledge about Propositional Logic, Week #4
Annotation and by creating and discussing practical logic
Discussion problems. The students, organized in teams of
two individuals, had to create a group of
resources to exchange ideas about the present
topic of discussion and elaborate their own
logic problem to be solved by other teams. The
teams could use the group forum to debate
about strategies and use the comments section
to discuss about uploaded materials. At the end
of the assignment, the students had to create a
Google Document with the logic problem
formulated by them.
3 This assignment is complementary to the Week Students groups
Team challenge previous one. After the completion of the #5
second assignment, the previously formed
teams had to challenge each other by
exchanging the logic problem created by them.
The challenging team could add comments or
other resources with hints to help the
challenged team, if they thought it was
necessary.
4 The objective of this assignment for the Week #6 Discussion Forum and
Final students was to practice all knowledge acquired to Week #8 Google Document
Evaluation throughout the course. A specific group was
created for this activity, where the teacher
uploaded a set of logic problems and the
students had to choose a distinct problem to
solve. At the end, the students added their
solutions in the activity group and discussed
them with the others through comments.
4.2 Phase 2 - Course Delivery
In this phase, the LearnWeb Formative Assessment Module (section 3.1) was used by
the teacher to provide timely feedback to student outcomes. During the course, the
teacher carried out the data analysis from three different perspectives: by Class, User
Personal perspective and Group perspective. During this phase the learning materials
were delivered and the researchers, the teacher and the technician, started to monitor
the students’ activities. After the enrollment, students worked on the learning
activities, while the teacher monitored their performance applying formative
assessment strategies. From the technical point of view, cluster analysis was used to
define a profile for each student according to their behavior online. The cluster
analysis was supported by the group perspective functionality (Fig. 4) in the
LearnWeb Formative Assessment Module and was based on specific variables as
reported in Table 1.
4.3 Phase 3 - Questionnaires and Classification Analysis
The third phase consisted of two complementary tasks: user questionnaires and data
classification analysis. Both of them started after the course was concluded (a
posteriori analysis).
a) Questionnaires
The questionnaire is a basic tool, not only for measuring individual perceptions about
a phenomenon, but for assessing users’ experiences during the course. The main aim
of the questionnaires analysis was to investigate the teacher’s opinion and the
students’ opinion about the formative assessment process adopted in the course. The
teacher designed the questions for students in collaboration with the technician, and
the technician designed the questions to be asked to the teacher.
Both questionnaires were designed so that answers were scored, and scores were
summed up, to obtain an overall measure of the attitudes and opinions of the teacher
and of the students, who answered the questionnaire on voluntary basis. One
advantage of using questionnaires is that respondents have more time to consider their
responses carefully without interference by the investigators. Our questionnaires
included two types of questions: open-ended and closed-ended. An example of a
multiple choice question is “Which one of the functionalities below would you judge
to be relevant to support the activities in your group? (Add resource, Add Tag,
Search, Comment, Change the privacy settings, Forum, Download of resources)”.
While the students were questioned about their study motivation and routine using the
LearnWeb and the usefulness of the functionalities provided by it, the teacher
answered questions about her expectation, in the beginning of the study, and her
impressions about the formative assessment after using LearnWeb as a teaching tool
(more details on the questionnaires results are given in section 5.2).
b) Classification Analysis
The second step in the third phase is the students’ classification analysis according to
their behavior in the course. Some tests involving decision tree techniques were
executed in order to support the teacher in better understanding a multiple variable
analysis in the Brazilian course. The classification analysis focused on correlating the
user variables listed in Table 1, with the dropout status (more details on the
classification results can be found in section 5.3).
4.4 Phase 4 - Final Analysis
At this stage, we executed the final analysis of the case study after reviewing the
results obtained over the previous phases. Other observable variables were also taken
into consideration for the analysis concluded in this phase, such as the total of
completed assignments of the students and dropout rates in the course.
5 Discussion of the Results
On the basis of the methodology and of our research question, in this section we
organize and comment about the main results in our study. The analysis was
performed in two different moments: at the end of the second week and at the end of
the sixth week.
5.1 Phase 2 results - Clustering results
As we mentioned in section 3.1, the Formative Assessment Module uses the K-Means
algorithm to identify the clusters according to the students interactions captured in the
log files in specific periods during the course. Through the K-Means algorithm, three
main groups were obtained based on metrics related to the students’ interactions (Fig.
4).
Every week, the teacher monitored the clustering results in order to identify the
most appropriate variables to delineate the interactive student profile. According to
the characteristics of each group, the teacher suggested to call them “Participative
Student Group”, “Medium Student Group” and “Non-Participative Student Group”. A
detailed analysis of the features of each group was carried out at the end of the second
week and at the end of the eight week.
The first round of analysis took into account the activities performed after the
second week, related to Assignments 1 and 2. The resulting groups formed a scenario
composed by a numerous “Medium Student Group” and two other groups:
“Participative Student Group” and “Non-Participative Student Group” (Fig. 6). The
smallest group in Brazilian scenario was the “Non-Participative Student Group,
composed by two students.
Fig. 6. Clusters built in the second week
In the Brazilian course, students belonging to the “Medium Student Group”
typically missed some deadlines, did not solve Assignment 1 or Assignment 2, and
opened some resources. The “Participative Student Group” included students who did
the assignments, opened some resources, and answered some colleagues. Finally, the
“Non-Participative Student Group” represented students who accessed the
environment only to enroll in the course. The teacher made an effort to provide some
feedback messages and review activities to motivate the class throughout the course.
During the student interaction in each assignment, the teacher sent out messages of
encouragement or warnings about close deadlines to specific groups of students. For
example, for the “Non-participating Student Group”, the teacher periodically sent out
motivational messages and tips about learning resources and assignments.
The second round of analysis took into account the activities carried out during the
eighth week, related to Assignment 4. Fig. 7 shows that the “Participative Student
Group” increased in comparison to the second week, since it is possible to identify a
group with an increased number of circles and with a larger diameter.
Fig. 7. Clusters built in the eighth week
The cluster named “Non-Participative Student Group” presented a modification
related to its members. After the second round of analysis, two members belonged to
the mentioned cluster, they were: Student 28 and Student 29. However, after the
eighth week, this group changed and it was composed by Student 21 and 29. Two
main reasons can be highlight to the reformulation of clusters between the second and
eighth round of analysis:
(i) After the second week, the teacher provided an extra support to motivate the
members of the “Non-Participative Student Group” with some feedback messages;
this motivated Student 28 to review his interactions and to improve his performance
in the course.
(ii) Student 21 lost his access login to LearnWeb and gave up interacting weekly
when the course advanced, regardless of the teacher feedback.
Some cluster reformulations were expected during the analysis as a natural
consequence of the periodic interventions of the teacher, of the students’ motivation
feeling, and of their interactions throughout the course.
If the study was interrupted at this stage of the analysis, it would not have provided
indicators to understand other behaviors. For example, the behaviors that may be
associated with students belonging to the group “Participative Student Group” and
“Medium Student Group”. Thus, the importance of continuing the analysis according
to the methodology presented in Section 4.
5.2 Phase 3a - Questionnaire results
After analyzing the students groups during the course delivery (Phase 2), we
investigated the teacher and students opinion about the formative assessment
approach adopted in the course. Questionnaires proved to be a quick and efficient way
of obtaining information from the participants to the virtual classroom. Both
questionnaires (for the teacher and for the students) were delivered at the same time
after the end of the course.
a) Teacher Questionnaire
The questionnaire about teacher’s perceptions investigated four main aspects: (i) the
teacher’s assumptions about the formative assessment process, (ii) the overview about
students interactions in the course, (iii) the teacher’s satisfaction, and (iv) the
teacher’s motivation. This phase aimed to investigate whether the teacher understood
the classroom needs and explored properly the environment to support them.
Initially, the teacher said that she was excited to involve her classroom in this
experience, and to monitor students’ activities since the beginning of the semester.
According to her answers, the formative assessment approach worked as planned. It
allowed her to re-think and to adjust some assignments according to the classroom
needs. This aspect was fundamental to motivate and decrease the level of students’
dropout in the course.
Even though the teacher was satisfied about the formative assessment analysis, she
would like to investigate the results further in order to understand some specific
student’s behaviors that remain unclear. For example, what are the functionalities
more used for the students in the course?
b) Student Questionnaire
The second questionnaire focused on the students’ perceptions on the course and on
their opinions about the formative assessment process. The proposed questions
explored four main aspects: (i) the student’s routine during the course, (ii) the
student’s motivation, (iii) the risk of student’s dropout in the course, and (iv) their
overview about the course. These aspects guided the authors to understand the
students’ awareness and acceptance of the formative assessment process.
First of all, the questionnaire analyzed the students routine in the learning process.
The study identified an interactional profile during the course. Most users accessed
the environment from 1 to 3 times per week, 73% of the students tried to solve all
assignments, and they interacted with the virtual environment at home and at school.
Regarding the students motivation, 54% of them affirmed to be motivated
(Motivated or Very Motivated) during the assignments. The most popular activities
were Adding resource, Download resource, and Creation of group.
Concerning the students’ dropout, one of the questions asked the students to
“Assess the risk of dropout of the activities through the course”. As a result, 46% of
the interviewees affirmed to think about it, due to the difficulties faced during the
learning process.
The importance of the course feedback was highlighted by students in the
questionnaire. One student commented: “The assignments were moderate in
difficulty, productive and practical. I was very motivated to solve all the [logic]
challenges. However, I had some issues because I did not have sufficient knowledge
of the theory at the time when the assignments were performed”. When the teacher
identifies this kind of issue, he/she can provide feedbacks to provide extra materials,
solve questions, and support the learning process. Besides that, the questionnaire also
showed that the students were somewhat motivated to use the platform during the
experiment.
5.3 Phase 3b - Classification results
The Classification Analysis (Phase 3) is useful to answer unsolved questions in the
course. For example, the Brazilian teacher wanted to understand better the link
between the students profile and the students’ dropout status in the course. The
dropout status is a metric related to the final situation of the students in the course
(“dropout” or “non-dropout”). Despite the teacher's efforts to motivate the class
during the course, dropout rates reached a percentage of 46% at the end of the course.
Several classification techniques were performed during this phase; among them
the decision trees provide the most interesting feedbacks for the specific scenario. The
decision tree is useful to create a model that predicts the value of a target variable
(dropout status) based on several input student’s variables (listed in the Table 1) [19].
It is important to emphasize that the group analysis did not influence the investigation
at this stage. In our study we applied the following algorithms: J48, NBTree, Random
Tree, ADTree and RepTree. Table 3 lists the parameters used for the comparative
analysis. All techniques were implemented using the Weka suite.
Table 3. Classification techniques
Tree Correctly Classified Incorrectly Classified ROC Area
Instances Instances
REPTree 76.92 % 23.08 % 0.85
Random Tree 92.31 % 7.69 % 0.95
NBTree 69.23 % 30.77 % 0.90
ADTree 92.31 % 7.69 % 0.98
J48 92.31 % 7.69 % 0.95
According to the results obtained, most of the algorithms are able to classify
correctly a part of the sample and provided good results. The only exception is the
NBTree tree that classified less than 70% of the sample instances correctly. The ROC
rate of all trees were constructed from a point of view on what can be demonstrated in
other classification techniques.
Several sorting rounds were performed using as individual metrics and grouped to
extract non trivial information. The trees normally present squares to represent the
node that corresponds to one of the input variables; there are edges to children for
each of the possible values of that input variable. Each leaf (rectangles) represents a
value of the target variable given the values of the input variables represented by the
path from the root to the leaf.
When interpreting the built decision trees, some findings were obtained. For
example, during an analysis of the J48 tree it was identified a link between the
dropout status and the students that did not comment, did not add resources, did not
add tag, did not create groups and did not execute resources, and search in LearnWeb
(Fig. 8). Moreover, the trees built using the ADTree and Random Tree algorithms
confirmed the link between the dropout status and some functionalities. However, the
REPTree algorithm (Fig. 9) identified a relationship among the students dropout
status, searching, and resource addition.
Fig. 8. J48 Tree (week 8)
Other user variables highlighted by the classifications mentioned above were not
important for this algorithm. Such information can be useful for the teacher during the
final course assessment, facilitating the review of the pedagogical strategies adopted
during the course, and providing additional knowledge to improve the next classes.
When we compared the students’ and the teacher's opinions, we found different
views about the usefulness of the functionalities in the course. The teacher believed
that the most useful functionalities are searching, resource downloading, sending
feedback messages, and resource addition. While most of the students cited as the
most interesting functionalities resource addition, group creation, and resource
downloading. Regarding the formative assessment process, the teacher highlighted as
the most useful functionalities (i) the possibility to send feedback messages to
students, and (ii) the possibility to visualize personalized charts to identify any
students fail status.
Fig. 9. RepTree (week 8)
5.4 Phase 4 - Final Analysis
In order to provide an overview of the whole process carried out in this research, we
take into account the EDM results presented by the clustering charts (Section 5.1) and
the classification analysis (Section 5.3), as well as the subjective view presented by
the students and the teacher through the answers to the questionnaires (Section 5.2).
Throughout the execution of all the phases of this research, the teacher analyzed the
results obtained in order to extract information that would be useful to help improve
the pedagogical practices used for the case study.
Before the start of the course, the teacher intended to use the Formative
Assessment functionalities available in LearnWeb as a motivation factor to carry out
more dynamic activities and instigate students to better learn the content discussed in
the classroom, as well as to encourage them to interact more with their colleagues.
Overall, the students were motivated to use the platform to complete the
assignments. Only three students claimed that they didn't feel motivated to use the
LearnWeb environment and completed 100% of the proposed assignments.
Despite these facts, the teacher pointed out that the behavior of the class
throughout the course was quite heterogeneous. This was clear while observing
students' behavior illustrated by the clustering analysis. A few students threatened to
give up interacting with the platform from the beginning of the course; while others
struggled to understand how LearnWeb works or they found the course content
difficult to understand. More than 76% of the students were very motivated to carry
out the activities and finalize them on time. Some behavior of the less active students
can be explained by the fact that more than a half of them had never used a
collaborative platform before. The teacher mentioned that some of the students were
not attracted to participate in the environment, preferring to discuss some aspects of
the assignments face-to-face rather than commenting or using the online forum.
As mentioned before, while understanding students behavior, the instructor worked
on motivating the non-participative students to interact and participate more in the
assignments. In this respect, the teacher described the LearnWeb Formative
Assessment Module as an “interesting tool that can offer the educator the opportunity
to review and analyze the student's’ profile throughout the course, offering the
opportunity for changes to be made in the learning practices and methodologies in
order to be more effective”.
In addition to this, the final results show that almost all students considered the use
of the formative assessment during the course as a positive experience and about 75%
of them would recommend the use of a virtual environment to support learning.
6 Conclusions and Future Work
In this paper, we presented an overview about teacher and students perception on the
Formative Assessment process. This methodology allowed understanding student
patterns and answering teacher issues during the formative assessment process. The
aim of this research extends beyond demonstrating the practical importance of the
learning analytics infrastructure in the formative assessment process. Additionally, the
main contribution of this study was to take into account the user perception about the
full process. The outcomes of this study contributed to a better understanding of
students’ interpretation of feedback and the impact that this had on their motivation
and goals.
Regarding the initial question of our research (What are the teacher and learners
perceptions about formative assessment outcomes in a Brazilian blended course using
the LearnWeb collaborative platform?), we collected and analysed the interest,
motivation and engagement of the students and of the teacher during the learning
process. Students’ scores improved, even though technical difficulties could not be
ignored during this process.
For this reason, we intend to improve the LearnWeb Formative Assessment
Module including better charts, temporal analysis, and supporting students self-
assessment during the course to address one of the students requests highlighted in the
questionnaire as mentioned in section 5.
In general, the formative assessment process was a successful experience in the
Brazilian course because there was an improvement in the class performance and the
dropout score rate was affected directly by the student motivation. Students reported
their impressions about this aspect in the questionnaire, as: “The LearnWeb platform
was a way to participate in different activities in a fun way.” and “It was a great
experience. Overall, the assignments were well worked out and I learned a lot”.
Overall, the results indicated that the use of online tools to support blended courses
can be a positive experience for all those involved. There was a high percentage of
students who recommended the use of a virtual environment as a way to complement
the activities carried out in a face-to-face course. From the teacher's point of view, a
collaborative environment that aided in formative assessment offered the opportunity
to better understand students' behavior and identify alternative methodologies to
prevent dropouts.
In addition to this, we also highlighted the contribution of the analysis of
LearnWeb, a virtual learning tool not yet used in Brazilian courses. It is also worth
noticing that our approach of using EDM techniques to support formative assessment
practices proved to be an effective way to improve the instructor's’ decision-making
process.
In the future we plan to evaluate the concordance between the perceptions of
teacher and students and the data gathered by the logs of the system.
7 Acknowledgments
The authors would like to thank the students for their contribution in this research
work.
8 References
1. Siemens, G., Baker, R.S.J.D.: Learning Analytics and Educational Data Mining: Towards
Communication and Collaboration. Proceedings of the 2nd International Conference on
Learning Analytics and Knowledge (2012)
2. Liñán, L.C., Pérez, Á.A.J.: Educational Data Mining and Learning Analytics: differences,
similarities, and time evolution. International Journal of Educational Technology in Higher
Education 12(3) (2015) 98-112
3. Long, P.; Siemens, G.: Penetrating the fog: analytics in learning and education. Educause
Review Online 46 (5) (2011) 31–40
4. Peña-Ayala, A.: Educational data mining: A survey and a data mining-based analysis of
recent works. Expert Systems with Applications 41 (2014) 1432–1462
5. Alencar, M., Santos, E., Netto, J. F.: Identifying Students with Evasion Risk Using Data
Mining. Proceedings of EdMedia. Quebec, Canada (2015) 773-778
6. Yang, D., Sinha, T., Adamson, D., Rosé, C. P.: Turn on, Tune in, Drop out: Anticipating
student dropout in Massive Open Online Courses Proceedings of the 2013 NIPS Data-driven
education workshop, Vol. 11. (2013)
7. Taibi D., Kantz D., Fulantelli G.: Supporting formative assessment in content and language
integrated learning: the MWS-Web platform. Int. J. Technology Enhanced Learning 6, 4
(2014) 361-379
8. Lin, J.W., Lai, Y.C.: Harnessing Collaborative Annotations on Online Formative
Assessments. Educational Technology & Society 16 (1) (2013) 263–274
9. Wang, M.: Designing online courses that effectively engage learners from diverse cultural
backgrounds. British Journal of Educational Technology 38(2) (2007) 294–311
10.Nicol, D.J., Macfarlane-Dick, D.: Formative assessment and selfregulated learning: a model
and seven principles of good feedback practice. Studies in Higher Education 31(2) (2006)
199-218
11.Black, P., Wiliam, D.: Developing the theory of formative assessment. Educational,
Assessment, Evaluation and Accountability 21(1) (2009) 5
12.Shelton, A., Smith, A., Wiebe E., Behrle, C., Sirkin, R., Lester J.: Drawing and Writing in
Digital Science Notebooks: Sources of Formative Assessment Data. Journal of Science &
Education Technologies 25(3) (2016) 474-488
13.Bennett, R.E.: Formative assessment: a critical review. Assessment in Education: Principles,
Policy & Practice 18(1) (2011) 5–25
14.Yukselturk, E., Ozekes, S., Türel, Y.K.: Predicting Dropout Student: An Application of Data
Mining Methods in an Online Education Program. European Journal of Open, Distance and
e-Learning 17(1) (2014) 118-133
15.Shum, S.B., Ferguson, R.: Social Learning Analytics. Educational Technology & Society
15 (3) (2012) 3–26
16.Blackmore, C.: Social learning systems and communities of practice. London, the United
Kingdom: Springer. (2010)
17.Marenzi, I., Zerr, S.: Multiliteracies and active learning in CLIL - the development of
LearnWeb2.0. In: IEEE Transactions on Learning Technologies (TLT). (2012)
18.Marenzi, I.: Multiliteracies and e-learning2.0. In: G. Blell, R. Kupetz (eds.) Foreign
Language Pedagogy, Content and Learner Oriented, Vol. 28, Peter Lang. (2014)
19.Rokach, L.: Data mining with decisions trees: theory and applications. ISBN
9789814590082. Series in Machine Perception Artificial Intelligence Vol. 81. World
Scientific. (2014)