=Paper=
{{Paper
|id=Vol-2920/paper_5
|storemode=property
|title=E-course Data in Learner Formative Assessment. Case Study
|pdfUrl=https://ceur-ws.org/Vol-2920/paper_5.pdf
|volume=Vol-2920
|authors=Tatiana Pavlova
}}
==E-course Data in Learner Formative Assessment. Case Study==
https://ceur-ws.org/Vol-2920/paper_5.pdf
E-course Data in Learner Formative Assessment. Case
Study∗
Tatiana Pavlova
pavtatbor@gmail.com
Herzen State Pedagogical University of Russia,
St. Petersburg, Russian Federation
Abstract
The purpose of the case study is to substantiate the e-course data utilising methodology for
students’ autonomous learning actions support. The principles of e-course data extraction and
application are associated with e- course specific instructional profile in line with interaction
techniques and digital resources. The proposed approach involves the correlation of the e-course
instructional profile and feasible LMS data sources for subsequent reflection in the student’s action
profile. A student’s reflective comprehension of his personal actions profile at the learning process
definite stages implies an effective formative assessment practice. The first results of teaching
technique implementation are discussed based on the collected action profiles review and student
feedback.
Keywords: e-course, learning related data, formative assessment, support of the learning
process, e-course student’s profile
1 Introduction
Digitalisation of the educational environment leads to changes in the learning process. To organize
educational activities, an important role is played by digital educational resources operating in a
variety of information systems. Educational resources of the developing digital environment have
significant features: they reflect new trends of knowledge society, as well as promising educational
priorities related to the personalizing of the learning process and digital competencies shaping. This
generates a need for new approaches in digital learning content design. Due to various multimedia
technologies and intelligent algorithms automating information processes, key changes relate to ways
and structures of knowledge representation in human-machine systems. The problem contains not only
the insufficient distribution of “non-classical” educational content structures and promising interactive
technologies but also the insufficient educators’ and students’ readiness to interact effectively with
new knowledge structures using new digital tools.
2 Materials and methods
2.1 Materials of investigations
The case study reveals the idea of a formative assessment technique applying the learning related
data of the e-course designed for student’s autonomous learning and its pedagogical support. The
∗
Copyright c 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
1
experimental group of masters studied the discipline “Multimedia technologies in the educational en-
vironment.” The methodology involves the design of personal students’ activity and performance
profiles in accordance with the set of teaching techniques for organizing educational interaction. Per-
sonnel student profiles and recommendations for their analysis are considered as e-course management
tools to support learning process control [Noskova & Pavlova, 2017].
The research was carried out in the fall semester 2020 - 2021 academic year.
2.2 Related works
Formative assessment ideas have extensive background in theory and practice. A significant number
of pedagogical publications are devoted to the formative assessment possibilities and advantages
[Black William, 2009], [Andersson & Palm, 2012], [Brink & Bartz, 2017], etc. Black William stated
that formative interaction “must be analysed as reflecting a teacher’s chosen plan to develop learning,
the formative interactions which that teacher carries out contingently within the framework of that
plan . . . and the internal cognitive and affective models of each student of which the responses and
broader participation of students provide only indirect evidence” [Black William, 2009].
Numerous publications provide justifications for flexible, reflexive teaching and self-directed
learning practices based on continuous feedback. The goals and effective techniques at var-
ious digital formative assessment cases presented [Yorke, 2003], [Rickards & Stitt-Bergh, 2016],
[Granberg et al., 2021], etc.
Brink Bartz claim that “A high school’s culture must embrace formative assessment as standard
operating procedure in all classrooms” [Brink & Bartz, 2017] .
Particular attention is paid to the study of the implementation of formative assessment in a digi-
tal environment applying a variety of digital tools [Long Siemens, 2011], [Pellegrino & Quellmalz,2010],
[Jivet et al., 2018]. In the modern educational process, formative assessment has close ties with learn-
ing analytics, which allows to receive, analyze and apply information about user actions in learning
management systems “for purposes of understanding and optimizing learning and the environments
in which it occurs” [Long Siemens, 2011].
With learning analytics trend associated research and practise cases “along the four follow-
ing dimensions: computer-supported learning analytics, which includes dropout/retention, student
performance, and evaluation; computer-supported predictive analytics, including collaborative learn-
ing and self-learning; computer-supported behavioral analytics, focused on modelling student learn-
ing; and computer-supported visualization analytics, such as graph or network based methods”
[Aldowah et al.,2019].
The following learning data analysis goals typology distinguished: predicting learner performance
and modeling learners, suggesting relevant learning resources, increasing reflection and awareness,
enhancing social learning environments, detecting undesirable learner behaviors, and detecting affects
of learners [Verbert et al., 2012].
To perform a formative assessment in a digital environment, a teacher must solve several signif-
icant problems in interconnection:
• data extraction for further analysis and determination of the purpose for their processing;
• data analysis methods selection;
• determining how to apply data analysis results in practice.
For formative assessment, combined LMS data and data obtained, for example, as a result of
surveys, student’s personal data, can be applied. This approach is typical for Dispositional Learning
Analytics (DLA). Shum Crick define DLA as “infrastructure that combines learning data (i.e. those
generated in learning activities through the LMS) with learner data (e.g., student dispositions, values,
and attitudes measured through self-reported surveys)” [Shum & Crick, 2012] .
2
The solution to the third problem includes actions to optimize educational interaction, commu-
nicating with students, and adjusting digital resources [Andersson & Palm, 2012].
The choice of a specific formative assessment option depends on the method used to teach
the subject, on the one hand, and on the other hand, it changes the method to a certain extent.
Vikulina and Vilkova also emphasize “the dependence of the effectiveness of formative assessment on
its compliance with professional beliefs and its acceptance by teachers” [Vikulina & Vilkova, 2019].
Goldman underlines that the main task is to ensure that the obtained data can benefit persons
to promote their ability “to make meaningful choices” [Goldman, 2016].
In several studies, LMS data are the basis for building an individual set of characteristics
of a particular student or student profile for more individualized support [Tempelaar et al., 2018],
[Jones, 2019]. Other researchers following Winne’s taxonomy of data use course performance mea-
sures, LMS system trace variables[Winne, 2013], [Zho & Winne, 2012], SIS based variables, and learn-
ing disposition variables for student profiling.
2.3 LMS data for student’s profiling in formative assessment technique
Since it is impossible to provide a general method for formative assessment based on LMS data,
teachers must themselves determine formative assessment techniques and tools designing e-learning
courses.
The study presents the approbation results of the case involving e-course data in formative
assessment implementation to help the student to review their learning actions features and polish
the individual path. The implementation peculiarities of this case are associated with the methodical
profile of a specific e-course. By the e-course methodical profile, it is implied the set of methodical
techniques and digital tools applied to arrange learning interaction based on digital resources. In
other words, this is the allocation of key features of e-resources on ton various grounds: according to
goals, according to levels, according to the variability of learning actions, etc. For each course, the
complexity of the characteristics will vary according to the key teaching method and the variety of
learning options.
Designing the information and communication environment of e-learning courses, teachers should
first focus on supporting students to understand the diversity of potential learning opportunities, and
rationally choose the personal way to achieve results. In the initial stages of training, students do not
always know how to prioritize and predict how they will act. Therefore, in the learning process, it is
very useful to receive feedback information, that helps to understand the effectiveness of the efforts
made.
We can apply such formative assessment methods as continuity assessment, standard compliance
assessment, selected target assessment level, preferred digital tool assessment, etc.
According to the proposed evaluation procedure, at the entry stage of the formative assessment
cycle, teachers and students perform the methodical overview of the e-course together.
Meanwhile, student’s attention focuses on the variability of learning opportunities, in particular
on the ability to control learning activity using data, accumulated in LMS due to users’ actions.
It is proposed to systematize the educational opportunities of the electronic course on three inter-
related grounds, consistent with the psychodidactic concept of learning in a digital environment
[Noskova, 2020]:
• techniques that contribute to theoretical concepts learning,
• communication techniques that provide various types of interaction of students with the teacher
and peers;
• methods of managing learning actions aimed at self-regulation and self-control.
Following this, students receive information about the principles of monitoring their educational
3
activities in the LMS and the extraction of educational data, which will be used to shape an individual
student profile.
The teacher, at his own discretion, determines the stage or stages of mastering the course when he
refers to the data of the system to demonstrate to students their personal profiles. The main purpose
of this formative assessment technique is to strengthen the reflective grounds for the student’s personal
learning trajectory self-correction. The student’s profile contains several bases for assessment in line
with the multifunctional e-course profile.
Following the general formative assessment idea, the student’s current personal learning profile
is the basis for the reflexive comprehension of his efforts and outcomes. It also helps decision-making
in teaching support.
Commonly, teachers, even realizing that the system is accumulating educational data, do not
use them at all or use only formal reporting applications, and do not constitute them as a part of a
full-fledged formative assessment cycle.
Personal student’s profile creation based on e-course learning activities monitoring can be carried
out through the cooperation of a teacher with a technical specialist who will help to compose the
appropriate requests for data extraction from the database through “ad hoc” queries.
Hence, the following sequence of steps can be proposed in the frame of formative assessment
technique applying the e-course data (fig.1):
- Systematization of the e-course learning opportunities on the basis of the three learning action
types:
* opportunities of knowledge acquisition, adaptation, and assimilation,
* opportunities of educative communication;
* opportunities to manage educative interaction;
- Correlation of the e-course methоdical profile with possible data sources of the learning man-
agement system for subsequent include data to personal students’ activity profile;
- Design a set of LMS database queries (in cooperation with technical experts);
- Extraction data from other data sources (surveys, classroom activities);
- Personal learning activity profiles visualization;
- Fixing of the personal profiles analysis stages while mastering the e-course (as formative assess-
ment tool);
- Demonstration of current personal learning profiles to students;
- Learning actions correction options identifying based on reflexive comprehension of personal
profiles;
- Personal learning profiles demonstration at a final e-course stage.
Matching personal profiles at various e-course stages can provide useful information for students
and teachers to improve the learning process. Obviously, it is not necessary to use this formative
assessment technique for all e-courses. But even a single application leads students to understand
that it is useful to comprehend the efficacy of their learning actions on different grounds, to choose
the preferred educational opportunities, and consciously build a personally efficacious route to achieve
learning results. As a result, this technique benefits metacognitive skills development.
4
Figure 1: The sequence of steps of formative assessment technique applying the e-course data. Source:
Own work
3 Results of Research and Discussion
Experimental work was carried out within the framework of the e-course “Multimedia technologies”
for master students of the educational program “Information Technologies in Education.”
The initiation of the formative assessment technique based on the e-course methodical profile and
the personal student’s learning actions profile was predetermined by the problems that were identified
in the course in the previous semesters. In particular, students did not pay enough attention to the
variable learning resources and assignment selection. Some students formally treated the discussion
of problematic issues and tasks for joint work, performed tasks out of time, that interfered with
joint work. Personal remarks to a specific student were not effective enough, since they did not
demonstrate learner the general nature of his activity and did not contribute to its complex reflective
comprehension.
Based on the methodical profile of the course, the sources of educational data identified in
three categories: data demonstrating the student’s interaction with the e-course content, data on
the student’s communication activity (several tasks performed on the forum), and data reflecting the
regulatory features of the student’s learning actions.
In the first category, in addition to the general quantitative features of students’ utility of e-
course resources, data related to assignments of various levels are collected. The tasks differed not
only by the complexity level but also involved various intellectual actions with thematic information
objects.
In the structure of each subject module of the course, an interactive lecture was designed. In
order to master the content of the lecture, the student was recommended to choose several tasks with
specific workspaces (shared documents). Students have the opportunity to work on multiple tasks
at the same time and make a certain contribution to the result of the joint work according to their
personal aspirations and preferences. Therefore the interactive lecture and task workspaces serve as
a data source for student’s learning action profile.
The student’s personal profile reflected quantitative data on selected tasks and his contribution
to the specific task.
The communication segment of the student’s personal action profile included such indicators
as collaborative assignment statistics, quantitative indicators of questions asked proactively in the
course forums, and quantitative indicators of answers in the course forums.
The personal profile segment, reflecting the regulatory features of learning behaviour and is the
most complicated part. It comprises data on attending classes, performing invariant, variable, extra
5
assignments and assignments of various levels; timeliness of questions and answers in the forums.
The data sources are mainly provided by standard LMS reports. The personal profile also contains
data about online classes attendance, although this indicator is not an important basis for students
to adjust their behavior.
The personal student’s profile deliberately did not include grades. This idea is also attributed to
the methodical specifics of the course, because it does not apply test control, and as the final task, the
students developed a multimedia project. To evaluate the final project, students were supplied with
assessment criteria, but there was no need to bring the creative students’ work to uniform quantitative
and qualitative characteristics.
In the final stage of the assessment, the comparison of the personal action profiles played a special
role. This technique made it possible to expand the scope of traditional summative assessment, i.e.,
students and the teacher had the opportunity not only to evaluate and discuss the final information
products but also to carefully analyze the process of moving towards educative results.
Table 1: Personal student’s action profile structure of pilot e-course
Figure 2 shows how much the profiles of students differ, although all students assessed positively
and received a credit in the discipline. During the ad hoc discussion, some students pointed out that
they were surprised by the significant difference between the nature of their educational activities and
6
the behaviour of their classmates (the information of other students was shown anonymously). This
served as a significant incentive for correcting further actions for all or some of the grounds.
Figure 2: Comparison of student’s profiles Source: Own work
Figure 3 shows the activities of the same student at the current and final stages of the assessment.
Significant changes have taken place in learning activities in terms of communication and task
selection.
Student profiles were also used as an additional objective basis for assigning a grade to the
discipline. This technique turned out to be more effective than the traditional use of cumulative
assessment in points, which does not provide an understanding of the nature of the activity, even if
the assessment carried out with clear assessment criteria.
Therefore, we found that for the students who participated in the e-learning course, the personal
activities data visualization method proved to be effective.
The universal part of this formative assessment technique involves identifying and grouping
variables of students’ learning activity on three grounds: interaction with learning resources, com-
munication activities, and regulatory aspects. The variables of the personal student’s activity profile
7
Figure 3: The profile of the same student at the current and final stages of the assessment. Source:
Own work
are determined at the stage of particular e-course potential analysis by identifying the main learning
opportunities for students. For the pilot e-course 13 variables were defined.
The definitions of all educative activity indicators should be discernible for students as a condition
for further correction based on the results of reflexive comprehension of the visualized activity profile.
One of the technique variants involves an input survey, to identify the students’ intentions for each
category of assessment. Such a survey will not only allow the teacher to receive feedback on students’
motivation but also allow focusing the attention of students on the specific course facilities.
Conclusion
Therefore, the results of the study demonstrated the impact of the formative assessment method
applying e-course data to visualize a student’s personal activity profile. It can be adapted to e-courses
of various disciplines and functionality. Formative assessment technique effect will manifest itself for
other disciplines due to students’ acquisition of new experience to assess their learning activity on
various grounds.
Further research is needed to highlight the principles of shaping personal activity profile for
various types of e-learning courses. It is also important to point out that the teacher’s assessment
activity changes significantly with such formative assessment methods implementation. Efforts from
the evaluation of educational achievements to the evaluation of students’ independent educational
behaviour and flexible management have been redistributed. Detailed information on the e-course
resources use and student’s activities allows a teacher to understand clearly how to improve the
content of the course and teaching skills.
Acknowledgement
The research was supported by by the Ministry of Science and Higher Education of the Russian
Federation (project No. FSZN-2020-0027).
References
[Aldowah et al.,2019] Aldowah H., Al-Samarraie H., Fauzy W. M. (2019). Educational data mining
and learning analytics for 21st century higher education: a review and synthesis. Telematics
Inf., Vol.37, Pp. 13–49.
[Andersson & Palm, 2012] Andersson C., Palm T. (2012). Characteristics of improved formative as-
sessment practice. Education Inquiry, 2017, vol. 8 (2), Pp. 104–122
8
[Black William, 2009] Black P., William D. (2009). Developing the theory of formative assessment.
Educational Assessment, Evaluation and Accountability, Vol. 21 (1), Pp. 5–31.
[Brink & Bartz, 2017] Brink M., Bartz D. (2017). Effective Use of Formative Assessment by High
School Teachers. Practical Assessment, Research, and Evaluation, Vol. 22, Art. 8. Pp. 1–10.
[Clark, 2012] Clark I. (2012). Formative assessment: Assessment is for self-regulated learning. Edu-
cational Psychology Review, 24, Pp. 205–249
[Goldman, 2016] Goldman, J. (2016). Privacy and individual empowerment in the interactive age. In
C. J. Bennett, R. Grant (Eds.), Visions of privacy: Policy choices for the digital age, Toronto:
University of Toronto Press, Pp. 97–115.
[Granberg et al., 2021] Granberg C., Palm T., Palmberg B.(2021). A case study of a formative as-
sessment practice and the effects on students’ self-regulated learning, Studies in Educational
Evaluation, Volume 68, Pp. 1-10.
[Ifenthaler & Widanapathirana, 2014] Ifenthaler D., Widanapathirana, C. (2014). Development and
validation of a learning analytics framework: Two case studies using support vector machines.
Technology, Knowledge and Learning, Vol. 19(1-2), Pp. 221-240.
[Jivet et al., 2018] Jivet I., Scheffel M., Specht M., Drachsler H. (2018). License to evaluate: Prepar-
ing learning analytics dashboards for educational practice. In: Proceedings of International
Conference on Learning Analytics and Knowledge, Pp. 31–40.
[Jones, 2019] Jones K. (2019). Learning analytics and higher education: a proposed model for estab-
lishing informed consent mechanisms to promote student privacy and autonomy. Int J Educ
Technol High Educ Vol.16 (24)
[Lemay et al., 2021] Lemay D., Baek C., Doleck T.( 2021). Comparison of learning analytics and
educational data mining: A topic modeling approach. Computers and Education: Artificial
Intelligence, Vol. 2, Pp. 1-14.
[Long Siemens, 2011] Long P., Siemens G. (2011). Penetrating the fog: Analytics in learning and
education. EDUCAUSE Review, Vol.46(5), Pp. 31–40.
[Noskova & Pavlova, 2017] Noskova T., Pavlova T. (2017). Pedagogical activity tools in electronic
environment. Education Online. Vol. 8/9 (215), Pp. 121-130 (in Russian).
[Noskova, 2020] Noskova Т. (2020). Didactics of the digital environment. Herzen University. Pp.382
(in Russian).
[Pellegrino & Quellmalz,2010] Pellegrino J., Quellmalz E. (2010). Perspectives on the Integration of
Technology and Assessment’. Journal of Research on Technology in Education, 43 (2), Pp.
119-134.
[Rickards & Stitt-Bergh, 2016] Rickards W.N., Stitt-Bergh M. (2016). High education evaluation,
assessment and faculty engagement. New Directions for Evaluation, Vol. 151, Pp. 11–20.
[Shum & Crick, 2012] Shum, B., Crick, R. (2012). Learning dispositions and transferable competen-
cies: Pedagogy, modelling and learning analytics. In S. Buckingham Shum, D. Gasevic, R.
Ferguson (Eds.), Proceedings of the 2nd international conference on learning analytics and
knowledge, Pp. 92-101.
9
[Tempelaar et al., 2018] Tempelaar D., Nguyen Q., Rienties B. (2018). Student profiling in a disposi-
tional learning analytics application using formative assessment. Computers in Human Behavior,
Vo. 78, Pp. 408-420.
[Trumbull & Lash, 2013] Trumbull E., Lash A. (2013). Understanding formative assessment: In-
sights from learning theory and measurement theory. San Francisco: WestEd. Available at:
https://www.wested.org/onlinep ubs/resource1307.pdf 2013 (accessed 21 March 2021).
[Verbert et al., 2012] Verbert K., Manouselis N., Drachsler H., Duval E. (2012). Dataset-Driven Re-
search to Support Learning and Knowledge Analytics. Educational Technology Society, Vol.15
(3), Pp.133–148.
[Viberg et al., 2018] Viberg O., Hatakka M., Bälter O., Mavroudi A. (2018). The current landscape
of learning analytics in higher education, Computers in Human Behavior, Vol. 89, Pp. 98-110.
[Vikulina & Vilkova, 2019] Vikulina М., Vilkova L. (2019). Elements of the methodology of formative
assessment of educational achievements in mastering a foreign language by university students.
PNRPU Linguistics and Pedagogy Bulletin. Vol. 2. Pp. 131 – 139 (in Russian).
[Winne, 2013] Winne P. (2013). Learning strategies, study skills and self-regulated learning in post-
secondary education. In: M. B. Paulsen (Ed.), Higher education: Handbook of theory and
research. Vol. 28, Pp. 37-40.
[Yorke, 2003] Yorke M. (2003). Formative assessment in higher education: Moves towards theory and
the enhancement of pedagogic practice. Higher Education, Vol. 45 (4), Pp. 477–501.
[Zho & Winne, 2012] Zho, M., Winne P. (2012). Modeling academic achievement by self-reported
versus traced goal orientation. Learning and Instruction, 22, Pp. 413-419.
10