The purpose of the case study is to substantiate the e-course data utilising methodology for students' autonomous learning actions support. The principles of e-course data extraction and application are associated with e- course specific instructional profile in line with interaction techniques and digital resources. The proposed approach involves the correlation of the e-course instructional profile and feasible LMS data sources for subsequent reflection in the student's action profile. A student's reflective comprehension of his personal actions profile at the learning process definite stages implies an effective formative assessment practice. The first results of teaching technique implementation are discussed based on the collected action profiles review and student feedback.
2.1
The case study reveals the idea of a formative assessment technique applying the learning related data of the e-course designed for student’s autonomous learning and its pedagogical support. The experimental group of masters studied the discipline “Multimedia technologies in the educational environment.” The methodology involves the design of personal students’ activity and performance profiles in accordance with the set of teaching techniques for organizing educational interaction. Personnel student profiles and recommendations for their analysis are considered as e-course management tools to support learning process control [Noskova & Pavlova, 2017].
The research was carried out in the fall semester 2020 - 2021 academic year. 2.2
Formative assessment ideas have extensive background in theory and practice. A significant number of pedagogical publications are devoted to the formative assessment possibilities and advantages [Black William, 2009], [Andersson & Palm, 2012], [Brink & Bartz, 2017], etc. Black William stated that formative interaction “must be analysed as reflecting a teacher’s chosen plan to develop learning, the formative interactions which that teacher carries out contingently within the framework of that plan . . . and the internal cognitive and affective models of each student of which the responses and broader participation of students provide only indirect evidence” [Black William, 2009].
Numerous publications provide justifications for flexible, reflexive teaching and self-directed learning practices based on continuous feedback. The goals and effective techniques at various digital formative assessment cases presented [Yorke, 2003], [Rickards & Stitt-Bergh, 2016], [Granberg et al., 2021], etc.
Brink Bartz claim that “A high school’s culture must embrace formative assessment as standard operating procedure in all classrooms” [Brink & Bartz, 2017] .
Particular attention is paid to the study of the implementation of formative assessment in a digital environment applying a variety of digital tools [Long Siemens, 2011], [Pellegrino & Quellmalz,2010], [Jivet et al., 2018]. In the modern educational process, formative assessment has close ties with learning analytics, which allows to receive, analyze and apply information about user actions in learning management systems “for purposes of understanding and optimizing learning and the environments in which it occurs” [Long Siemens, 2011].
With learning analytics trend associated research and practise cases “along the four following dimensions: computer-supported learning analytics, which includes dropout/retention, student performance, and evaluation; computer-supported predictive analytics, including collaborative learning and self-learning; computer-supported behavioral analytics, focused on modelling student learning; and computer-supported visualization analytics, such as graph or network based methods” [Aldowah et al.,2019].
The following learning data analysis goals typology distinguished: predicting learner performance and modeling learners, suggesting relevant learning resources, increasing reflection and awareness, enhancing social learning environments, detecting undesirable learner behaviors, and detecting affects of learners [Verbert et al., 2012].
To perform a formative assessment in a digital environment, a teacher must solve several significant problems in interconnection: data extraction for further analysis and determination of the purpose for their processing; data analysis methods selection; determining how to apply data analysis results in practice.
For formative assessment, combined LMS data and data obtained, for example, as a result of surveys, student’s personal data, can be applied. This approach is typical for Dispositional Learning Analytics (DLA). Shum Crick define DLA as “infrastructure that combines learning data (i.e. those generated in learning activities through the LMS) with learner data (e.g., student dispositions, values, and attitudes measured through self-reported surveys)” [Shum & Crick, 2012] .
The solution to the third problem includes actions to optimize educational interaction, communicating with students, and adjusting digital resources [Andersson & Palm, 2012].
The choice of a specific formative assessment option depends on the method used to teach the subject, on the one hand, and on the other hand, it changes the method to a certain extent. Vikulina and Vilkova also emphasize “the dependence of the effectiveness of formative assessment on its compliance with professional beliefs and its acceptance by teachers” [Vikulina & Vilkova, 2019].
Goldman underlines that the main task is to ensure that the obtained data can benefit persons to promote their ability “to make meaningful choices” [Goldman, 2016].
In several studies, LMS data are the basis for building an individual set of characteristics of a particular student or student profile for more individualized support [Tempelaar et al., 2018], [Jones, 2019]. Other researchers following Winne’s taxonomy of data use course performance measures, LMS system trace variables[Winne, 2013], [Zho & Winne, 2012], SIS based variables, and learning disposition variables for student profiling. 2.3
LMS data for student’s profiling in formative assessment technique Since it is impossible to provide a general method for formative assessment based on LMS data, teachers must themselves determine formative assessment techniques and tools designing e-learning courses.
The study presents the approbation results of the case involving e-course data in formative assessment implementation to help the student to review their learning actions features and polish the individual path. The implementation peculiarities of this case are associated with the methodical profile of a specific e-course. By the e-course methodical profile, it is implied the set of methodical techniques and digital tools applied to arrange learning interaction based on digital resources. In other words, this is the allocation of key features of e-resources on ton various grounds: according to goals, according to levels, according to the variability of learning actions, etc. For each course, the complexity of the characteristics will vary according to the key teaching method and the variety of learning options.
Designing the information and communication environment of e-learning courses, teachers should first focus on supporting students to understand the diversity of potential learning opportunities, and rationally choose the personal way to achieve results. In the initial stages of training, students do not always know how to prioritize and predict how they will act. Therefore, in the learning process, it is very useful to receive feedback information, that helps to understand the effectiveness of the efforts made.
We can apply such formative assessment methods as continuity assessment, standard compliance assessment, selected target assessment level, preferred digital tool assessment, etc.
According to the proposed evaluation procedure, at the entry stage of the formative assessment cycle, teachers and students perform the methodical overview of the e-course together.
Meanwhile, student’s attention focuses on the variability of learning opportunities, in particular on the ability to control learning activity using data, accumulated in LMS due to users’ actions. It is proposed to systematize the educational opportunities of the electronic course on three interrelated grounds, consistent with the psychodidactic concept of learning in a digital environment [Noskova, 2020]: techniques that contribute to theoretical concepts learning, communication techniques that provide various types of interaction of students with the teacher and peers; methods of managing learning actions aimed at self-regulation and self-control.
Following this, students receive information about the principles of monitoring their educational activities in the LMS and the extraction of educational data, which will be used to shape an individual student profile.
The teacher, at his own discretion, determines the stage or stages of mastering the course when he refers to the data of the system to demonstrate to students their personal profiles. The main purpose of this formative assessment technique is to strengthen the reflective grounds for the student’s personal learning trajectory self-correction. The student’s profile contains several bases for assessment in line with the multifunctional e-course profile.
Following the general formative assessment idea, the student’s current personal learning profile is the basis for the reflexive comprehension of his efforts and outcomes. It also helps decision-making in teaching support.
Commonly, teachers, even realizing that the system is accumulating educational data, do not use them at all or use only formal reporting applications, and do not constitute them as a part of a full-fledged formative assessment cycle.
Personal student’s profile creation based on e-course learning activities monitoring can be carried out through the cooperation of a teacher with a technical specialist who will help to compose the appropriate requests for data extraction from the database through “ad hoc” queries.
Hence, the following sequence of steps can be proposed in the frame of formative assessment technique applying the e-course data (fig.1): - Systematization of the e-course learning opportunities on the basis of the three learning action types: * opportunities of knowledge acquisition, adaptation, and assimilation, * opportunities of educative communication; * opportunities to manage educative interaction; - Correlation of the e-course methоdical profile with possible data sources of the learning management system for subsequent include data to personal students’ activity profile; - Design a set of LMS database queries (in cooperation with technical experts); - Extraction data from other data sources (surveys, classroom activities); - Personal learning activity profiles visualization; - Fixing of the personal profiles analysis stages while mastering the e-course (as formative assessment tool); - Demonstration of current personal learning profiles to students; - Learning actions correction options identifying based on reflexive comprehension of personal profiles; - Personal learning profiles demonstration at a final e-course stage.
Matching personal profiles at various e-course stages can provide useful information for students and teachers to improve the learning process. Obviously, it is not necessary to use this formative assessment technique for all e-courses. But even a single application leads students to understand that it is useful to comprehend the efficacy of their learning actions on different grounds, to choose the preferred educational opportunities, and consciously build a personally efficacious route to achieve learning results. As a result, this technique benefits metacognitive skills development.
Experimental work was carried out within the framework of the e-course “Multimedia technologies” for master students of the educational program “Information Technologies in Education.”
The initiation of the formative assessment technique based on the e-course methodical profile and the personal student’s learning actions profile was predetermined by the problems that were identified in the course in the previous semesters. In particular, students did not pay enough attention to the variable learning resources and assignment selection. Some students formally treated the discussion of problematic issues and tasks for joint work, performed tasks out of time, that interfered with joint work. Personal remarks to a specific student were not effective enough, since they did not demonstrate learner the general nature of his activity and did not contribute to its complex reflective comprehension.
Based on the methodical profile of the course, the sources of educational data identified in three categories: data demonstrating the student’s interaction with the e-course content, data on the student’s communication activity (several tasks performed on the forum), and data reflecting the regulatory features of the student’s learning actions.
In the first category, in addition to the general quantitative features of students’ utility of ecourse resources, data related to assignments of various levels are collected. The tasks differed not only by the complexity level but also involved various intellectual actions with thematic information objects.
In the structure of each subject module of the course, an interactive lecture was designed. In order to master the content of the lecture, the student was recommended to choose several tasks with specific workspaces (shared documents). Students have the opportunity to work on multiple tasks at the same time and make a certain contribution to the result of the joint work according to their personal aspirations and preferences. Therefore the interactive lecture and task workspaces serve as a data source for student’s learning action profile.
The student’s personal profile reflected quantitative data on selected tasks and his contribution to the specific task.
The communication segment of the student’s personal action profile included such indicators as collaborative assignment statistics, quantitative indicators of questions asked proactively in the course forums, and quantitative indicators of answers in the course forums.
The personal profile segment, reflecting the regulatory features of learning behaviour and is the most complicated part. It comprises data on attending classes, performing invariant, variable, extra assignments and assignments of various levels; timeliness of questions and answers in the forums. The data sources are mainly provided by standard LMS reports. The personal profile also contains data about online classes attendance, although this indicator is not an important basis for students to adjust their behavior.
The personal student’s profile deliberately did not include grades. This idea is also attributed to the methodical specifics of the course, because it does not apply test control, and as the final task, the students developed a multimedia project. To evaluate the final project, students were supplied with assessment criteria, but there was no need to bring the creative students’ work to uniform quantitative and qualitative characteristics.
In the final stage of the assessment, the comparison of the personal action profiles played a special role. This technique made it possible to expand the scope of traditional summative assessment, i.e., students and the teacher had the opportunity not only to evaluate and discuss the final information products but also to carefully analyze the process of moving towards educative results.
Figure 2 shows how much the profiles of students differ, although all students assessed positively and received a credit in the discipline. During the ad hoc discussion, some students pointed out that they were surprised by the significant difference between the nature of their educational activities and the behaviour of their classmates (the information of other students was shown anonymously). This served as a significant incentive for correcting further actions for all or some of the grounds.
Significant changes have taken place in learning activities in terms of communication and task selection.
Student profiles were also used as an additional objective basis for assigning a grade to the discipline. This technique turned out to be more effective than the traditional use of cumulative assessment in points, which does not provide an understanding of the nature of the activity, even if the assessment carried out with clear assessment criteria.
Therefore, we found that for the students who participated in the e-learning course, the personal activities data visualization method proved to be effective.
The universal part of this formative assessment technique involves identifying and grouping variables of students’ learning activity on three grounds: interaction with learning resources, communication activities, and regulatory aspects. The variables of the personal student’s activity profile are determined at the stage of particular e-course potential analysis by identifying the main learning opportunities for students. For the pilot e-course 13 variables were defined.
The definitions of all educative activity indicators should be discernible for students as a condition for further correction based on the results of reflexive comprehension of the visualized activity profile. One of the technique variants involves an input survey, to identify the students’ intentions for each category of assessment. Such a survey will not only allow the teacher to receive feedback on students’ motivation but also allow focusing the attention of students on the specific course facilities.
Therefore, the results of the study demonstrated the impact of the formative assessment method applying e-course data to visualize a student’s personal activity profile. It can be adapted to e-courses of various disciplines and functionality. Formative assessment technique effect will manifest itself for other disciplines due to students’ acquisition of new experience to assess their learning activity on various grounds.
Further research is needed to highlight the principles of shaping personal activity profile for various types of e-learning courses. It is also important to point out that the teacher’s assessment activity changes significantly with such formative assessment methods implementation. Efforts from the evaluation of educational achievements to the evaluation of students’ independent educational behaviour and flexible management have been redistributed. Detailed information on the e-course resources use and student’s activities allows a teacher to understand clearly how to improve the content of the course and teaching skills.
The research was supported by by the Ministry of Science and Higher Education of the Russian Federation (project No. FSZN-2020-0027).