Learning analytics based formative assessment: Gaining insights through interactive dashboard components in mathematics teaching Kholod Abu-Raya 1, and Shai Olsher 1 1 University of Haifa, 199 Abba Hushi Blvd., Haifa, Israel Abstract Conducting a student centered discussion in a mathematics classroom is not a trivial task. A teacher must follow their students work, and then use the relevant information in order to conduct a meaningful discussion. This challenge is greater when digital environments are involved, or when the students work remotely and only submit their work online. One possible solution for this challenge could be in the form of accessible learning analytics that could assist the teacher to gain insight about their student’s work. We report on a formative assessment platform that automatically analyzes student submissions and characterizes them according to preset conditions that are topic specific. These characterizations are then used in several interactive reports that form a teacher’s dashboard that is designed to enable multiple levels of analysis by the teachers in planning a classroom discussion. We describe the different components and demonstrate their use in mathematics classrooms in Israel. Keywords 1 Teacher dashboard, topic specific learning analytics, online formative assessment 1. Rationale and Background Teaching mathematics with technology requires specific competencies [1],[2]. And when executing blended learning many teachers reach uncharted territory, as their practices require refinement in order to employ the different instruments available, specifically when using the STEP platform [2]. In order to facilitate this process, the teacher dashboard in the STEP platform is designed to have a low entry point, which enables teachers to use it quite similarly to common existing platforms, but also offers different paths for deeper and more complex analytics, available for the teachers as they become more comfortable and fluent with the platform. This report describes the dashboard components in a rising level of complexity: both of the use of the teachers, and for the richness and specificity of the analytics provided. In Blended learning settings teachers are required to assess in real time their student’s understanding in multi-participant classes and direct their teaching accordingly. Many technological learning environments have been developed to provide the teacher with an immediate snapshot of students' work [3]. Under these conditions, teachers choose examples for the discussion in an unsystematic way, or focused on errors [4]. Influenced by these approaches that call to provide information that can be used in the feedback process, technological platforms for formative and summative assessment have been developed. One example is ASSISTments, that combines teaching, learning, and assessment in parallel. The system performs an automatic evaluation, provides immediate results on the students' work, and offers them feedback in the form of questions, scaffolding and hints, while diverting the teacher's attention from giving grades for teaching and discussion [5]. Another platform is Desmos, which allows teachers to create interactive presentations, get sketches of graphs drawn by the students, and conduct discussions on their basis. Teachers can see in real time the screen of all students during their work, as well as the progress of the class as a whole in the control panel of the system [2]. The ECTEL2021: AI for Blended learning, September 20, 2021, Bozen-Bolzano, Italy EMAIL: kholodmoed@gmail.com (Kholod Abu-Raya); olshers@edu.haifa.ac.il (Shai Olsher) ©️ 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 1 STEP platform, a formative assessment platform implements the use of Example eliciting tasks, with more than one correct answer, to support conceptual learning. Students are asked to construct an example that meets or contradicts certain conditions, after which the environment analyzes the answers according to mathematical characteristics [6]. It can be seen above that digital learning systems enable a change in the role of the teacher. Teachers are released from the time-consuming task of evaluating student submissions, and can concentrate on their main function, which is guiding the class discussions in accordance with the students' needs [7]. Technological learning environments could provide solutions for teachers when they provide the teacher with an immediate picture of their students' work. Each environment offers different learning analytics (LA) for the teacher. LA is sometimes referred to as a single display that aggregates various indicators about learners, learning processes, and learning contexts into one or multiple visualizations. More studies are needed on the long-term effects and affordances of learning dashboards if these technologies are to become part of the common toolbox of teachers and learners in years to come [8], but it remains a challenge to provide a meaningful LA to the teacher in real time. Researchers have different attitudes toward the potential of technology in changing teaching practices. On one hand, some researches realized that the potential of technology to change teaching is unlikely to be significant [9]. Teachers who integrate digital systems in their classes choose to do so according to their usual habits and their views on teaching mathematics in general [10]. In Israel, one such environments that has been recently introduced in schools (STEP) showed the potential to change the teachers’ practice when conducting classroom discussions. Recent research found that when teachers are systematically exposed to the students' responses to the tasks, they found a larger and more varied set of characteristics of the students' responses than they initially expected. In other words, STEP helped teachers notice their students’ understanding differently compared to the usual practice. In addition, when teachers are provided with accessible statistical data, they used it in their decision making, shifting from focusing mostly on errors to decisions based on data that contain a variety of correct answers and errors [1]1. In the following we present STEP’s dashboard components. STEP currently has six different reports include: a table report, grid report, histogram, Venn diagrams, perceptual landscape, and bubbles report. Each of these components was designed to address specific pedagogical needs, and with these design principles and usage ideas was introduced to mathematics teachers in professional development programs. In this report we will focus on four of the components: table report, grid report, histogram report, and Venn diagrams. For each component we will present separately the goals, mechanisms, and use cases for a task that was enacted in Israeli middle schools (grades 8 and 9) on the topic of functions, during the 2020-21 school year. 2. STEP dashboard components 2.1. Grid report The grid report presents a snapshot of each one of the student submissions in a collage (Figure 1), in a presentation that resembles many other platforms (e. g. TI-Inspire). The goal of this report is to enable teachers to have all of the classroom’s student’s work accessible in a single report, while enabling the teacher also to filter student work according to pre-designed characteristics. This filtering process can assist the teacher in locating different phenomena in the students’ work rather than just go over all of the snapshots trying to locate work that is relevant for the ensuing classroom discussion of next task. The report has an anonymous mode as well as an identified mode for the display of student answers so that a teacher can choose between a general discussion to specifically highlighting different student’s work. In addition, STEP enables to determine which characteristics will be taken into consideration when assessing whether the submission meets the requirements of the task, and these characteristics are displayed in a different color. 2 Figure 1: Grid report with filtering interface expanded The filtering mechanism is quite similar to any e-commerce website in which you can look for a certain product, and then narrow the search by characterizing what you want and also do not want the search results to include. For example, if I am looking for a purse that is brown and medium sized, I can check each one of the characteristics for the relevant meta-data category (color, size). The grid report enables a similar mechanism which allows teachers to choose one or several characteristics, or their negation, and also create “and” of “or” relations between them (Figure 2), thus enabling in-depth analysis that is based on the characterizations of the answers. Beside the filtering of the answers, each picture is clickable, loading the actual interactive diagram that the student submitted, thus enabling the teacher to manipulate the diagram in the process of the classroom discussion, and also provides the teacher with a link to the individual student report generated for this submission (Figure 3). The use case for this report is for a task on finding a linear function equation from two given point, carried out in an 8th grade mathematics lesson. The students were asked to: “Construct a linear function whose graph passes through the two given points. If you believe this cannot be done, explain why “. The given interactive diagram included multiple linked representation of the function, and the students could choose the different points by pressing a “new points” button which generated semi randomized pairs of points[12]. The teacher chose to address the characteristic “two points with same Y value”. This teacher indicated that this characteristic describes a constant function, and helps students in efficiently solving this type of a task, saving them time. 3 Figure 2: Grid report with filtering activated, name display button (I), and/or buttons (II) and negation button activated(III) Figure 3: Grid report with active interactive diagram and link to student report 4 The teacher presented examples for student submissions with this characteristic (Figure 4), and asked the students what did these students do? What type of function do you get in this situation? Do they think it is easier? How would you find the equation? Furthermore, the teacher also addressed submissions in which the two point coincide which enables students to have non-constant functions. The teacher indicated that this choice of points enabled students to understand that an equation can be written without calculating the slope and plugging in the X and Y values of the points. Figure 4: Filtered grid report presented during a classroom discussion 2.2. Table report The table report presents a row for each student, and a column for each characteristic indicating which characteristics are evident and which are not for each student submission (Figure 5), in a presentation that is quite similar to a conventional spreadsheet. The goal of this report is to provide the teachers with elaborate information for each task for every student in the class. The table report was designed following requests from teachers to have a detailed description for each student separately. The table report enables the teacher to catch in a glimpse which characteristics are prominent in their student’s submissions and which are not. It also enables teachers to see which students did not submit an answer at all and potentially approach them. 5 Figure 5: Table report The use case for this report is for a task on finding several linear functions that intersects a given point, and also intersect the positive part of the Y axis, carried out in an 8th grade mathematics lesson. The students were asked to: “Choose a point and construct several linear functions' expressions whose graph passes through the given point and intersects the positive part of the y-axis “. The given interactive diagram included multiple linked representation and point generating button as described in section 2.1. The teacher indicated based on the report that none of the students paid any attention to the table of values provided in this task (numeric representation characteristic in the second column). The teacher emphasized that the table of values helps students identify characteristic of the function, and found it important to illustrate the value of the numeric representation, connecting it to the function. The teacher continued to show the students how the table of values could assist them in verifying their solutions. 2.3. Histogram report The histogram report presents a distribution of the characteristics across the different submissions. A single report shows for each characteristic a visual representation of a column which height corresponds with the number of submissions which have this characteristic (Figure 6). Note that characteristics that are not present in the student’s submissions do not appear in this report. The goal of this report is to provide a relative visual representation of all of the characteristics for a given task, providing insight into the relative distribution of characteristics among student submissions. This interactive report also enables to choose (by clicking) a certain characteristic and then display a filtered grid report below the histogram, enabling the teacher to further analyze student work directly. The use case for this report is for a task on quadratic functions, identifying the extremum and calculating the distance between to extrema of two different quadratic functions. The task: “Given the functions f(x), g(x) from the family y=a(x-p)^2+k. Claim: there is only one option for the functions f(x) and g(x) such that the distance between the extremum points of each of the functions is 5 units. If you think this claim is true provide the algebraic expression for each of the functions. If not. Give 5 examples for different functions using the interactive diagram”. The teacher is this case noticed that most of the students gave correct answers, and also that most of the students gave a distance that was not a vertical distance, which she did not expect. Initially she presented the filtered grid report for the characteristic “vertical distance” and conducted a discussion around it, and then moved on to the filtered grid report of a distance that has a slope (Figure 6) 6 Figure 6: Histogram report 2.4. Venn diagram The Venn diagram presents the interrelations between characteristics in the student’s submissions (Figure 7). The report shows up to three different characteristics simultaneously (due to the circles representation constraints of Venn diagrams), which are color coded to indicate which one represents which characteristic. The goal of the Venn diagrams is to show the distribution of more complex phenomena in the submissions of students. Phenomena that could not be captured by only one 7 characteristic but rather in the relations between several characteristics. The diagram provides indication whether certain characteristics coincide, or perhaps do not exist together, etc. The presentation provides also the total submission’s number thus enabling to analyze which phenomenon is more prominent in student work. Similar to the histogram report, clicking on a part of the diagram displays a filtered grid report according to the selected region (e. g. intersection of 3 characteristics). Figure 7: Venn diagram The use case for this report is for the same task for which the histogram report was used. In fact, it was the histogram report that encouraged the teacher to use another representation of the data for her classroom. In the histogram report the teacher noticed that most of the student submissions were correct, and also that most of the submissions had distance with a slope. This finding intrigued the teacher, since she did not expect that students that did not yet study how to calculate distance on a plain. Hence, the teacher moved on to the Venn diagram and checked the interrelations between the two characteristics (Figure 8). Figure 8: Venn diagram usage The teacher then noticed that only 4 out of 11 submissions were both correct and with a slope. She stated that even though there were many submissions with a slope - they were not correct, since the calculation of the distance took into consideration only the difference between the X values. Except for one student that used the Pythagorean theorem and calculated by the length of the hypotenuse the distance. So the teacher asked this student to reveal her solution and used it to explain to the other students why it was not correct to calculate the distance the way they did. 8 3. Summary and conclusion In this report we described dashboard components of an online formative assessment platform in mathematics (STEP). This platform is designed to facilitate blended learning by combining both automatic analysis of student answers for online interactive tasks, as well as accessible learning analytics in various levels of complexity available for teachers. The learning analytics are accessible through different interactive reports that could be used both in real-time teaching in the classroom and also in preparation and review of homework assignments towards a classroom discussion in the following lesson. The use cases presented show that teachers use the different reports while employing different strategies ranging from locating work with specific characteristics (e. g. choosing points with the same Y value), to meaningful complicated logical conjunction of characteristics (e. g. not many students both solved correctly and used a distance with a slope). These examples show that the design principles embedded in the platform indeed facilitate different usage schemes, and enable teachers to gradually gain expertise and address complex phenomena in their students’ work, phenomena made accessible by the different learning analytics components in the platform dashboard, and which exceeds well beyond the correctness of the answer. 4. Acknowledgements This research was supported by the Israel Science Foundation (Grant 522/13) and the Trump Family Foundation (Grant 191). 5. References [1] Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A new framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054. [2] Tabach, M., (2021). Competencies for teaching mathematics in the digital era: Are we ready to characterize them? In Inprasitha, M., Changsri, N., Boonsena (Eds.). Proceedings of the 44th Conference of the International Group for the Psychology of Mathematics Education, Vol. 1, pp. 32-47. Khon Kaen, Thailand: PME. [3] Clark-Wilson, A. (2010). Emergent pedagogies and the changing role of the teacher in the TI- Nspire Navigator-networked mathematics classroom. ZDM, 42(7), 747-761. [4] Stacey, K., Price, B., Steinle, V., Chick, H., & Gvozdenko, E. (2009). SMART assessment for learning. Paper presented at the Conference of the International Society for Design and Development in Education, Cairns, Australia, September 28 – October 1, 2009. [5] Koedinger, K. R., McLaughlin, E. A., & Heffernan, N. T. (2010). A quasi-experimental evaluation of an on-line formative assessment and tutoring system. Journal of Educational Computing Research, 43(4), 489-510. [6] Olsher, S., Yerushalmy, M., &Chazan, D. (2016). How Might the Use of Technology in Formative Assessment Support Changes in Mathematics Teaching? For the Learning of Mathematics, 36(3), 11-18. [7] Yerushalmy, M., & Elikan, S. (2010). Exploring reform ideas for teaching algebra: Analysis of videotaped episodes and of conversations about them. In Learning through Teaching Mathematics (pp. 191-207). Springer Netherlands. [8] Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., ... & Dillenbourg, P. (2016). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30- 41. [9] Kirkwood, A. & Price, L. (2013). Examining Some Assumptions and Limitations of Research on the Effects of Emerging Technologies for Teaching and Learning in Higher Education. British Journal of Educational Technology, 44(4), 536–543. 9 [10] Drijvers, P., Doorman, M., Boon, P., Reed, H., & Gravemeijer, K. (2010). The Teacher and the Tool: Instrumental Orchestrations in the Technology-Rich Mathematics Classroom. Educational Studies in mathematics, 75(2), 213–234 [11] Olsher, S., & Raya, K. A. Teacher's attention to characteristics of Parabola sketches: differences between use of manual and automated analysis. In Conference on Technology in Mathematics Teaching–ICTMT 14 (p. 213). [12] Bagdadi, J. (2019) Let Students Choose the Given: Design Principles for Technology- based Assessment. Paper presented at CIEAEM71, Braga, Portugal. 10