Learning Analytics as an Investigation Tool for Teaching Practicioners

Abstract

Formative assessment plays an important role in teaching, to both provide learners with valuable feedback and to improve the teaching process. Recently, novel tools have been developed to support teachers in this context. These tools allow one to analyze the students' activity in detail and guide and improve the teaching process. We present one such tool, and the scenarios in which this tool would be used, following an evaluation. From these scenarios, we deduce the structure of a general analytical process for teachers that yields general requirements for the corresponding analytical tools.

Introduction

Feedback on the learner’s performance represents an indispensable element in learning scenarios. Adequate feedback presented in a timely fashion may improve and accelerate student learning. However, the task to provide such feedback is challenging in classical learning scenarios due to the required effort to analyze students’ performance. Novel techniques and tools from the field of computer-based assessment (CBA) may overcome this problem. These allow for solving exercises at a computer, recording not only the final outcome, but also the whole of a student's learning process on the level of individual solution steps. This is especially important in cases where feedback is required not only based on the reproduction of factual knowledge, but on the application of strategies and algorithmic thinking, such as in various fields in Mathematics. Applying corresponding forms of formative CBA may easily result in large amounts of log data from learners. In principle, this data may allow teachers to analyze a learner’s performance in detail, to provide helpful feedback to individuals, and to adapt teaching in general. Thus approaches to analyze the vast amounts of data are required. For instance, summary views are required to detect symptoms such as misunderstandings or excessive successes, while sampling methods have to be supported to infer a problem seen in summaries.

The contribution of this paper is an analysis of the analytical process of the teachers we aim to support, based on a range of scenarios. Moreover, we depict demands on teaching and learning analytics tools and visualizations.

The remainder of the paper is structured as follows: First we give an overview on the theoretical background and related work. Then, we introduce tools developed in the SAiL-M project showcasing novel opportunities to include CBA in the classroom, and how this enables teachers to analyze students’ performance in a new way. We present three different scenarios, which explain in more detail the different motivations of teachers for analyzing data, and analyze the processes observed. Open questions form the conclusion.

Background and Related Work

Formative assessment is intended to improve teaching and learning. This kind of assessment aims at providing an overview of the students’ understanding and at identifying areas of misconception. When conducting formative assessments, the teacher faces the challenge of interpreting the data obtained as assessment results and adapting instruction based on that information. When applying the concept of formative assessments in a university setting, the challenge already arises when trying to organize assignments for a large number of participants: assignments have to be corrected, errors have to be analyzed, data has to be collected and feedback to the students has to be given. Doing this on a regular basis, very little time remains for interpreting the obtained data and for thinking about strategies of instructional change.

A teacher’s role as a researcher or investigator is being increasingly highlighted. For example, the research review [ 10 ] highlights the importance of the teachers’ ability to synthesize students’ and tools’ usage. Corresponding approaches are connected to the field of action research [ 6 ], targeting to improve teaching through reflection and development of adequate action in their classroom setting. Adequate formative assessment typically also represents a crucial element in such approaches.

Computers have been proposed frequently as a means to perform assessments frequently and in an effective manner. In fact, early approaches in the area of programmed instruction include this notion [ 9 ]. Intelligent tutoring systems (ITS) also integrate different types of assessment components as part of their inner loop [ 12 ]. In both cases, the results of the assessment are being utilized to adapt the learning process. However, the corresponding information is rarely being provided to a teacher or tutor for further analysis. CBA refers to a set of different approaches for educational assessment, both in the classroom and in large-scale testing situations, based on question types with which computers can effectively interact, including scoring and score reporting, while still gathering meaningful measurement evidence [ 7 ]. In practice, most of the applied techniques boil down to multiple-choice-like questionnaires, which may provide limited detail on students' conceptions and misconceptions, especially in the context of conveying knowledge beyond factual and higher-level competencies. Intelligent CBAs [ 2 ] represents a recent approach to overcome these limitations, utilizing methods from the field of ITS to assess more complex solutions processes on the level of individual solution steps. As a result, such assessments may provide much more detailed information of students’ performance, raising demands in approaches to analyze the large amounts of recorded data efficiently. Different to ITS approaches, intelligent assessment does not rely on automatic assessment alone, but rather introduces a semi-automatic approach, thus causing further demand for adequate methods to support a teacher in the assessment of solution processes.

There have been a number of proposals for defining Learning Analytics, which to some extent take different objectives and only partially overlap [ 8 ]. We do connect to the field of Visual Analytics [ 5, 11 ], and for this reason we understand Learning Analytics as a specific focus and application area of Visual Analytics. That is, Learning Analytics relates to approaches and technologies targeted to allow for analytical reasoning facilitated by visual interfaces employed for teaching or learning. Objectives are the detection of interesting aspects and patterns in learner and learning data, building hypotheses based on these detected structures, confirming such hypotheses, drawing conclusions, and possibly communicating the results of this analytical process. In the context of this paper we will discuss how Learning Analytics relates to formative assessment, and what specific requirements can be stated on corresponding solutions from the perspective of a teacher-as-investigator. 3

Intelligent Assessment Tools

automatic assessement Learning

Tool launch tool

In the context of the research project SAiL-M (Semi-automatic Analysis of Individual Learning Processes in Mathematics), we have developed various interactive learning tools for the field of mathematics. Our learning tools implement the approach of intelligent assessment and use the general-purpose logging architecture SMALA (SAiL-M Architecture for Learning Analytics) for recording all semantically relevant interactions between the learners and the tools.

The SAiL-M learning tools are web-based software applications that can be accessed as learning activities from within a learning management system (LMS). Authenticated users of the LMS can use the learning tools and solve exercises interactively. The actions of the learner are analyzed and automated feedback is provided (generally detecting standard errors or standard solution paths).

The SMALA logging infrastructure provides the learning tools with the extra functionality of recording all interactions that occur between the learner and the learning tool. In order to document individual learning processes, the learning tools send all semantically relevant interactions as events to the SMALA logging service with information such as the pseudonym, the input, and the displayed feedback. The events are stored in the SMALA database and from there, the data logs get analyzed and represented by suitable log views. Authorized teachers can can access these log views from the SMALA web server thanks to the SMALA log architecture in Figure 1. Available log views vvileoileowgwgss include both summary views on activities and logs views performances of the whole group of learners, vaitcieti-s how to deploy aonfdinsdeisvsiidounalvileewarsnionng sptreopc-ebsys-esste.pWreecdoersdcirnibges them below. start tool session In the winter term 2011/2012 the learning LMS tools were used and evaluated by about 200 students and 6 teachers in the Universities of

Course Education in Heidelberg, Karlsruhe and Ludresource resource creates iwnivgessbtiugragt.eTbhoethgtohael aocfcetphteanecvealaunadtiounsefwualnsestos of the learning tools from the students’ point of view and from the teachers’ point of view.

The evaluations showed the toolset and observation mechanism to be acceptable for students and to allow them to seek help effectively (see [ 4 ]). Interviews with four participating teachers confirmed that individual learning processes were reproducible by the SMALA session views. However, teachers requested other statistical indicators and richer summary views for getting a general overview of the students’ activity and performance. In particular, they were interested in statistics on assessment results (e.g., type and number of detected problems), number of feedback requests and the level of activity per student, which were not available during the evaluation. Based on these outcomes, we have developed analytical process scenarios as a means to illustrate the integration of SMALA log views into the teaching practice and to get a better understanding of the teachers’ requirements for such log views. 4

Analytical Process Scenario

When integrating formative CBA tools in a classical teaching environment, the teachers typically introduce the learning tools during the lecture. The tools can then be used at home or in the lab. Based on the assessment results, the teachers can then decide on appropriate adaptions of their teaching strategy. In this section, we describe illustrative scenarios involving the usage of learning analytical tools.

Scenario I: Checking the students’ activity. The teacher introduces the topic of functions and relations. In order to demonstrate the concepts of injectivity and surjectivity, she interactively constructs relations by using the tool Squiggle-M, a tool to exploratively discover the properties of finite mappings. At the end of the session the teacher asks the students to do exercises 1 and 2 from the learning tool as homework and to perform their own explorations with the tool as they wish, with a one week delay. One day before the next session, the teacher checks whether the students have worked with the learning tool. She opens the SMALA log views for the Squiggle-M learning activity and gets a visual representation of the learner’s activities. According to the diagram most of the students have solved both exercises. Their level of activity shows that they were interacting intensely with the tool. Only a few students have done only one exercise with a low level of activity. Based on his experience, he decides that 74% is reasonable as a proportion of high involvement. cording to the view, several students have worked the first exercises. In order to get an impression of the students’ performance, the teacher opens a second view that shows the successful sessions. She realizes that there are only very few students that came to a correct solution. When drilling down to a representation of the type of problems that were detected by the automatic analysis component of the learning tool, which is linked to a detailed session view, the teacher realizes that most of the problems occurred in the step of finding the correct induction statement. According to this report, 80 % of the students gave up at this stage of the solution process. Because the teacher considers these findings critical, he decides to explain this part of complete induction again in more detail in the coming session.

Scenario III: Providing individual feedback. A student is doing the homework for the mathematics class and opens the learning tool ComIn-M in order to solve the exercises online. She selects the first exercise that the teacher asked the students to do. After some minor difficulties in entering the mathematical formulae, she successfully enters the base case for the proof by mathematical induction. In order to find out whether she did fine so far, she requests an automatic analysis of the current solution by pressing the “Verify” button. A green check mark appears on the screen, confirming that her intermediate solution is correct. She continues by selecting the correct assumption and then tries to figure out the induction statement. As she is not sure what to enter, she guesses a statement and requests an automatic analysis from the tool. This time, the tool marks the current solution as wrong and displays a short description of the problem that was detected. Now the student tries another solution, requests an analysis again, but again the tool reports a problem. The student is afraid that she cannot find the correct solution on her own, so she uses the “Ask Tutor” feature of the learning tool. By simply clicking the corresponding link, a dialog window opens and lets her enter a message to the tutor. When she submits the dialog, her message is sent along with a link to her SMALA logging session to the responsible tutor. Later that day, the tutor checks her email and finds a notification that a ComIn-M user needs personal help from her. She reads the message and follows the link to the SMALA logging session. This log view shows the recorded interaction sequence between the student and the learning tool until the point of help request as illustrated in Figure 3: it shows an easily readable overview of each the terms the student has input and all the problems that were reported by the tool.

Investigating the last state of the solution process, the teacher quickly finds out that the student did not replace the index variable correctly. So she sends her advice back to the requesting student, addressing the concrete problem that she detected in the solution of the Fig. 3. SMALA session view student.

4.1 The Analytical Processes

In the scenarios described above the teachers typically perform four parallel reasonings, which could be carried out alternating or in parallel: based on their knowledge of the domain and learning tools, they have expectations of the learners’ activities; these expectations are compared to the analytical views in an explorative browsing way; this browsing leads to interpretations of the learning processes, which results in strategies being assembled to further teaching actions.

Process 1: Determine the expectations. Based on their course plan and assign

ment, the teacher has expectations about the students’ usage of the learning tools. Typical expectations of interest would be: this assignment should have been fully (or barely) completed since it is easy (or challenging); can we find typical problems?; this technical problem is likely to happen; or expect to see sufficient evidence in the analytics views to decide on deepening a subject or not. These expectations are constantly adjusted based on the processes below.

Process 2: Log views analysis. Typically, teachers perform a multi-step analysis on the assessment results [ 3 ]. First, teachers look at overall scores and learning outcomes to get an overview of the general class performance. Such summaries of the assessment results should highlight weaknesses both by content area and by student. Thereby, it is possible to detect common problems and difficulties. It is also possible to identify low-performing students that need special support and further guidance. In a second step, teachers perform an in-depth analysis of selected individual solutions and errors. The detailed analysis shall reveal insights into the reasons for errors. Ideally, not only the product of learning should be considered in this analysis step, but the whole process leading to the final product of learning.

Process 3: Interpret the learning processes. The interpretation of the log views leads to an understanding of the learning process. The interpretations depend on the professional experience of the teacher and is often based on so-called “thresholds”. A threshold in this context is defined as a “criteria for determining whether student[s] performance[s] require[s] an instructional response” [ 3 ]. Teachers can use thresholds as an indicator as to whether a student has mastered the content covered by the assessment. This allows teachers to decide on the need for an adaptation to the teaching plan, for the classroom or for the individual student.

Process 4: Preparing the instructional response. The outcome of the analytic process is the most important and the most challenging objective: how can the interpreted analysis be turned into action? What are the necessary measures to address the detected problems? Again, it is the experience of the practicing teacher that can help in answering these questions. In this step, research meets practice by developing concrete teaching strategies from research findings. Although it is the individual teacher who is ultimately responsible for selecting an appropriate action strategy, there are numerous sources of ideas and suggestions for finding such strategies [ 1 ]. 5

Open Questions and Conclusions

Teachers do require special solutions for learning analytics tools that support them in the analysis of data from formative CBAs. Analytical processes such as the 4-step procedure described in this paper have to be supported. In addition, exploratory approaches with no initial hypotheses in mind must be supported. Here, hyperlinking between the representations certainly supports an exploratory investigation.

Currently, our SMALA interface is restricted to simple depictions of data corresponding to typical goals and analysis steps, as well as displaying the usage of the learning tools. We are currently extending this by developing standard graphs for detailed information and special visualizations supporting overview/detail representations and the interactive filtering and thresholding of data (e.g., table lens techniques [ 13 ]) as well as analyzing temporal aspects in the data [ 14 ]. A more detailed analysis is required to evaluate the potential of integrating such visualizations into the interface for regular teachers, which are not experts in exploiting visual representations.

References

1. Altrichter , H. & Posch , P. ( 2007 ). Lehrerinnen und Lehrer erforschen ihren Unterricht (4 . Auflage). Bad Heilbrunn: Julius Klinkhardt.

2. Bescherer , C. , Herding , D. , Kortenkamp , U. , Müller , W. , Zimmermann , M. ( 2011 ). ELearning Tools with Intelligent Assessment and Feedback . In S. Graf et al. (eds.): Adaptivity and Intelligent Support in Learning Environments . IGI Global.

3. Goertz , M.E. , Oláh , L.N. , & Riggan , M. ( 2009 ). Can interim assessments be used for instructional change? (CPRE Policy Brief R- 51 ). Philadelphia, PA: University of Pennsylvania, Consortium for Policy Research in Education.

4. Libbrecht , P. , Rebholz , S. , Herding , D. , Müller , W. , & Tscheulin , F. ( 2012 ). Understanding the Learners' Actions when using Mathematics Learning Tools . In: Jeuring, J. et al. (Eds.): Proceedings CICM 2012. LNCS 7362 . Springer.

5. Keim , D. , Mansmann , F. , Schneidewind , J., Thomas , J. , & Ziegler , H. ( 2008 ). Visual analytics: Scope and challenges . In S. Simoff, M. Böhlen , and A . Mazeika, editors, Visual D a t a M i n i n g , v o l u m e 4 4 0 4 o f L N C S , p a g e s 7 6 - 9 0 . S p r i n g e r . S e e http://www.springerlink.com/content/5275180h84863347/.

6. Kemmis , S. , & McTaggart , R. ( 1982 ). The action research planner . Victoria, Australia: Deakin University Press.

7. Scalise , K. & Gifford , B. ( 2006 ). Computer-Based Assessment in E-Learning: A Framework for Constructing “Intermediate Constraint” Questions and Tasks for Technology Platforms . The Journal of Technology, Learning, and Assessment , 4 ( 6 ). http://www.dokeos.com/doc/thirdparty/Computer%20Based% 20Assessment .pdf

8. Siemens , G. ( 2011 ). What are learning analytics ? See http://www.elearnspace.org/blog/2010/08/25/what-are -learning-analytics/.

9. Skinner , B. F. ( 1958 ). Teaching machines . Science 128 ( 967 - 77 ), 137 - 158 .

10. Slavin , R. E. , Lake , C. , & Groff , C. ( 2009 ). Effective programs in middle and high school mathematics: A best-evidence synthesis . Review of Educational Research , 79 ( 2 ), 839 - 911 .

11. Thomas , J.J. & Cook , K. A. ( 2006 ). A visual analytics agenda . IEEE Computer Graphics and Applications , 26 ( 1 ): 10 - 13 .

12. VanLehn , K. ( 2006 ). The Behavior of Tutoring Systems. Int. J. Artif . Intell. Ed., 16 ( 3 ), 227 -- 265 .

13. Rao , R. & Card , S. K. ( 1994 ). The table lens: merging graphical and symbolic representations in an interactive focus + context visualization for tabular information . In CHI'94 (pp. 318 - 322 ).

14. Aigner , W. , Miksch , S. , Müller , W. , Schumann , H., und Tominski, Ch.: Visual Methods for Analyzing Time-Oriented Data . IEEE Transactions on Visualization and Computer Graphics , Vol. 14 , No. 1 , 2008 .