Combining Multimodal Learning Analytics with Backward Design to Assess Learning

Abstract

In this position paper, I describe a potential avenue for leveraging multimodal learning analytics research to produce evidence about how learning analytics improves learning. Recently, several members of the learning analytics community have called for an increased focus on the learning side of learning analytics, particularly in generating an evidence base. I argue here that one method for better understanding learning via analytics is to utilize a backward design approach. In backward design, an instructor begins with a specific objective and assessment and designs the pedagogical approach to meet those objectives. I extend this practice to learning analytics and suggest learning analytics design also take a backward design approach: how do we design learning analytics to fit a specific learning context and give insight into whether or not the learning objectives were achieved? By focusing specifically on the learning objective in context, this approach may advance the field by generating specific evidence for how multimodal learning analytics can be designed to assess real-time learning, rather than trying to fit existing learning analytics to the learning objective. This may lead to actionable research that could help communicate information about learning both to the student and the teacher.

Keywords

• backward design
• assessment
• evidence
• biology education

INTRODUCTION

Clickers, e-textbooks, adaptive reading assignments, and learning management systems are examples of technologies used in the higher-education classroom that provide potential avenues for capture of multimodal student learning data. Much of this data, such as clickstream or time on task, is easily collected and may be low-hanging fruit for understanding learning. However, what are socalled “analytics of convenience” really telling us about learning? Some have cautioned drawing rigid conclusions on findings generated solely by learning analytics to avoid too much inference of what an individual’s behaviors mean (Siemens, 2015) . Others have called for focusing on the learning side of learning analytics, rather than the easily-capturable analytics (Hackbarth, 2017) . Furthermore, recent work in the learning analytics field has called for an increased focus by the community on generating evidence that learning analytics actually improves learning and pedagogy (Ferguson & Clow, 2017) . In the case of educational technology, the technology notoriously comes first, causing educators to design around the technology instead of vice versa (Laurillard, 2012). How can we leverage technology to meet the specific goals of educators, instead of forcing educators to adapt to the technology? In this position paper, I propose that we can combine multimodal analytics with the principles of backward design to create deductive analytics targeting explicit research questions or learning phenomena in specific contexts. By focusing on a specific aspect on learning and intentional design of analytics to meet those goals, this may lead to more evidence for how learning analytics can improve teaching and learning in practice. 2 2.1

BACKWARD DESIGN

Backward Design and Learning “Backward design” is a term coined by Grant Wiggins and Jay McTighe (2005) to describe a pedagogical approach where educators begin first with the desired learning outcome or result and then design the methods, materials, activities, and assessments to reach the desired learning outcome. Backward design has three distinct phases: (1) Identification of desired results, or determining what students should understand or be able to do after the unit/semester has passed; (2) Deciding what evidence, such as performance on an assessment, will demonstrate that the student achieved the desired outcome; (3) Designing appropriate instructional activities to fit the learning objectives and the method of assessment (Wiggins and McTighe, 2005) . An instructor will not necessarily pass through each of these stages in order, but may cycle between them as learning activities are developed (Whitehouse, 2014) . Said otherwise, in backward design the focus is on the ultimate learning goal and how that learning will be assessed instead of simply what topics need to be covered in a course, as dictated by tradition or a textbook (Wiggins & McTighe, 2005) . For example, in the context of my non-majors biology course, one of my learning objectives is for students to be able to relate authentic science practices and nature of science understanding to course topics. To achieve this goal, I have specific assessments (a group project, exam items) and methods of achieving those objectives, such as completing case studies in class. My approach is backward because I started with my learning objective in mind, not with a particular project, activity, or preferred textbook. The key benefits of backward design over traditional design is students are more likely to be “hands on, minds on” rather than engaging in habitual or entertaining tasks that may not necessarily contribute to student learning. 2.2

Comparing Backward Design to Learning Design

Learning design is defined as using design knowledge when developing a learning experience, including full courses or individual lessons (Koper, 2005) . Good learning experiences have good design at their base, and this design is generalizable to other learning experiences (Koper, 2005) . Backward design does not necessarily have any underlying design that is generalizable to other learning experiences. If two learning experiences have similar objectives, it may be possible that one can generalize to the other. One could consider backward design as a facet of overall good learning design. When applying learning analytics to design, one method of applying useable pedagogical feedback is to design analytics to capture the learning process, or certain checkpoints to monitor student progress (Lockyer, Heathcote, & Dawson, 2013) . However, this application of learning analytics to understanding pedagogy relies on using existing metrics, such as viewing student downloads from a learning management system to monitor student progress in a course or using social-network analysis to see how students complete a task (Lockyer, Heathcote & Dawson, 2013) . Using a backward design approach, not only is the learning environment designed around certain objectives, but the learning analytics are intentionally designed as well around those objectives. 2.3

Backward Design and Multimodal Learning Analytics

Using a backward design approach to design of multimodal learning analytics, researchers would start with a theory-driven research question or learning phenomenon and then choose or design analytics to match the question at hand. Although analytics of convenience or extant technologies may be useful, in the context described here, their existence is considered secondary to the educational objective. In this way, we are considering “what education needs from technology” (Laurillard, 2012, p. 8) rather than what technology is available for education and research. Use of a backward design paradigm with multimodal learning analytics parallels design-based research in that both involve the researcher working to design materials according to the specific context of interest (Barab & Squire, 2004). Multimodal analytics are of particular use since learning occurs in both digital and physical spaces, and allows a more robust method for application of backward design when choosing and implementing learning analytics. 2.4

Examples of Learning Analytics Work

In my work, we recently examined the language used by experts and novices as they engaged in simulated authentic science inquiry (Peffer and Kyle, 2017). Experts and novices differed in their expertise in authentic science practices, and we used analytics to determine which verbs were used more frequently by experts or novices. The use of expert-like hedging language is one of many sophisticated practices that my current work is pedagogically targeting. Another example of backward design in analytics is the work of Quigley, Ostwald, and Sumner (2017 ) which examined the modeling practices of high school students using EcoSurvey, a tool used to model ecological systems. Using modeling theory as a guide, the authors designed the analytics to capture important sequences used by the students and detect differences between teachers. Their work may provide insights in how teachers can receive personalized feedback on their instruction to promote their professional development. This is in contrast to studies such as Samson, Czarnik, and Gross (2017 ) or Park, Denaro, Rodriguez, Smyth, and Warschauer (2017) where easily capturable digital behaviors, such as clickstream data, were used to examine student performance. The analytics were not customized, such as in the backward design approaches used by Peffer and Kyle (2017) or Quigley, Ostwald, and Sumner (2017 ).

3 APPLYING BACKWARD DESIGN TO MULTIMODAL LEARNING

ANALYTICS

Since assessment is a key component of backward design, using multimodal learning analytics as assessments embedded in a backward design paradigm is logical and could provide many useful insights about learning. Within the context of today’s classroom, which coexists in both digital and physical spaces, using multimodal analytics could be particularly advantageous in backward design. The key benefit to using backward design in a multimodal context, with many options for capturing analytics, is to be deliberate in choosing what kinds of analytics will be the most useful to examine. How do we use each space to best capture data around a learning episode? How could devices in the physical world such as biometric sensors or smart furniture be combined with analytics in the digital world such as clickstream or natural language processing? For example, in my non-majors biology course we often discuss high profile current events such as controversial genetic technologies. Say I task students to work in groups and research a polarizing topic. Each group would then present an argument to the class, citing evidence that they found. A possible research question could be how do students choose and evaluate evidence. From the digital perspective, I could examine how many different sources of information are used, for how long they are accessed, and in what order students viewed the sources. In the physical space, I could examine language between participants around the topic at hand and biometrics. For example, what kind of biological response occurs when a student looks at contradictory information? How does this relate to their interactions with their peers? What does the data taken together tell us about learning in a multimodal space? The key differentiating factor here is starting with what I want to know rather than what is already available and designing or choosing analytic techniques to suit the learning and research objectives. 4

DISCUSSION

Although the potential for learning analytics to revolutionize research and teaching in the digital era is undisputed, there is a need for deductive, theory-driven learning analytics research to advance the field and leverage these new insights into actionable research that improves student learning outcomes (Hackbarth, 2017) . Furthermore, educational needs and goals should be considered when designing analytics, and not vice-versa (Laurillard, 2012). Rather than look at easily captured data or “analytics of convenience” (e.g., clickstream data, time spent logged into a Learning Management System) and correlating these behaviors with student performance in a course, the proposed application of learning analytics here follows a backward design approach where the learning analytics are designed across physical and digital space to help achieve or assess specific learning objectives. For example, Diana et al. (2017) described how a real-time dashboard could be used by an instructor to match low and high performing students. In the hypothetical example above, an instructor could use a real-time dashboard to facilitate just-in-time teaching where the instructor views each group’s progress and intervenes as needed based on the information presented on the dashboard. The analytics are intentionally designed across spaces to meet the pedagogical needs of the teacher or to provide information to the student.

Using backward design and intentionality about what will be collected and why it is important to collect will fine tune efforts to better understand learning through the use of analytics. This is particularly advantageous when considering how to meet the need in the learning analytics field to generate evidence that the learning analytics field is improving student learning (Ferguson & Clow, 2017) . Although important insights about learning can be obtained via easily capturable analytics, and oftentimes this is an excellent place to start, it is also important to balance these studies with the focused, backward approaches proposed here. This may also be important when considering what methods for capturing analytics across spaces are the best investments for limited resources. Is that cool new technology fun to use, or is it going to provide important information about learning? Are we choosing a modality because it is the hot new thing (and therefore may not be that useful), or because it will help us achieve a specific goal? Wiggins and McTighe (2005) refer to these activities as “hands-on without being minds on;” learning is limited to the activity, and is not long lasting. I encourage those designing studies to consider what aspect of learning they wish to understand through learning analytics and intentionally choose what kinds of multimodal analytics to utilize. This mindset will help generate the evidence needed to give credence to the field of learning analytics, and shift the focus from the analytics to the learning. 5

ACKNOWLEDGEMENTS

I would like to thank Alan Hackbarth and David Quigley for enriching conversation on this topic and helpful comments while preparing this manuscript. Peffer, M. E., & Kyle, K. (2017, March). Assessment of language in authentic science inquiry reveals putative differences in epistemology. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 138-142). ACM.

the learning sciences, 13(1), 1-14.

Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The journal of Diana, N., Eagle, M., Stamper, J. C., Grover, S., Bienkowski, M. A., & Basu, S. (2017, March). An instructor dashboard for real-time analytics in interactive programming assignments. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference. ACM. (pp. 272-279).

References

Ferguson , R. , & Clow , D. ( 2017 ). Where is the evidence? A call to action for learning analytics . In Proceedings of the Seventh International Learning Analytics & Knowledge Conference. ACM.

Hackbarth , A. ( 2017 ). Are we Losing Sight of the Treees for the Forest? A Case for Localized Longitudinal Analytics . In Shehatea, S. & Tan , J.P-L (Eds .), Practitioner Track Proceedings of the 7th International Learning Analytics and Knowledge Conference (LAK17) (pp. 77 - 82 ). Simon Fraser University, Vancouver, Canada ( 13 -17 March): SoLAR. https//solarresearch.org/wp-content/uploads/2017/02/ Final-LAK17- Practitioner-TrackProceedings.pdf

Koper , R. ( 2005 ). An introduction to learning design . In Learning Design (pp. 3 - 20 ). Springer.

Laurillard , D. ( 2013 ) Teaching as a design science: Building Pedagogical patterns for learning and technology . Routledge.

Lockyer , L. , Heathcote , E. , & Dawson , S. ( 2013 ). Informing pedagogical action: Aligning learning analytics with learning design . American Behavioral Scientist, 57 ( 10 ), 1439 - 1459 .

Park , J. , Denaro , K. , Rodriguez , F. , Smyth , P. , & Warschauer , M. ( 2017 ). Detecting changes in student behavior from clickstream data . In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 21 - 30 ). ACM.

Quigley , D. , Ostwald , J. L. , & Sumner , T. ( 2017 ). Scientific modeling: using learning analytics to examine student practices and classroom variation . In Proceedings of the Seventh International Learning Analytics & Knowledge Conference . pp. 329 - 338 . doi:/10.1145/3027385.3027420.

Samson , P. , Czarnik , A. , Gross , M. ( 2017 ) Relationships Between Digital Measures of Student Engagement and Exam Scores: Is the LMS enough ? In Shehatea, S. & Tan , J.P-L (Eds .), Practitioner Track Proceedings of the 7th International Learning Analytics and Knowledge Conference (LAK17) (pp. 77 - 82 ). Simon Fraser University, Vancouver, Canada ( 13 -17 March): SoLAR. https//solarresearch.org/wp-content/uploads/2017/02/ Final-LAK17-PractitionerTrack-Proceedings .pdf

Siemens , G. Learning Analytics of and in Mediational Processes of Collaborative Learning. 2015 Computer Supported Collaborative Learning Conference , University of Gothenburg, Gothenburg, Sweden. 10 June 2015 , Symposium.

Whitehouse , M. ( 2014 ). Using a backward design approach to embed assessment in teaching . School Science Review , 95 ( 352 ), 99 - 104 .

Wiggins , G. , & McTighe , T. ( 2005 ). Understanding by Design. (Expand 2nd ed .). Alexandria, VA: Association for Supervision and Curriculum Development .