On Using Hybrid Pedagogy as Guideline for Improving Assessment Design Christian Köppe1[0000−0003−0326−678X] and Rody Middelkoop2 1 Utrecht University, Utrecht, Netherlands c.koppe@uu.nl 2 HAN University of Applied Sciences, Arnhem/Nijmegen, Netherlands rody.middelkoop@han.nl Abstract. An essential element of higher education is the assessment of student work, either formative for improving learning or summative for looking back at what has been achieved. However, applying assessments as part of larger assignments is prone to some phenomena such as stu- dents not being aware of the quality of their work during the assignment or assessing at non-suitable moments in time, resulting in unnecessary low grades. In this work we discuss dichotomy-thinking as possible rea- son and how Hybrid Pedagogy as design guideline can help with finding appropriate solutions. Besides discussing this approach in general we also provide concrete examples of how it was applied for the design of assessment strategies in a course on software engineering. Keywords: assessment design · hybrid pedagogy. 1 Introduction Assessments, both formative and summative, form an essential element in higher education: they provide insight in the outcomes of student learning, offer oppor- tunities for feedback and check whether learning goals have been met. A com- mon course design comprises various assessment types such as written exams or obligatory tests, and often there are one or more larger and longer-running assignments as well. These assignments are usually assessed after the final ver- sion has been handed in. Additionally, there are some moments when feedback is given or even an intermediate assessment, based on the student’s current work status. While this is an established approach, we still can observe some phenomena which potentially have a negative impact on student performance and hence hin- der learning, even though in various degrees. With respect to timing and quality issues, the following specific examples of such phenomena can be recognized: – Timing Phenomenon: Snapshot Assessments In many longer-running assignments such as projects, case studies or research trajectories, there are some fixed assessment moments: usually at the end of a course and somewhere midterm or after regular periods in time, e.g. every Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 30 C. Köppe and R. Middelkoop two weeks. These midterm assessments often result in lower grades because they are a snapshot in time, looking at work in progress. Many teachers use this approach for showing the students that they need to work harder and deliver better quality, hereby hoping that this motivates the students. But even though it might become obvious to students where their shortcomings are, the grade is given and usually can’t be improved. This is unnecessarily frustrating, especially if the students actually are able to deliver much better quality, just not at this snapshot moment. It seems that in that case the assessment is somehow disconnected from the desired learning outcomes. – Timing Phenomenon: Feedback Timing In many cases, students get feedback some time after they have handed in some work. Usually students have already continued working on the project or focus on other assignments, which makes it more difficult for them to relate the feedback to their work as it is not present in their heads anymore. However, there are also occasions where feedback is given directly, e.g. as part of an assessment session or during a work group. These moments are of more value as the feedback is easier to directly relate to the work, but these moments are usually also more rare as they require more time of the teacher. They furthermore often do not fit into the planning of both students and teachers. – Quality Phenomenon: Quality Unawareness Students often seem unsure about the quality of their final work or have a rough feeling about the quality but are unable to predict the grade. Even if they apply self-assessment they are not sure if the teacher will come to the same result. Therefore the final grades often come as surprise, either being lower or higher than expected which might result in decreasing confidence in a fair grading system. – Quality Phenomenon: Little Value of Feedback Feedback which is given by teachers as part of assessments is not always experienced as valuable by students. It is often for looking back and not experienced as relevant for future work. If the feedback focus is on the quality of the work (which it should), then it is often experienced as not directly relevant for getting a higher grade. So its value is limited in a grade-centered educational system as it is present in most higher educational institutions. There likely are many reasons for these phenomena. We assume that one reason is the lack of awareness of alternatives. As a consequence, educational de- signers of such assignment and assessment strategies tend to rely on well-known standard solutions. A potential cause for the lack of alternative educational de- sign solutions might be a proneness to thinking in dichotomies, which was also discussed for various other educational domains (see e.g. [14, 17, 21, 6]). We believe that consciously intermingling these dichotomies—the core of the concept of Hybrid Pedagogy—might help with opening the space for new solutions which positively influence the aforementioned phenomena and there- fore can help with improving the design of educational strategies. In this work we discuss how this hybrid approach was applied for the design of assessment On Using Hybrid Pedagogy as Guideline for Improving Assessment Design 31 strategies which potentially have a positive impact on student performance and in consequence improve learning. In the next section, we will shortly introduce dichotomies in education and describe the concrete dichotomies which have impact on the aforementioned phenomena. This is followed by an introduction to Hybrid Pedagogy and the description of concrete examples of how hybridity was used for finding alternative solutions which potentially influence the phenomena as described above. The paper concludes with a summary and an outlook on future work. 2 Dichotomies in Education A dichotomy can generally be defined as two things that are on the outer ends of a specific dimension and often contrasting or opposing each other. Such di- chotomies can be encountered in many aspects of education. In mathematics education, Sierpinska discussed the dichotomy of practical versus theoretical thinking and the issues related to this [17]. Warren et al. explore the dichomoty of everyday versus scientific modes of thinking in science learning [21]. Heames and Service discuss a variety of dichotomies which influence the teaching tech- niques in business education [6]. Stommel provides more generic examples from education such as physical versus digital learning spaces, informal versus formal learning contexts, individual teachers and students versus collaborative commu- nities, academic products versus learning process, or learning in schools versus learning in the world [18]. Following these examples, we can identify some specific dichotomies which might have impact on the above mentioned phenomena. We will discuss these also with respect to the timing and quality aspects. – Timing Dichotomy: Planning for Organization versus Planning for Learning Fixed assessment moments are often dictated by organizational planning is- sues and not intended as milestones, related to quality aspects. They are dictated by the availability of the assessor, a certain moment based on the total duration of the assignment (such as after each third or at the half of the assignment duration), academic holiday planning and other similar is- sues. This seems understandable as most educational institutions work with fixed time structures such as two semesters per year, courses with a delim- ited duration of multiple weeks to few months or other timely restrictions. It also happens that students take courses in parallel and that, with the goal of making studying more easy, assessment moments are distributed in a way so that students do not get overworked. The contrast would be to adjust assessment to student’s learning, meaning that assessment happens when students have achieved a learning goal or created a product that ful- fills some pre-defined quality criteria. However, there are only few examples related to this planning for learning-end of the dimension, one of them being Programmatic Assessment [16]. 32 C. Köppe and R. Middelkoop This dichotomy is an example where most educators choose, for above men- tioned reasons, mainly one dimension: planning for organization. This is likely one of the reasons for Snapshot Assessments. – Timing Dichotomy: Synchronous versus Asynchronous Feedback Most assignments contain the delivery of some work products, and students get feedback from teachers on these products. There usually are two feed- back modes: (1) synchronous feedback where feedback is given directly and immediately on some product and the feedback receiver and giver interact with each other and (2) asynchronous feedback where feedback is provided some time after the product has been handed in and no direct interaction takes place. Both have advantages and disadvantages. With synchronous feedback, which is usually given in working groups or face-2-face sessions, feedback given is more relevant because it arrives at a teachable moment. Students are still engaged in working on the product, they are still thinking about the task domain [4]. The disadvantage of synchronous feedback is that is costs more time, the teacher is not able to thoroughly look at the product, and planning is not easy as most teachers do not get sufficient time for providing larger amounts of synchronous feedback. The latter is also related to the issue of planning for organization. Asynchronous feedback offers the advantage that the teacher has more time to deeply assess the product and to provide more detailed feedback. Planning is not a big issue, it usually does not matter much if the feedback arrives a bit earlier or later. However, when feedback arrives students likely already continued working on the product or even moved on to the next assignments or learning tasks. In both cases the feedback is arriving when they are not engaged in working on the product anymore and therefore of less value. The spare use of synchronous feedback, mainly because of planning issues, and the disadvantages of the more often applied asynchronous feedback mode are contributing to the phenomenon Feedback Timing. – Quality Dichotomy: Teacher versus Student Grade Determination Responsi- bility Some dichotomies are not directly recognizable as such. When thinking about the responsibility of determining the grades for students, most teachers would not dare to argue that this responsibility lies anywhere else than by them- selves. Likely reasons are that they fear loss of control on the quality of the work or grade inflation due to student’s over-assessment. This means in con- sequence that the students never will be fully aware of what quality level they have achieved with their product and in consequence what grade they could expect for it. Self-assessment does provide some help here, but is of- ten applied independently from actual grading3 and therefore only partially helpful. There are however examples where teachers have students indeed grade their own work. If done well, most authors report various positive effects of self- grading such as quicker and more detailed feedback for students, deeper 3 For example, the widely used learning management system Blackboard has two distinct modules: one for assignments with grading and one for self-assessments. On Using Hybrid Pedagogy as Guideline for Improving Assessment Design 33 understanding of the topic, and greater awareness of own strengths, progress, and gaps [3, 5, 19]. Not involving students in the grading process is contributing to Quality Unawareness – Quality Dichotomy: Formative versus Summative Assessment As teachers we often either give feedback only—intended for supporting learning and improvement—or we provide a grade with some justification, usually after some work has been finished. These relate to the assessment functions of being either formative or summative. Both are valuable but also have some shortcomings: even though formative assessment helps the students to know where they stand, they are dependent on the teacher to provide them with this information. This feedback is also often experienced as todo-list by the students, potentially resulting in the effect that elements of their work where no specific feedback is given on are seen as good enough. Furthermore, the feedback which is valued most by students is which parts already are good enough for getting a sufficient grade. This kind of feedback does not trigger a growth mindset as they likely won’t do more work on parts which are already of sufficient quality. It keeps the students reactive. Summative assessment on the other hand is mainly for looking back. Its relevance for the students is often limited, as the work has been finished and the students usually already moved on to the next assignment or course. Only intrinsically motivated students see the value of such feedback as being relevant for future work as well. The distinction between formative and summative assessments contributes to student’s experience of Little Value of Feedback. The above described dichotomies are four examples which we assume to have impact on the phenomena. In the next section we will discuss how these di- chotomies can be addressed by using hybridity as explicit design guideline. 3 Hybrid Assessment Design As also described by Heames and Service, applying kaleidoscope thinking—using another viewpoint when something seems difficult from a certain point of view— is a good start for developing solutions [6]. Hybridity, or Hybrid Pedagogy, can be such a different viewpoint. It refers to “a mixture of different parts into a new breed, form or culture” and “in higher education implies a pedagogical design that mixes different discourses, formats, tools, people, contexts etcetera” [7]. Rorabaugh and Stommel describe hybridity as follows: ”[...] hybridity suggests hesitation at a threshold. Hybridity is not an attempt to neatly bridge the gap, but extends the moment of hesitation and thereby confuses easy categorization. And, as we allow two things to rub against each other, two things that might not otherwise touch, we invite them to interact, allowing synthesis (and even perforation) along 34 C. Köppe and R. Middelkoop their boundaries. As the digital and analog—the physical and virtual— commingle, we must let go of the containers for learning to which we’ve grown accustomed. We must open to random acts of pedagogy—to con- nections that are, like the web, associative and lively but sometimes violent and deformed. In this, hybridity is not always safe, moving in- cessantly (and dangerously) toward something new—something as yet undetermined.” [15, unpaginated] Hybrid Pedagogy is not a new concept and there are a growing number of examples of hybrid practices in education [13, 9, 8]. However, besides describing existing hybrid practices it can also help educational designers as design tool. Applying hybridity as guideline might help with widening the solution space through dissolving existing dichotomies. This was also applied during assessment design for a semester on object-oriented software engineering for addressing the phenomena described earlier in this work. The resulting solutions are described in more detail in the next sections. We hereby follow the flow of application of these solutions instead of discussing them separated into timing and quality aspects. – Solution: Self-Grading In the example in Figure 1, the unused solution space suggests to share the responsibility of determining the grades with the students. This could be done by sharing the responsibility with them (in various degrees) or even by completely moving it to the students (as applied in self-grading). Fig. 1. Explicit exploration of unused solution spaces (in green) towards students being responsible The semester we designed was for the second year of a part-time study Com- puter Science. The students were a diverse group with many already doing professional work in the field. Part of this semester was a long-running case study with various aspects of software engineering such as requirements elici- tation or software design. As we wanted to integrate academic and workplace learning and wanted to increase the value of the study for them, we also ap- On Using Hybrid Pedagogy as Guideline for Improving Assessment Design 35 plied the hybrid practice of Bring Your Own Assignment4 . This means that the students could decide themselves about the content of the case study as long as they were able to fulfill the assessment criteria described in a set of rubrics. However, assessing these work products would have cost the teachers much more time due to the potential variety of used techniques and application domains. Besides that we also wanted to help students being more aware of the quality of their products. The solution was to put the initial responsibility for grading to the students themselves and moving the responsibility of the teacher to determining if the result of this self-grading was correct. Whenever students think they achieved a certain quality level in a product, based on a self-assessment using the provided rubrics, they were encouraged to apply self-grading and hand in a grading request. Applying this self-grading helped the students to become more aware of the quality of their work. This effect was increased by adding the following practice of grade motivation. – Solution: Grade Motivation To make sure that this self-grading is done appropriately, we added the requirement that not only a grade (based on rubrics) had to be requested, but that also a sufficient motivation had to be added in order to show that the quality of the work is in accordance with the rubric quality level and associated grade. This motivation had to be not just a repetition of the rubric descriptions, but a thorough underpinning of the achieved quality level. Figure 2 shows an example of a complete grading request including (1) for whom it was, (2) for which assignment (the case study) and rubric, (4) the requested grade according to the quality description in the rubric, (4) the motivation for the grade, (5) the reference to the actual work product, and (6) a link to the grading queue tool (see Grading Queue solution below). Fig. 2. Example of a grading request (in Dutch) including the grade motivation (#4), adapted from [11] 4 Bring Your Own Assignment [13]: Students are less motivated to work on offered standard-assignments, so have them work on assignments they proposed themselves. 36 C. Köppe and R. Middelkoop Having to provide this motivation increased the awareness of the quality of their work, as they had to determine it much more deeply and explicitly. This could in consequence also lead to a general improvement of their self- assessment skills, which is part of future research. – Solution: Continuous Assessment The first idea during designing the semester was to define some of fixed assessment moments. But when looking at it using the hybridity viewpoint, the option came up that assessments can take place whenever students think they’ve achieved some certain quality levels for (parts of) their work. This idea was also triggered by the usage of learning outcomes and the fact that some of the students already created some products, e.g. at their professional work, which could serve as evidence that they already achieved the learning outcomes. So we interchanged the snapshot moments with student-defined milestones. Fig. 3. Planning of the semester, the blue post its on the left show the introduction of an assignment, the yellow post its on the right determine the final deadline, students were encouraged to grade themselves between these moments as often as they want In consequence, grading is not limited to only looking back at some finished work, but was also applied to partial work results whenever these reached some pre-specified quality levels as described in the rubrics. The grades are combined with feedback which is still relevant to the work as it can be used for improvement until the final deadline. This way, grades and the associated feedback are used for looking back and forward, making them a more valuable combination. On Using Hybrid Pedagogy as Guideline for Improving Assessment Design 37 – Solution: Grading Queue While synchronous direct feedback is valuable, it is often hard to realize in sufficient quantity. The idea was to make asynchronuous feedback as syn- chronuous as possible, hereby combining the advantages of both. In the ex- ample in Figure 4, this means that feedback does not have to be either syn- chronous or asynchronous, but that it also could be asynchronous in such a timely manner that it feels more synchronous (and also has the benefits of synchronous feedback). Fig. 4. Explicit exploration of unused solution spaces (in green) which are neither clearly synchronous or asynchronous The resulting practice is a Grading Queue5 . After students performed a Self-Grading, including a Grade Motivation, they had to add an issue to the queue (in our case a Kanban board), hereby letting the teachers know that they had performed a self-grading and are now waiting for feedback (see Figure 5 for an example). The teachers then picked the longest waiting requests, examined them and returned them with corresponding feedback. The effect was that most grading requests were handled not longer than 1 or 2 days after they were handed in. This way the feedback that came with the handling was given close to the moment of finishing that part of the work. In consequence this feedback was given during learning and students could still act on it, some of the characteristics of effective feedback [2]. This solution therefore addressed the issue of Feedback Timing. – Improvement Encouragement Another important aspect for adding value to the feedback is to have students react on it. The above described solutions allowed such reactions. Additional elements of the applied assessment strategy comprised these practices: • Act On Feedback [20] - Applied for closing the feedback loop by making sure that students have time to act on the feedback they have been given. 5 Grading Queue (aka Grading Request Kanban) [10]: Provide an easily ac- cessible overview of all open grading requests, sorted by waiting time. Handle the grading requests in a structured, timely, and transparent manner. 38 C. Köppe and R. Middelkoop Fig. 5. Example of Grading Queue (adapted from [11]) • Grade It Again, Sam [1] - Core of this practice is to permit your students to change and re-submit an assignment for re-evaluation and re-grading, after you have graded it and provided feedback.- • Go For Gold [10] - Encourage the students to continue improving their work, even—or especially—when they already acquired a sufficient grade for it. • Repair It Yourself [12] - Let students correct their wrong or incorrect solutions, so that they understand better how to do it right. 4 Conclusion In this work we described how we applied Hybrid Pedagogy as guideline for ad- dressing some remaining phenomena of standard assessment strategies. First ex- periences show that these practices indeed help with addressing the phenomena. However, further research is needed to more thoroughly evaluate their effective- ness. We believe that explicitly using hybridity as guideline during educational design can help to widen the solution space and to identify potential alternative practices that help to address existing challenges in educational strategies. Ex- ploring this approach will also be part of future work. Further research is needed to determine if and to what extent the approach of using Hybrid Pedagogy as design guideline is also applicable in other educational domains. References 1. Bergin, J., Eckstein, J., Völter, M., Sipos, M., Wallingford, E., Marquardt, K., Chandler, J., Sharp, H., Manns, M.L. (eds.): Pedagogical Patterns: Advice for Educators. Joseph Bergin Software Tools, New York, NY, USA (2012) On Using Hybrid Pedagogy as Guideline for Improving Assessment Design 39 2. Chappuis, J.: Seven Strategies of Assessment for Learning. Pearson College Div (2014) 3. Crowell, T.L.: Student Self Grading: Perception vs. Real- ity. American Journal of Educational Research 3(4), 450– 455 (2015). https://doi.org/10.12691/EDUCATION-3-4-10, http://pubs.sciepub.com/education/3/4/10/ 4. Dow, S., Kulkarni, A., Klemmer, S., Hartmann, B.: Shepherding the crowd yields better work. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW. pp. 1013–1022 (2012). https://doi.org/10.1145/2145204.2145355 5. Edwards, N.M.: Student Self-Grading in Social Statistics. College Teach- ing 55(2), 72–76 (apr 2007). https://doi.org/10.3200/CTCH.55.2.72-76, http://www.tandfonline.com/doi/abs/10.3200/CTCH.55.2.72-76 6. Heames, J.T., Service, R.W.: Dichotomies in Teaching, Application, and Ethics. Journal of Education for Business 79(2), 118–122 (nov 2003). https://doi.org/10.1080/08832320309599099 7. Hilli, C., Nørgård, R.T., Aaen, J.H.: Designing Hybrid Learning Spaces in Higher Education. Dansk Universitetspædagogisk Tidsskrift 15(27), 66–82 (2019), https://tidsskrift.dk/dut/article/view/112644 8. Kohls, C.: Hybrid learning spaces. In: Proceedings of the VikingPLoP 2017 Con- ference on Pattern Languages of Program - VikingPLoP. pp. 1–12. ACM Press, New York, New York, USA (2017). https://doi.org/10.1145/3158491.3158505, http://dl.acm.org/citation.cfm?doid=3158491.3158505 9. Kohls, C., Nørgård, R.T., Warburton, S.: Sharing is Caring. In: Proceedings of the 22nd European Conference on Pattern Lan- guages of Programs. pp. 34:1—-34:6. EuroPLoP ’17, ACM, New York, NY, USA (2017). https://doi.org/10.1145/3147704.3147741, http://doi.acm.org/10.1145/3147704.3147741 10. Köppe, C., Manns, M.L., Middelkoop, R.: Educational Design Patterns for Student-Centered Assessments. In: Preprints of the 26th Conference on Pattern Languages of Programs, PLoP’19. Ottawa, Canada (2019) 11. Köppe, C., Manns, M.L., Middelkoop, R.: The Pattern Language of Incremental Grading. In: Proceedings of the 25th Conference on Pattern Languages of Pro- grams, PLoP’18. Portland, OR, USA (2019) 12. Köppe, C., Niels, R., Holwerda, R., Tijsma, L., Van Diepen, N., Van Turn- hout, K., Bakker, R.: Flipped classroom patterns: designing valuable in-class meetings. In: Proceedings of the 20th European Conference on Pattern Lan- guages of Programs - EuroPLoP ’15. vol. 08-12-July, pp. 1–17. ACM Press, New York, New York, USA (2015). https://doi.org/10.1145/2855321.2855348, http://dl.acm.org/citation.cfm?doid=2855321.2855348 13. Köppe, C., Nørgård, R.T., Pedersen, A.Y.: Towards a pattern language for hybrid education. In: Proceedings of the VikingPLoP 2017 Conference on Pattern Languages of Program - VikingPLoP. pp. 1–17. ACM Press, New York, New York, USA (2017). https://doi.org/10.1145/3158491.3158504, http://dl.acm.org/citation.cfm?doid=3158491.3158504 14. Lyons, L.L., Freitag, P.K., Hewson, P.W.: Dichotomy in thinking, dilemma in actions: Researcher and teacher perspectives on a chemistry teaching prac- tice. Journal of Research in Science Teaching 34(3), 239–254 (mar 1997). https://doi.org/10.1002/(SICI)1098-2736(199703)34:3¡239::AID-TEA3¿3.0.CO;2- T 40 C. Köppe and R. Middelkoop 15. Rorabaugh, P., Stommel, J.: Hybridity, pt. 3: What Does Hybrid Pedagogy Do? - Hybrid Pedagogy (2012), http://www.digitalpedagogylab.com/hybridped/hybridity-pt-3-what-does-hybrid- pedagogy-do/ 16. Schuwirth, L.W., Van Der Vleuten, C.P.: Programmatic assessment: From assessment of learning to assessment for learning. Medical Teacher 33(6), 478–485 (jun 2011). https://doi.org/10.3109/0142159X.2011.565828, http://www.tandfonline.com/doi/full/10.3109/0142159X.2011.565828 17. Sierpinska, A.: On Practical and Theoretical Thinking and Other False Dichotomies in Mathematics Education. In: Hoffmann, M., Lenhard, J., Seeger, F. (eds.) Ac- tivity and Sign, pp. 117–135. Springer-Verlag, New York (2005) 18. Stommel, J.: Hybridity part 2, what is hybrid pedagogy (2012), https://hybridpedagogy.org/hybridity-pt-2-what-is-hybrid-pedagogy/ 19. Strong, B., Davis, M., Hawks, V.: Self-Grading In Large Gen- eral Education Classes: A Case Study. College Teaching 52(2), 52–57 (apr 2004). https://doi.org/10.3200/CTCH.52.2.52-57, http://www.tandfonline.com/doi/abs/10.3200/CTCH.52.2.52-57 20. Warburton, S., Bergin, J., Kohls, C., Köppe, C., Mor, Y.: Dialogical assessment patterns for learning from others. In: Proceedings of the 10th Travelling Confer- ence on Pattern Languages of Programs - VikingPLoP ’16. pp. 1–14. ACM Press, New York, New York, USA (2016). https://doi.org/10.1145/3022636.3022651, http://dl.acm.org/citation.cfm?doid=3022636.3022651 21. Warren, B., Ogonowski, M., Pothier, S.: “Everyday” and “Scientific”: Rethinking Dichotomies in Modes of Thinking in Science Learning. In: Nemirovsky, R., Rosebery, A.S., Solomon, J., Warren, B. (eds.) Every- day Matters in Science and Mathematics, pp. 129–158. Routledge, New York, NY, USA (dec 2004). https://doi.org/10.4324/9781410611666-11, https://www.taylorfrancis.com/books/e/9781410611666/chapters/10.4324/9781410611666- 11