Learnersourcing Towards a More Inclusive Academic Culture Carolyn D. Sealfon1,2 1 Department of Physics, University of Toronto, Canada 2 Ronin Institute, USA Abstract This brief position paper from a physics professor’s perspective invites the learnersourcing community to collaborate towards creating more inclusive assessments of student learning. Through illuminating diverse perspectives and examples of quality learning, we could broaden norms about how knowledge and abilities are valued and enable more vibrant and inclusive yet rigorous learning environments. 1. The case for higher-order skills To thrive in an increasingly technological and interconnected world, we need both analytical thinking skills and compassion. In this information age, it is even more valuable for learners to develop transferable, 21st century skills, such as creativity, systems thinking and responsibility, than to learn discipline-specific content [1]. For example, learner-centred science education offers a powerful opportunity to help people develop higher-order abilities such as designing experiments to investigate new phenomena, collecting and analyzing data, devising and testing creative explanations, evaluating reasoning and experimental results, and proposing and inves- tigating applications of ideas [2, 3]. (Naturally, people learn these abilities through disciplinary content. Knowledge of disciplinary content is necessary, but not sufficient.) Enabling more creative expressions and evaluations of rigorous learning can also make learning more intrinsically fun, e.g. [4]. While we can use strategies like gamification to motivate learners’ engagement [5], such mechanisms can tend to reward conformity rather than creativity. Through fostering an inclusive learning community, learner-centred approaches can harness the joy of collaborative creativity and a shared appreciation for qualities beyond providing correct answers (e.g. [6] and references therein). There is considerable research about how to design science learning experiences to help students develop higher-order skills, e.g. [7, 8, 9, 10, 11, 12, 13]. These approaches especially help under-represented students [14]. However, faculty are slow to adopt research-based instructional approaches [15, 16], and, especially at large scales, higher-order learning is challenging to assess. Could we leverage learnersourcing to facilitate and assess instructional approaches that foster higher-order skills? Learnersourcing: Student-generated Content @ Scale, June 01, 2022, New York City, NY, USA $ carolyn.sealfon@ronininstitute.org (C. D. Sealfon) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) 2. Assessments set priorities Higher education serves as a gateway to economic stability and success in many parts of the world. Yet perceptions of educational success often rely on an individualistic, colonial framework based on sorting students by ability, talent, or potential. Many historical and current ways of assessing knowledge and abilities disadvantage marginalized groups, some by design and some by tradition, e.g. [17, 18, 19, 20]. The role of instructors and educational systems to classify students sometimes interferes with the role of instructors and educational systems to educate students. To transform educational institutions into equitable systems will require rethinking how we conduct and use assessments. The learnersourcing community faces both an opportunity and a responsibility to contribute to creating more equitable next-generation assessments. Within educational institutions, how learning is assessed has tremendous implications for students, instructors, and the quality of learning. Grading is effectively an act of classifying students’ learning artifacts into categories interpreted as levels of performance. Grade-based classifications of students are then used to determine students’ access to opportunities. Further, priorities tend to be shaped by assessments. Grading schemes influence what students prioritize in their studying. How institutions and faculty gauge the success of their courses influences how instructors facilitate learning. Through measuring success via traditional assessments, institutions incentivize instructors to use traditional teaching methods that have been exclusionary. While there is increasing demand for accountability to results, we lack effective and affordable metrics, easy for all instructors to adopt, to equitably assess higher- order skills at large scales. Learnersourcing offers tremendous potential to facilitate meaningful learning and feedback at large scales, e.g. [5, 21]. Currently, assessments too often assume an objective definition of expertise or mastery, dictated entirely by instructors, as the assessment goal. In truly inclusive, learner-centred education, learners have agency to affect the standards by which their learning is assessed [22, 23], such as engaging students as partners [24] in creating rubrics. While we should naturally continue to harness instructors’ expertise, we could better integrate feedback from learners to challenge experts’ assumptions and implicit biases in evaluating the quality of learning artifacts. Learnersourcing systems offer the opportunity to integrate learners’ contributions in large classes and at large scales, including across multiple institutions. 3. Learnersourcing metrics of assessment Ideally, assessments are learning activities along a learning journey, part of the formative feedback loops between and among students and instructors, and well-aligned with learning goals that will help students in the long term. Yet assessments of learning often involve multiple- choice or closed-answer instruments which, while feasible to implement at large scales, can inadvertently emphasize low-level factual and procedural learning and disincentivize creativity and innovation for both instructors and students, e.g. [25]. Best practices to assess higher- order skills involve using a rubric to evaluate an open-ended assignment or project, which requires significant person-hours by people trained in applying the rubric. Often, rubrics are designed by instructors and experts with little student input. Grading approaches, rubrics, and algorithms can all reflect societal biases [26, 27, 28, 17]. We need new ways to assess higher-order transferable learning of complex skills at large scales to find valid, transparent approaches that are easy for all instructors to implement, inherently helpful to both students and instructors as formative assessments, and learner-centred through integrating student feedback in defining assessment categories. Learnersourcing seems uniquely positioned to address this challenge. We could leverage how learners classify learning artifacts to learn more about our expert classification schemes, turning the struggle of aligning learners’ peer evaluations with expert evaluations in learnersourcing systems [5] into an opportunity. For example, starting with a research-validated rubric developed by experts, we could look for patterns in how different students or different instructors evaluate work differently based on the rubric. Rather than striving for universally high inter-rater reliability, we could harness the information in the variability of the ratings. Through clustering patterns of peer ratings, we could potentially uncover other valid classifications and diversity in the ways that people interpret or express higher-order skills. This would help us identify hidden biases in the rubrics. We may be able to interpret these alternate clusters and use them to revise our rubrics or indicate the limitations of how disciplinary experts categorize and score student responses. There would be a feedback loop, as in all good education, between the learners and the learning standards. Ideally, we would also expand the ways that students share their insights beyond the written word [29] to enable multiple representations [30, 4], so students could include sketches, drawings, videos or music as part of their learning artifacts. Such an assessment tool, if easily applied at large scales, would also help probe and reward inclusive and learner-centred pedagogies. Acknowledgments Thanks to Ilya Musabirov and B. Gottheil for helpful comments, and Cultivating Ensembles for helpful ideas and experiences. References [1] OECD, The future of education and skills: Education 2030, Organisation for Economic Co-operation and Development (OECD) Education Working Papers (2018). [2] E. Etkina, A. Van Heuvelen, S. White-Brahmia, D. T. Brookes, M. Gentile, S. Murthy, D. Rosengrant, A. Warren, Scientific abilities and their assessment, Physical Review special topics-physics education research 2 (2006) 020103. [3] E. Etkina, D. T. Brookes, G. Planinsic, Investigative Science Learning Environment, Morgan & Claypool Publishers, 2019. [4] V. A. Segarra, B. Natalizio, C. V. Falkenberg, S. Pulford, R. M. Holmes, Steam: Using the arts to train well-rounded and creative scientists, Journal of microbiology & biology education 19 (2018) 19–1. [5] H. Khosravi, G. Demartini, S. Sadiq, D. Gasevic, Charting the design and analytics agenda of learnersourcing systems, in: LAK21: 11th International Learning Analytics and Knowledge Conference, 2021, pp. 32–42. [6] R. M. Holmes, M. M. Qureshi, Performing as scientists: An improvisational approach to student research and faculty collaboration., Bioscene: Journal of College Biology Teaching 33 (2006) 23–29. [7] S. Freeman, S. L. Eddy, M. McDonough, M. K. Smith, N. Okoroafor, H. Jordt, M. P. Wen- deroth, Active learning increases student performance in science, engineering, and mathe- matics, Proceedings of the National Academy of Sciences 111 (2014) 8410–8415. [8] E. Etkina, Millikan award lecture: Students of physics-listeners, observers, or collaborative participants in physics scientific practices?, 2015. [9] E. Etkina, G. Planinšič, Defining and developing "critical thinking" through devising and testing multiple explanations of the same phenomenon, The Physics Teacher 53 (2015) 432–437. [10] J. Handelsman, S. Miller, C. Pfund, Scientific Teaching, W. H. Freeman, 2006. [11] L. Dee Fink, Creating significant learning experiences: an integrated approach to designing college courses, Jossey-Bass higher and adult education series, Jossey-Bass, 2003. [12] J. Bransford, N. R. Council, How People Learn: Brain, Mind, Experience, and School - Expanded Edition, National Academy Press, 2000. [13] E. F. Redish, Teaching Physics with the Physics Suite, John Wiley & Sons Inc., Somerset, 2003. [14] E. J. Theobald, M. J. Hill, E. Tran, S. Agrawal, E. N. Arroyo, S. Behling, N. Chambwe, D. L. Cintrón, J. D. Cooper, G. Dunster, et al., Active learning narrows achievement gaps for underrepresented students in undergraduate science, technology, engineering, and math, Proceedings of the National Academy of Sciences 117 (2020) 6476–6483. [15] C. Henderson, M. H. Dancy, Barriers to the use of research-based instructional strategies: The influence of both individual and situational characteristics, Physical Review Special Topics-Physics Education Research 3 (2007) 020102. [16] M. E. Bathgate, O. R. Aragón, A. J. Cavanagh, J. K. Waterhouse, J. Frederick, M. J. Graham, Perceived supports and evidence-based teaching in college stem, International journal of STEM education 6 (2019) 11. [17] J. Randall, D. Slomp, M. Poe, M. E. Oliveri, Disrupting white supremacy in assessment: Toward a justice-oriented, antiracist validity framework, Educational Assessment (2022) 1–9. [18] J. L. Kincheloe, S. R. Steinberg, A. D. Gresson III, Measured lies: The bell curve examined, ERIC, 1997. [19] S. Geiser, Norm-referenced tests and race-blind admissions: The case for eliminating the sat and act at the university of california. research & occasional paper series: Cshe. 15.17., Center for Studies in Higher Education (2017). [20] J. V. Clark, Closing the achievement gap from an international perspective: Transforming STEM for effective education, Springer, 2013. [21] C. B. Hilton, M. B. Goldwater, D. Hancock, M. Clemson, A. Huang, G. Denyer, Scalable science education via online cooperative questioning, CBE—Life Sciences Education 21 (2022) ar4. [22] P. Freire, Pedagogy of the Oppressed: 50th Anniversary Edition, Bloomsbury Publishing, 2018. [23] b. hooks, Teaching to transgress: education as the practice of freedom, Routledge, 1994. [24] P. Felten, A. Cook-Sather, C. Bovill, Engaging students as partners in learning and teaching: A guide for faculty, John Wiley & Sons, 2014. [25] A. Crowe, C. Dirks, M. P. Wenderoth, Biology in bloom: implementing bloom’s taxonomy to enhance student learning in biology, CBE-Life Sciences Education 7 (2008) 368–381. [26] P. Steinke, P. Fitch, Minimizing bias when assessing student work, Research & Practice in Assessment 12 (2017) 87–95. [27] J. M. Malouff, E. B. Thorsteinsson, Bias in grading: A meta-analysis of experimental research findings, Australian Journal of Education 60 (2016) 245–256. [28] W. Guo, A. Caliskan, Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases, in: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021, pp. 122–133. [29] K. Jones, T. Okun, White supremacy culture, Dismantling racism: A workbook for social change (2001). [30] D. Rosengrant, E. Etkina, A. Van Heuvelen, An overview of recent research on multiple representations, in: AIP Conference proceedings, volume 883, American Institute of Physics, 2007, pp. 149–152.