Learning technology-enabled (meta)-cognitive scaffolding to support learning aspects of written argumentation

Abstract

This paper1 reports on an AI-informed and NLP-based work in progress. It shares the technology, educational and cognitive approaches for enabling science students to engage with automated (AI) personalized (meta)-cognitive scaffolding to learn aspects of written scientific argumentation. We briefly report on the features and functionalities of MindWare technology and preliminary and brief results of a small-scale pilot to gauge the impact of technology-mediated scaffolding on students' learning of how to argue (in written form). CCS Concepts •Computing computing methodologies

Keywords

• Cognitive Computing
• Learning Technologies
• Argumentation
• Natural Language Processing
• Science Education

-

Research in the area of metacognition and scaffolding for learning emphasizes the need to provide adequate, sufficient and timely external support to enable the enacting of the students’ metacognitive processes [ 1 ]; [ 14 ]; [ 29 ]. The past few years have seen a surge in research related to technology-mediated assessment of written output by foreign language learners and learning analytics-informed reflective writing [ 36 ]; [ 15 ]; [ 16 ]; [ 10 ]; [ 3 ]; [ 34 ]. The use scaffolded automated feedback to support metacognitive learning of written argumentation is, however; an underexplored domain. This work is a contribution to this domain, with a specific focus on application in the context of science undergraduate education.

Most commonly, scientists learn to develop a written scientific argument by mimicking their supervisor, peers and scholarly papers in their discipline. It is increasingly recognized that for Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the first author, Noureddine Elouazizi, reachable at [e-mail]. © 2018 Copyright held by the owner/author(s). students to effectively develop argumentation skills, they must explicitly learn how to argue and reason [ 22 ]; [ 18 ]. This is because to develop or critique an argument, students need to explicitly learn how to advance claims, take stances, justify ideas they hold, and be challenged about the ways they construct their arguments [ 19 ]; [ 46 ]. Hence, to develop their argumentation skills, students need to gain an understanding of the meta-linguistic and meta-cognitive features of argumentation. Explicit teaching of written argumentation in science might, however, seem an overwhelming challenge as it requires both content knowledge and knowledge about how to structure a written argument.

Cognisant of these challenges, we developed a learning technology, dubbed MindWare, to provide iterative formative feedback on written argumentation as a support for instructors and students at our university. In this paper, we: (a) provide a brief overview of the pedagogical, computational and cognitive approaches that the learning technology is based on and (b) briefly report on the preliminary results of a small-scale pilot of the tool. 2

Personalized Learning Environments and

Scaffolding

Personalized learning is a pedagogical approach that puts the learner, their progress, and their learning at the heart of the pedagogical experience [ 8 ]. This approach allows students to proceed at their own learning pace, and can be supported by a combination of human and automated processes. The use of automated processes requires technologies that give students control, actionable information, and feedback, and allows them to take responsibility for their own learning. When used in a course, learning technologies that support personalized learning are expected to monitor individual students’ progress at a micro-level, and supply automatic feedback [ 8 ].

The pedagogy of learning to argue and arguing to learn [ 36 ]; [ 10 ], suggests that personalized learning environments need to cater to both the cognitive and the meta-cognitive aspects of learning to argue. There is reason to believe that such an approach lends itself to pedagogically sound scaffolding [ 48 ]. We define scaffolding as providing need-based assistance to students. Effective scaffolding requires that the why, the what and the how of the scaffolding is related to the expected assessment methods and learning outcomes [ 2 ]. In our case, this included explicit scaffolding of the usages of the argumentation voices of hedging, stancing, and logical connectors in written argumentation, as produced by several drafts of essays written by students as parts of their formative assessment in a First-Year Seminar (SCIE 113) course where students learn to construct and deconstruct (scientific) arguments [ 5 ]. 3

The metacognition of argumentation

There are at least three approaches to argumentation: (a) argumentation as a logical product (b) argumentation as a rhetorical process and (c) argumentation as an epistemic tool [ 6 ]. We adopt the perspectives in (b) and (c). We assume that written language is the direct cognitive by-product that externalizes how students build arguments supported by evidence. We define argumentation as a complex meta-cognitive act produced by a writer, and evaluated by a reader. Assuming that language is core to learning and that thought and language are inseparable [38], examining students’ argumentation offers opportunities for gaining insights into how students engage in scientific reasoning.

Drawing on the reasoning above, we assume that the argumentation voice exhibited in student essays is a direct window to students’ reasoning. This reasoning is externalized, in written form, through the way students formulate a claim (premise/thesis statement), how they elaborate on that premise, how they hedge, take a stance, and the logical connections they adopt in their essays. We further assume that in the process of taking the argument from an initial draft to writing the final product that will be submitted for summative assessment, the students would have engaged in many meta-cognitive aspects related to written argumentation.

To enable the students to engage in the cognitive and the metacognitive aspects of learning to argue (in written form), there are a set of pedagogical requirements that need to be met by the scaffolding process-es, enabled through learning technology. These requirements which we derive from the literature of metacognition for learning [ 12 ]; [ 47 ]; [ 49 ]; [ 7 ] include following: (i) learning technology functionalities that help students monitor their own thinking process, (ii) internalize self-monitoring techniques, and (iii) develop higher order cognitive processing techniques (through asking higher order questions) [ 12 ]; [ 47 ]. 4

Technology-enabled Scaffolding of Written

Argumentation Voice

The past decades have witnessed an increase in studies that investigate students’ argumentation skills in educational contexts and how these might be enhanced [38]; [ 27 ]; [41]; [ 28 ]; [ 42 ]. As Scheuer et al. [ 33 ] observes, (automated) support for learning argumentation is missing from most formal courses. To address this gap, many technology and learning scientists embarked on the exploration of different technology designs to support aspects of representing argumentation to simulate and diagnose reasoning [ 42 ]; [40]; [ 44 ]; [ 10 ]; [ 43 ], and to support conversational argumentation [ 35 ]; [39]. This has led to the development of a number of technologies that are designed to improve learning through diagramming argumentation [ 19 ]; [ 43 ], and to enable scaffolding and argumentative communication through visualization [ 44 ]. In parallel, with this work on how to (re)present an argument, the last two decades have also witnessed the emergence of advanced techniques for mining different aspects of argumentation from text. This includes the automatic classification of argument components [ 34 ]; [ 10 ]; [ 35 ], the identification of argumentation structures [ 45 ], and the separation of argumentative from non-argumentative text units [ 14 ]; [ 42 ].

We build on these general approaches to mining and representing aspects of argumentation, and on the specific insights that relate to how computational argumentation methods can be used to analyze essays for pedagogical purposes. In this respect, the general computational argumentation method that we have adopted relates to that of Persing and Ng [ 27 ], Song et al. [ 34 ], Walton et al., [ 42 ] and Klebanov et al [ 28 ]. We share with these scholars the goals of extracting argument structures from essays by recognizing (structural) argument components and jointly modeling their types and relations between them.

MindWare (our software), a beta version at this point, has two clusters of functionalities one for the students and one for the instructors. The instructors in our educational context are scientists and do not have any training in language sciences and argumentation analysis per se. The usage of MindWare is intended to complement the feedback provided by the instructors, such that they can focus their feedback on content, such as the quality of the evidence provided in support of the argument. The software is designed to provide feedback on students’ written argumentation voice, focusing specifically on the usages of hedging, stancing, logical connectors and coherence. Students submit a number of drafts (the number to be set by the instructor) and the performance of the students is visualized in a set of color coded gauges, heatmaps and graphs that provide students with feedback on the aspects of their argumentation that require improvements (see Figure 1).

The dashboard also provides feedback on students’ performance on aspects of their argumentation across different drafts of their essays is also displayed. (see Figure 1). Instructors can use the software to view the submissions and the performance of a particular student, and/or a of group of students, and they can see which aspects that students commonly struggle with in terms of mastering the components of the argumentation voice, and as such can design pedagogical intervention accordingly. Instructors are able to do this through having access to a dashboard that provides the instructors with an overview of different aspects of argumentation in students’ essays. For example, in Figure 2, the heat map provides an overview of the areas of argumentation that the class is struggling with. The heat map with areas colored in yellow and red indicates aspects of written argumentation that some of the students in that course section are struggling with, and which requires the pedagogical attention of the instructor. In terms of the computational model, MindWare is equipped with Natural Language Processing and Machine Learning modules that analyze and weigh the usage of the components of an argumentation voice, viz., the balanced use of stancing, hedging, logical connections, and coherence. For example, MindWare can identify and evaluate the degree of stancing in an essay [ 10 ]. That is, whether the writer is arguing for a specific stance. In contrast to describing, stancing is used to express one’s position. When writers take a stance, they not only express factual information but they also indicate their commitment with regard to what they said/wrote. The presence (or the lack thereof) of the components of the argumentation voices of stancing, hedging and logical connections can shape the reader’s opinion of the writer and of their argument in such a way that succeeds (or fails) to convey an adequate epistemic vigilance on the part of the writer. 5

Gauging the Impact

In this study, we piloted MindWare with the aim of supporting the metacognitive processes that underlie learning aspects of written argumentation in the context of a first-year science course. Part of our scaffolding strategies were planned in advance and focused on enabling and supporting the learning of the aspects of written argumentation, aspects that are crucial for establishing an argumentation voice in an essay as they are inherent in the exercise of epistemic vigilance within a written text [ 6 ]. This includes the (balanced) use of hedging, stancing, logical connections and coherence as indispensable components of an argumentation voice.

The AI-based machines in MindWare weigh the usage of these features in an essay and provide feedback (in visual and numerical form) to the learner. Other parts of the scaffolding in MindWare are provided dynamically, based on the response of the student, and such scaffolding is supported by an automatic feedback. An overview of the metacognitive scaffolding strategies we employed in MindWare is provided in table 1.

To gauge the impact of MindWare, in particular its ability to enable metacognitive scaffolding and support the use of argumentation voice, we conducted a small-scale pilot in a firstyear science course. Our pilot was run in two course sections of the same course. Each section had 25 volunteering students, and with students having the option to pull out of the study at any time when/if they want. Data collection was carried out in three stages and data of students who did not complete all the three stages was discarded.

In stage one, students responded to a pre-task survey, gauging their familiarity with the investigated concepts (hedging, stancing and logical connections), and the confidence level in using such components. Only after completing stage one, students were granted access to MindWare. In this stage, they were invited by the course instructors to submit a maximum of five drafts of their written essays, and explore the software, including receiving feedback before submitting the final version to the instructor for final assessment and grading. In this process, students were granted access to interact with an artificial agent to ask questions about different aspects of written argumentation and get automated feedback. In this stage of the pilot, 26 out of 50 students worked consistently in MindWare environment. This stage lasted for two weeks. After submitting the final version of their essay to the instructors, in stage three, students were asked to respond to a set of survey questions to reflect on their learner experience and specifically their perceptions about their own performance regarding the usage of the components of the argumentation voices in their written scientific essays. Of the entire cohort of 56 students, 54 participated in stage 1, 26 participated in stage 2 and 19 responded to the post-task survey.

On a scale of 1 to 10, students were asked to rate their familiarity with the indispensable components of the argumentation voices of hedging, stancing and logical connections in an essay. The left part in Figure 3 provides an overview of the pre-task survey responses. In the pre-task survey responses, only 15% of the students indicated that they are familiar to very familiar with the components of the argumentation voice of hedging, stancing and logical connection. After two weeks scaffolding through the use of MindWare, 51% of the students reported that they were very familiar with how to use the components of the argumentation voice in written essay.

Likewise, we observed that the confidence of the students in using the components of the argumentation voices in their essays increased. In the pre-task survey, 17.33% of the students reported that they were confident to very confident in using the components of the argumentation voice in their essays. Compared to the pre-task survey, in the post-task survey, 53% of the students reported that they become very confident in using the components of the argumentation voice in their written essays, after two weeks of technology-enabled scaffolding in the post-task survey.

Overall, it seems that students’ familiarity with the components of the argumentation voice in their written essays and their confidence in using such components increased after using the meta-cognitive scaffolding strategies, as enabled through MindWare. 6

Conclusion

As indicative as this early stages data overview may seem, it is neither conclusive, nor comprehensive. It is necessary to carry an extensive analysis of how the specific components of the argumentation voice have evolved or devolved across the drafts of the essays the students have submitted to MindWare. Moreover, we need to analyze the significance, if any, of the changes in the grades of the students within the experimental group, and compare the results to those of a control group of students, a course section that did not participate in the pilot study, using MindWare to scaffold aspects of written argumentation. In future work, we plan to carry out an extensive analysis to address and report on these pending aspects of our research into the interplay between the use of AI and NLP-informed learning technology, (meta)cognitive scaffolding, and learning of written scientific argumentation.

Acknowledgments

We gratefully acknowledge the financial support for this project, provided by: (a) UBC’s TLEF innovation grant (project grant: 22G36907) and (b) by the Science Centre for Learning and Teaching (Skylight) at the UBC’s Faculty of Science. We are grateful also to Scie113 students and instructors for participating in this research.

References

[1] Azevedo , R. , Guthrie , J. T. , & Seibert , D. 2004 . The role of self-regulated learning in fostering students' conceptual understanding of complex systems with hypermedia . Journal of Educational Computing Research , 30 , 87 - 111 .

[2] Azevedo , R. , & Hadwin , A. F. 2005 . Scaffolding self-regulated learning and metacognition-implications for the design of computer-based scaffolds . Instructional Science , 33 ( 5-6 ), 367 - 379 .

[3] Buckingham Shum et al. 2017 . Towards reflective writing analytics: Rationale, methodology, and preliminary results . Journal of Learning Analytics , 4 ( 1 ), 58 - 84 .

[4] Burstein , J. 2003 . The E-rater® scoring engine: Automated essay scoring with natural language processing . Lawrence Erlbaum Associates Publishers.

[5] Birol , Gülnur , et al. 2013 . Research and Teaching: Impact of a First-Year Seminar in Science on Student Writing and Argumentation . In Journal of College Science Teaching , 043 ( 01 ).

[6] Bermejo-Luque , L. 2011 . Giving Reasons: A Linguistic-Pragmatic Approach to Argumentation Theory . Argumentation Library , vol. 20 . Dordrecht: Springer.

[7] Brown , A. L. 1987 . Metacognition, executive control, self-regulation, and other more mysterious mechanisms . Hillsdale, NJ: Lawrence Erlbaum.

[8] Conati C. and Maclaren H. 2009 . Empirically Building and Evaluating a Probabilistic Model of User Affect. User Modeling and User-Adapted Interaction , 19 , 267 - 303

[9] de Groot , R. et al. 2007 . Computer supported moderation of e-discussions: The ARGUNAUT approach . In C. Chinn, G. Erkens & S. Puntambekar (Eds.), Mice, minds, and society-The computer supported collaborative learning ( CSCL) Conference 2007 , (pp. 165 - 167 ). International Society of the Learning Sciences.

[10] Elouazizi , Noureddine , et al. 2017 . Automated analysis of aspects of written argumentation . In LAK '17 Proceedings of the Seventh International Learning Analytics & Knowledge Conference . (pp. 606 - 607 ). The Association for Computing Machinery . DOI: http://dx.doi.org/10.1145/3027385.3029484.

[11] Foltz , P.W., S. Gilliam , and S. Kendall . 2000 . Supporting content-based feedback in online writing evaluation with LSA. Interactive Learning Environments , vol. 8 ( 2 ), pp. 111 - 129 .

[12] Flavell , J. H. 1979 . Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry . American Psychologist , 34 ( 10 ), 906 - 911 .

[13] Florou , Eirini et al. 2013 . Argument extraction for supporting public policy formulation . In Proceedings of the 7th Workshop on Language Technology for Cultural Heritage , Social Sciences, and Humanities , pages 49 - 54 , Sofia, Bulgaria, August. Association for Computational Linguistics.

[14] Ge , X. , & Land , S. M. 2004 . A conceptual framework for scaffolding illstructured problem-solving processes using question prompts and peer interactions . Educational Technology Research and Development , 52 ( 2 ), 5 - 22 .

[15] Gibson , A. , & Kitto , K. 2015 . Analysing reflective text for learning analytics: An approach using anomaly recontextualisation . Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15) , 16 - 20 March 2015, Poughkeepsie, NY , USA (pp. 275 - 279 ). New York: ACM.

[16] Gibson , A. , Kitto , K. , & Bruza , P. 2016 . Towards the discovery of learner metacognition from reflective writing . Journal of Learning Analytics , 3 ( 2 ), 22 - 36 .

[17] Herrenkohl , L. and Guerra , M. 1998 . Participant structures, scientific discourse, and student engagement in fourth grade . Cognition and Instruction , 16 ( 4 ), 431 - 473 .

[18] Jermann , Patrick and Pierre Dillenbourg . 2003 . Elaborating new arguments through a CSCL scenario . In J. Andriessen, M. Baker & D. Suthers . Arguing to Learn: Confronting Cognitions in Computer Supported Collaborative Learning environments . CSCL Series , vol. 1 . Amsterdam: Kluwer.

[19] Klebanov , B. , B. , et al. 2016 . Argumentation: Content, structure, and relationship with essay quality . In Proceedings of the Third Workshop on Argument Mining (ArgMining 2016 ), pages 70 - 75 . Association for Computational Linguistics.

[20] Kakkonen , T. , Myller , N. , and Sutinen , E. 2006 . Applying Part-Of-Speech Enhanced LSA to Automatic Essay Grading . In Proceedings of the 4th IEEE International Conference on Information Technology: Research and Education (ITRE 2006 ).

[21] Landauer , T. K. , et al. 1997 . How well can passage meaning be derived without using word order? A comparison of Latent Semantic Analysis and humans . In M. G. Shafto & P. Langley (Eds.), Proceedings of the 19th annual meeting of the Cognitive Science Society (pp. 412 - 417 ). Mawhwah, NJ: Erlbaum.

[22] Linn , M. C. , Bell , P. , & Hsi , S. 1998 . Using the Internet to enhance student understanding of science: The knowledge integration environment . Interactive Learning Environments , 6 ( 1-2 ), 4 - 38 .

[23] Moens , M-F , et al. 2007 . Automatic detection of arguments in legal texts . In ICAIL '07: Proceedings of the 11th International Conference on Artificial Intelligence and Law , pages 225 - 230 , New York, NY, USA, 2007 . ACM Press.

[24] Mochales , R. and M.-F. Moens . 2008 . Study on the Structure of Argumentation in Case Law . In Legal Knowledge and Information Systems. Jurix 2008 . IOS Press.

[25] McAlister , S. , Ravenscroft , A. , & Scanlon , E. 2004 . Combining interaction and context design to support collaborative argumentation using a tool for synchronous CMC . Journal of Computer Assisted Learning: Special Issue: Developing Dialogue for Learning, 20 ( 3 ), 194 - 204 .

[26] Mayer , R. E. 1996 . Learning strategies for making sense out of expository text: the SOI model for guiding three cognitive processes in knowledge construction . Educational Psychology Review , 8 , 357 - 371 .

[27] Persing , I. and Vincent Ng . 2015 . Modeling argument strength in student essays . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 543 - 552 . Association for Computational Linguistics.

[28] Ranney , M. , and Schank , P. 1998 . Toward an integration of the social and the scientific: Observing, modeling, and promoting the explanatory coherence of reasoning . In S. Read & L. Miller (Eds.), Connectionist models of social reasoning and behavior (pp. 245 - 274 ). Mahwah, NJ: Lawrence Erlbaum.

[29] Roll , I. , Holmes , N. G. , Day , J. , & Bonn , D. 2012 . Evaluating metacognitive scaffolding in guided invention activities . Instructional Science , 40 , 691 - 710 .

[30] Suthers , D. D. 2001 . Architectures for computer supported collaborative learning . In Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT 2001 ) (pp . 25 - 28 ), Madison.

[31] Suthers , D. D. , et al. 2008 . Beyond threaded discussion: Representational guidance in asynchronous collaborative learning environments . Computers & Education , 50 ( 4 ), 1103 - 1127 .

[32] Schwarz , B. B. , and Glassner , A. 2007 . The role of floor control and of ontology in argumentative activities with discussion-based tools . International Journal of Computer-Supported Collaborative Learning (ijCSCL) , 2 ( 4 ), 449 - 478 .

[33] Scheuer , O. , Loll , F. , Pinkwart , N. , & McLaren , B. M. 2010 . ComputerSupported Argumentation: A Review of the State-of-the- Art . International Journal of CSCL , 5 ( 1 ): 43 - 102 .

[34] Song , Yi , et al. 2014 . Applying argumentation schemes for essay scoring . In Proceedings of the First Workshop on Argumentation Mining , pages 69 - 78 . Association for Computational Linguistics.

[35] Sumsion , J. , & Fleet , A. 1996 . Reflection: Can we assess it? Should we assess it? Assessment & Evaluation in Higher Education , 21 ( 2 ), 121 - 130 .

[36] Teufel , S and M. Moens . 1999a. Discourse-level argumentation in scientific articles: human and automatic annotation . In Towards Standards and Tools for Discourse Tagging. ACL 1999 Workshop.

[37] Ullmann , T. D. , Wild , F. , & Scott , P. 2012 . Comparing automatically detected reflective texts with human judgements . Proceedings of the 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR-TEL ʼ12) , 18 September 2013 , Saarbrucken, Germany (pp. 101 - 116 ).

[42] Walton , D. , Chris Reed , and Fabrizio Macagno . 2008 . Argumentation schemes . New York, NY: Cambridge University Press.

[43] Woolf , B. P. , et al. 2005 . Critical thinking environments for science education . In C. K. Looi, G. McCalla , B. Bredeweg , & J. Breuker (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence and Education ( AI-ED 2005 ) (pp . 702 - 709 ). Amsterdam: IOS.

[44] Wiemer-Hastings , P. , and Zipitria , I. 2001 . Rules for Syntax, Vectors for Semantics . In Proceedings of the 23rd Annual Conference of the Cognitive Science Society , Mahwah, NJ. Erlbaum.

[45] Wyner , Adam , et al. 2010 . Semantic processing of legal texts . In Approaches to Text Mining Arguments from Legal Cases , pages 60 - 79 . SpringerVerlag, Berlin, Heidelberg.

[46] Wiebe , J., T. Wilson, R. Bruce , M. Bell , and M. Martin . 2004 . Learning Subjective Language . Computational Linguistics , 30 ( 3 ).

[47] Winne , P. H. 2011 . A cognitive and metacognitive analysis of self-regulated learning . In B. J. Zimmerman and D. H. Schunk (Eds.), Handbook of selfregulation of learning and performance (pp. 15 - 32 ). New York: Routledge.

[48] Wood , D. , Bruner , J. , & Ross , G. 1976 . The role of tutoring in problem solving . Journal of Child Psychology and Psychiatry , 17 , 89 - 100 .

[49] Zimmerman , B. J. , & Schunk , D. H . (Eds.). 2011 . Handbook of self-regulation of learning and performance . NY: Routledge.