=Paper=
{{Paper
|id=Vol-3046/paper1-imhe
|storemode=property
|title=TecCoBot: technology-aided support for self-regulated learning. Automatic feedback on writing tasks via Chatbot
|pdfUrl=https://ceur-ws.org/Vol-3046/imhe_2020_paper_1.pdf
|volume=Vol-3046
|authors=Norbert Pengel,Anne Martin,Tamar Arndt,Roy Meissner,Alexander Neumann,Peter de Lange,Heinz-Werner Wollersheim
}}
==TecCoBot: technology-aided support for self-regulated learning. Automatic feedback on writing tasks via Chatbot==
https://ceur-ws.org/Vol-3046/imhe_2020_paper_1.pdf
TecCoBot: Technology-aided support for
self-regulated learning ?
Automatic feedback on writing tasks via Chatbot
Norbert Pengel1[0000−0002−3263−6877] , Anne Martin1[0000−0001−8237−6770] , Roy
Meissner1[0000−0003−4193−8209] Tamar Arndt1[0000−0001−8170−3346] , Alexander
Tobias Neumann2[0000−0002−9210−5226] , Peter de Lange2[0000−0002−3494−7513] ,
and Heinz-Werner Wollersheim1[0000−0002−4690−5839]
1
Leipzig University, Faculty of Education, Leipzig, Germany,
[norbert.pengel,anne.martin,roy.meissner,tamar.arndt,
wollersheim]@uni-leipzig.de
2
RWTH Aachen University, Aachen, Germany
[neumann,lange]@dbis.rwth-aachen.de
Abstract. In addition to formal learning at universities, like in lec-
ture halls and seminar rooms, students are regularly confronted with
self-study activities. Instead of being left to their own devices, students
might benefit from a proper design of such activities, including pedagog-
ical interventions. Such designs can increase the degree of activity and
the contribution of self-study activities to the achievement of learning
outcomes.
Especially in times of a global pandemic, self-study activities are increas-
ingly executed at home, where students already use technology-enhanced
materials, processes, and digital platforms. Thus we pick up these build-
ing blocks and introduce TecCoBot within this paper. TecCoBot is not
only a chatbot, supporting students in reading texts by offering writing
assignments and providing automated feedback on these, but also imple-
ments a design for self-study activities, typically only offered to a few
students as face-to-face mentoring.
Keywords: mentoring · automated feedback · chatbot · self-study ac-
tivities · technology-aided learning · self-regulated learning · knowledge
graphs · design-based research · educational design research
1 Introduction
Self-study activities are an integral part of today’s workload in higher education
[1]. However, studies on the workload of students show that self-study activities
plays a subordinate role and that too little time is spent on independent learning
[2]. One approach that is widely recognised as effective in positively influencing
?
This work was supported by the German Federal Ministry of Education and Research
for the tech4comp project under grant No 16DHB2102
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution
4.0 International (CC BY 4.0).
2 Norbert Pengel et al.
students is mentoring [3]. However, individual face-to-face mentoring requires a
lot of time and effort. Especially in higher education, where a large number of
students is often supervised by only a small number of lecturers, it is impossible
to provide holistic mentoring to all students.
As part of the German research project tech4comp, which searches for de-
sign concepts to scale mentoring activities by utilising technology, we developed
a chatbot that is able to provide informative feedback for a large number of
students fast, as part of self-study activities. This is even more important, as
students currently cannot be at a university due the corona pandemic 2020 and
thus cannot make use of face-to-face mentoring.
Various models describe the knowledge and research process in (educational)
design research. Based on a review of existing models of design research, a generic
model was developed [7], which contains three core phases in a flexible and it-
erative structure: 1) analysis/exploration, 2) design/construction, and 3) evalu-
ation/reflection, which we currently use in our work.
In this paper we address the results of the first two phases, while our focus
is on a prototype for a technology-aided support system that provides students
with automated feedback in form of knowledge graphs on submitted writing
assignments via a chatbot.
2 Requirements and Related Work
Evaluating student needs In accordance with the multi-method approach,
we use both qualitative and quantitative methods. We used the “Teaching Anal-
ysis Poll (TAP)” as a qualitative method for exploring students needs in terms
of learning. Actually it is an instrument for course evaluation due to open-ended
questions. This method has a low degree of structuring and thus the greatest
possible exploratory character of result generation. In 2019 296 of 628 enrolled
students were surveyed in an educational science seminar in teacher education
at Leipzig University. The test persons were asked about the parts of teaching-
learning setting and activities of teaching staff of universities that either pro-
moted or hindered learning. The evaluation of the results was also carried out
with a qualitative method of empirical social research, the qualitative content
analysis in direct coding on the data material. A coding guideline prepared for
TAP3 served as a structuring aid for this. The structuring analysis was carried
out in several rounds in which the data material was compared with the indi-
vidual categories. These TAPs took place in all seminar groups in the above
mentioned period and offered qualitative intermediate feedback on the courses,
which was already condensed to statements that were accepted by a majority
in a Think-Pair-Share procedure. Altogether, a clear picture of the learning sit-
uation as perceived by the students could be derived. In addition to the large
number of texts to be read in the seminars, lack of focus on the content and its
structure, which is difficult to recognise, were mentioned as obstacles to learning.
3
guideline:https://https://epub.uni-regensburg.de/35604/
Supporting students in reading texts 3
Following this poll, a quantitative survey with a closed questionnaire is planned.
The aim of this survey is again to determine students’ need for support and to
test hypotheses derived from the qualitative data (in this respect it is a sequen-
tial research design). Accompanying this, however, the theoretical research that
also takes place in this phase is important for structuring the initial situation of
research process [6].
Mentoring It is understood as dyadic relationship between mentor and mentee
and can be an effective pedagogical measure to support learners in their learning
process [4]. A holistic view of mentoring includes cognitive, motivational and
emotional dimensions of this interaction, while ensuring learner autonomy and
self-determination.
If mentoring is to be designed to promote and motivate learning, the find-
ings of learning research are important, especially on self-regulated learning and
the related metacognitive regulation. The greatest effects of mentoring can be
expected when optimal conditions for effective learning processes are in place,
including frequent and high-quality feedback, practice and consolidation of what
has been learned. Currently, we focus on mentoring to support domain-specific
learning processes, especially knowledge.
In order to systematize the concept of mentoring, which is presented in the
research literature as being not very homogeneous, we refer to its functional
criteria [8], [9]. These include (among other things) support for the transfer
of scientific expertise, which aims to promote the students’ knowledge relevant
to the chosen subject area. The development of the prototype will focus on
knowledge of the students.
Generating feedback For mentoring to be accepted by students, the qual-
ity of feedback is important [10]. Findings from empirical educational research
clearly show that feedback can have a positive influence on learning and de-
velopment processes. In addition to personal factors of the sender and recipient
(e.g. expertise, self-efficacy, attribution patterns) and situational factors (cultural
conditions, binding character), most important is to design the transmission of
feedback in a way that promotes development of the student [11,13,12]. A dis-
tinction can be made between informative and controlling feedback [14]. The
latter impairs intrinsic motivation by building up pressure and thus, according
to the theory of self-determination [14], a condition for sustainable learning. Ac-
cording to a synthesis of meta-analyses of feedback in schools [15], informative
feedback is also considered more effective than praise or punishment. Important
for the effectiveness of feedback is that it offers learners clues or encouragement
and relates directly to learning goals.
Accordingly, we provide writing assignments and automated feedback for
accompanied self-study activities. The communication of these tasks and feed-
back are transmitted with a chatbot and is characterized by an understandable
everyday language without academic language in order to increase the possibil-
ity of reflection and also to stimulate cognitive, metacognitive and motivational
4 Norbert Pengel et al.
strategies. Through this intervention communicated by chatbot, students should
understand content and integrate it into their existing knowledge structure. By
activating metacognitive strategies, it should be possible for them to better mon-
itor their own learning process.
Las2Peer Las2Peer
Chatbot
Student Text Feedback
Student Document
Data
Cleaning
T-Mitocar
Text Model
}
Comparison
Expert Text Feedback
Fig. 1. Workflow of the mentoring prototype for text feedback
3 Approach and Implementation
Feedback with T-MITOCAR We are using the software T-MITOCAR to
provide individual feedback on writing assignments. T-MITOCAR constructs
re-representations of knowledge from prose text through a computer-linguistic
analysis, without the need to incorporate an external knowledge base [16,17].
The resulting knowledge graphs look similar to mind maps and may be modeled
from texts passages, like book chapters, or whole texts, like written by students.
In our case, the knowledge graph of a students text is compared with the graph
of reading assignment text (reference text) to generate concrete feedback. For
instance, this feedback can inform students about which key concepts are part of
their text model and also show how these are interlinked. It can also inform about
which concepts overlap with the model of the reference text, which are different,
and about how the concepts are linked in the assignment text. It should also
be noted that similarity to the assignment text depends on content and on the
writing task, so similarity is not always a goal parameter [18,19,20,21]. All this
information can aid reflection and text revision and, importantly, in combination
with active writing it can foster an in-depth study of the topic in self-regulated
learning [22].
Feedback provided by Chatbot Chatbots are conversational interfaces, al-
lowing humans to interact with software using natural language. Therefore chat-
Supporting students in reading texts 5
bots are known to be relatively easy to use and intuitive with a low entry barrier
and good accessibility [24], which makes them suitable for university context
with users on different levels of digital literacy. Classic mentoring typically takes
place within conversations between a mentor and mentee. The conversational na-
ture of chatbot interactions thus seems to be an easy choice to digitally emulate
mentoring processes. In contrast to most static graphical interfaces a chatbot
is inherently adaptive to some degree since it reacts to the users input in the
course of conversation, depending on the chatbot’s underlying knowledge base.
To a large extent the efficacy of mentoring lies in its adaptiveness towards the
mentee and his or her needs and competences [25]. For this a chatbot can fulfill
an important requirement for technology-aided mentoring although we still need
to determine the elements the chatbot needs to adapt to through the course of
conversation. Other elements known to be key factors for good mentoring are
the trust and respect within a mentoring relationship [26]. Reportedly chatbots
are potentially trusted and felt safe to share information with by their users
depending on implementation and design details of the chatbot, like for exam-
ple its human-likeness, humor or professional appearance [23]. This as well as
other specifics of the conversational design, like ethical factors, transparency,
the degree of human-likeness or opportunities for playfulness within the chatbot
conversation, still need to be subject to our future consideration.
Incorporating the results of the analysis and exploration phase, first ideas for
solutions were generated and implemented as part of a prototype4 This prototype
consists of three parts: the chatbot, a UI componentinserted into the LMS and
a service, creating the knowledge graphs and the comparisons of these graphs.
A first test took place in April 2020. In summer 2020 we will make the chatbot
available to students for the first time. In the following we describe one prototype
as result of the design and construction phase.
Implemented Prototype A first prototype was developed, tackling the re-
quirements and needs described above. It is presented to end users, e.g. students,
as a special view in a learning management system (LMS), providing a person-
alized chat interface with a chatbot. This chatbot was implemented and trained
using the open source natural language processing framework RASA5 .
The chatbot’s main purpose is to offer writing assignments to students,
gather the results and provide feedback. Therefore students are able to upload
their written texts within the chat. Such individual texts are proxied through a
las2peer network (see next paragraph) to a service generating the feedback. The
first step this service executes is a cleaning process for the issued texts, as the
input for further processing steps needs to be in a specific format. Subsequently
the service involves T-MITOCAR to do two things: 1) request a knowledge graph
that visualizes the concepts and concept connections of the students’ text, and 2)
request a graph comparison of the just created graph with a reference knowledge
graph. T-MITOCAR generated this reference graph from another written text,
4
https://gitlab.com/Tech4Comp
5
https://rasa.com
6 Norbert Pengel et al.
i.e. from a textbook, that contains domain knowledge about the same topic the
students’ text is about. To hand back proper feedback to students, a feedback
template is filled with specific comparison results, which is eventually converted
to a feedback PDF document. Depending on the mentoring scenario, either the
students knowledge graph, the reference knowledge graph or the textual com-
parison of the two knowledge graphs is transferred through the las2peer network
back to the chatbot, which finally provides the feedback document via the chat
interface to the individual student. This whole process is depicted in figure 1.
The former described frontend is a LMS-Plugin providing a single thread of
a RocketChat6 instance to end users, implemented as a web-component. Besides
the actual chat the plugin also handles authentication, authorization, and data
protection in accordance to the European GDPR rules. Regarding authentication
and authorization an OpenId Connect (OIDC) login barrier is used, which makes
sure that student credentials stay safe. Using OIDC furthermore allows students
to use one account to login to the LMS, the chat client, and the las2peer network.
The latter one is a decentralized open source environment for transferring and
storing user data without inheriting a central authority [28]. It connects several
nodes in a peer-to-peer fashion, protecting stored data and communication by
using asymmetric encryption.
4 Summary and Future work
We have sketched the prototype TecCoBot that is able to present automated
feedback on writing assignments provided by the software T-MITOCAR. Tec-
CoBot scales the potential of a special face-to-face mentoring scenario to a large
number of students, by enhancing already existing processes, materials and plat-
forms. This technology-aided support enables providing students a part of men-
toring on demand. Through the use of a software we are able to provide all
students individual feedback in time, even in courses with a large number of
students and few lecturers. Our approach allows students to receive feedback
where otherwise they either would have to do without or significant additional
human resources would have to be employed. In this way we make the support
for learning from texts in self-study activities available to as many students as
possible and whenever they want. The technologies we have used allow to con-
sider the requirements for privacy, security and informational self-determination.
In the near future we will conduct an application-oriented evaluation regarding
the implemented chatbot, as well as reflect about the described processes, which
will probably lead to new and adapted research questions. Furthermore we will
extend the current chatbot prototype with more parts of mentoring. We also
want to collect specific data of students to make technology-aided mentoring
more effective and generating knowledge about how it works. Finally we want
to identify design concepts to make parts of face-to-face mentoring scalable.
6
https://rocket.chat/
Supporting students in reading texts 7
References
1. European Union: ECTS Users’ Guide. Luxembourg: Publications Office of the Eu-
ropean Union (2015).
2. Metzger, C.: ZEITLast: Lehrzeit und Lernzeit. Studierbarkeit von BA-
/BScStudiengängen als Adaption von Lehrorganisation und Zeitmanagement unter
Berücksichtigung von Fächerkultur und neuen Technologien. In: S. Mandel, M.
Rutishauer and E. Seiler Schiedt (Hrsg.). Digitale Medien für Lehre und Forschung.
Waxmann, Münster (2010). (S. 287-302).
3. Eby, L.T., Dolan, E.L.: Mentoring in postsecondary education and organizational
settings. In: P.J. Hartung, M.L. Savickas, W.B. Walsh (Eds.). Handbook of Career
Intervention, Volume 2. Psychological Association, Washington, DC, USA (2015). (S.
383-395).
4. Ziegler, A.: Mentoring - Konzeptuelle Grundlagen und Wirksamkeitsanalyse. In: H.
Stoeger, A. Ziegler, D. Schimke und H. Stöger (Hrsg.): Mentoring - Theoretische
Hintergründe, empirische Befunde und praktische Anwendungen. 1. Auflage. Pabst
Science Publishers, Lengerich (2009), S. 7-30.
5. Koebis, L.: Digital-gestütztes Mentoring an Hochschulen. Working Paper tech4comp
(2020).
6. Reinmann, G.: Welchen Stellenwert hat die Entwicklung im Kontext von Design
Research? Wie wird Entwicklung zu einem wissenschaftlichen Akt. design-based re-
search, 63–78 (2014).
7. McKenney, S., Reeves, T.C.: Conducting educational design research. Routledge,
Taylor and Francis Group, London (2019).
8. Nora, A., Crisp, G.: Mentoring students: Conceptualizing and validating the multi-
dimensions of a support system. Journal of College Student Retention: Research,
Theory & Practice, (9), 337–356 (2007).
9. Gershenfeld, S.: A Review of Undergraduate Mentoring Programs. Review of Edu-
cational Research, 84(3), 365–391 (2012).
10. Höher, F.: Übersicht über den aktuellen Forschungsstand zu Mentoring. In: F.
Höher (Hrsg.). Vernetztes Lernen im Mentoring. Springer, Wiesbaden (2014).
11. Kluger, A., DeNisi, A.: he Effects of Feedback Interventions on Performance: A
Historical Review, a Meta-Analysis, and a Preliminary Feedback Intervention Theory.
Psychological Bulletin. 119. (1996).
12. Hattie, J.: Lernen sichtbar machen. Schneider-Verlag, Hohengehren (2013).
13. Hattie, J., Timperley, H.: The power of feedback. Review of educational research
77, 81–112 (2007).
14. Deci, E. L., & Ryan, R. M.: Intrinsic motivation and self-determination in human
behavior. New York: Plenum Press. (1985).
15. Hattie, J.: Influences on student learning. Inaugural professorial address, University
of Auckland, New Zealand (1999).
16. Pirnay-Dummer, P.: Knowledge Elicitation. In J. M. Spector (Ed.), The SAGE
Encyclopedia of Educational Technology (pp. 438-442). SAGE, Thousand Oaks, CA,
(2015).
17. Pirnay-Dummer, P.: Linguistic analysis tools. In C. A. MacArthur, S. Graham &
J. Fitzgerald (Eds.), Handbook of Writing Research (pp. 427-442). Guilford, Guilford
Publications (2015).
18. Aimeuer, E.: Application and Assessment of Cognitive-Dissonance Theory in the
Learning Process. Journal of Universal Computer Science 4(3), 180–187 (1998).
8 Norbert Pengel et al.
19. Helbig, H.: Knowledge representation and the semantics of natural language.
Springer, Berlin, New York (2006).
20. Jonassen, D.: On the Role of Concepts in Learning and Instructional Design. Ed-
ucational Technology Research and Development 54(2), 177–196 (2006).
21. van den Broek, P.: Using Texts in Science Education: Cognitive Processes and
Knowledge Representation. Science 328(5977), 453–456 (2010).
22. Tynjaelae, P., Mason, L., Lonka, K. (2001). Writing as a Learning Tool. Integrating
Theory and Practice. Dordrecht: Springer (Studies in Writing, 7).
23. Følstad, A., Nordheim, C. B., & Bjørkli, C. A.: What makes users trust a chatbot
for customer service? An exploratory interview study. Lecture Notes in Computer
Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes
in Bioinformatics), 11193 LNCS, 194–208. (2018).
24. Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., & McTear, M. (2019).
Usability testing of a healthcare chatbot: Can we use conventional methods to as-
sess conversational user interfaces? Proceedings of the 31st European Conference on
Cognitive Ergonomics: “ ‘Design for Cognition,’ ” (September), 207–214. (2019)
25. Vandewaetere, Mieke, and Geraldine Clarebout. "Advanced technologies for per-
sonalized learning, instruction, and performance." Handbook of research on edu-
cational communications and technology. Springer, New York, NY, 2014. 425-437.
(2014).
26. Alemdag, Ecenaz; Erdem, Mukaddes: Designing an e-mentoring program for novice
teachers in Turkey and investigating online interactions and program outcomes. In:
Mentoring & Tutoring: Partnership in Learning 24 (2), S. 123–150. (2017).
27. Følstad, A., Nordheim, C. B., & Bjørkli, C. A.: What makes users trust a chatbot
for customer service? An exploratory interview study. Lecture Notes in Computer
Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes
in Bioinformatics), 11193 LNCS, 194–208 (2018).
28. Klamma, Ralf and Renzel, Dominik and de Lange, Peter and Janßen, Holger:
las2peer - A Primer, August 2016, DOI: 10.13140/RG.2.2.31456.48645