Efective presentation skills are crucial in today's world. However, students often lack these skills due to limited training. To address this gap, technology-enhanced learning applications can play a central role. While prior research has primarily focused on supporting students with presentation rehearsals and slide design, the critical aspect of composing presentation messages has received limited attention. This PhD project aims to investigate how computer-based tools can support students in developing the essential skills required for composing efective presentation messages. To achieve this goal, we will utilize participatory design within the context of a design-based research approach.
However, many students lack communication skills [8]. One reason for this is that they did not receive suficient communication skills training [9, 8, 10]. Communication skills are still often overlooked, especially in engineering degrees [11]. And even if communication skills are trained, there are too few feedback opportunities for each of the students [9]. This is due to giving individual feedback to students being very time-consuming, especially in larger classes. Therefore, teachers do not have the resources to give a suficient amount of feedback to all of their students [9, 8]. sentation skills and that teachers can not provide more Portugal. nEvelop-O
message composition and message delivery mean? In a presentation, the goal is to transmit a message to the audience. So first, one needs to compose this message means going from an unstructured mass of knowledge, ideas and arguments but also experiences, stories and
So we know that students need more training in pre- sage composition and message delivery [12]. What does Proceedings of the Doctoral Consortium of the 18th European Conference on Technology Enhanced Learning, 4th September 2023, Aveiro, in order to be able to deliver it. Composing a message © 2023 Copyright © 2023 for this paper by its authors. Use permitted under Creative Commons emotions connected to a topic to having a clear plan of which points to make in which order, with which exam- which has also been criticized by users (e.g. [20, 17, 21]) ples, explanations, etc. in order to transmit the chosen or consisted of only one person ([19]). An extensive message. To illustrate this further, we can compare the review of the existing literature demonstrates that the process of preparing a presentation with the preparation most responsive audience has been implemented in [18]. of a theater play. In a theater play the message compo- There, the audience reacted to the start and end of a sition would be the main task of the play’s author. The presentation as well as to the volume of the presenter’s message delivery, in contrast, would then be figured out voice and to whether the presenter was turning their by the actors and the director when actually performing back towards the audience. This made the practising the play. While the message composition, both in theater situation more authentic. However, the audience was and in presentations, could be done without speaking still non-respondent to e.g. facial expressions, gestures, or moving, mastering the delivery can only be done by verbal aspects (e.g. appropriate language) and the actual actually performing the planned presentation or play. content of the presentation (e.g. jokes, logical structure This is because the message delivery is about how to use of the presentation, suficient explanations). non-verbal communication like body posture, gestures, As mentioned before, there is also research on supand facial expressions as well as pauses, tone, volume etc. porting presenters with designing better slides. In these in synergy with the message to deliver. examples, the feedback is mainly based on strict compuWhen coming back to presentation skills training, this tational measurements (e.g. [22, 19, 23]). These measuremeans that the message delivery part can only be evalu- ments provide feedback on the readability of slides (e.g. ated and given feedback upon by watching the presenter font family, font size, contrast) but also on the amount of perform the presentation. The message composition part, content (e.g. number of slides, number of words). Howin contrast, could also be evaluated and given feedback ever, the actual content of the slides is, to the best of our upon based on a written outline of the presentation [12]. knowledge, neglected in all of these approaches. As a
There are multiple existing approaches for computer- result, they also do not provide support for the message supported presentation skills training. Most of these sys- composition. tems focus on the message delivery and more specifically Generally, content-related features are quite rare. To the non-verbal communication. Some also provide sup- the best of our knowledge, the only approach aiming port for the design of slides (e.g. font size)[13]. Existing to provide content-related feedback is the approach of research for both kinds of approaches will be explained Trinh et al. [19]. They extract the most important keyin more detail in the following. words per presentation slide and check whether those
The approaches focusing on the message delivery keywords have been said by the presenter during the mainly give feedback on body language (e.g. posture, rehearsal. While this is somehow content-related, the gestures, gaze) and aural aspects of communication (e.g. approach, again, focuses on giving feedback related to speaking volume, pitch, filler sounds). They aim to sup- the message delivery and not to the message composiport students with practising presentations since practis- tion. The scarcity of features related to message compoing presentations generally leads to an increase in presen- sition is confounding given that the appropriateness of tation skills and a decrease in presentation anxiety [14]. e.g. non-verbal communication and therefore also the Furthermore, research has shown that the feedback given corresponding feedback is dependent on the content of in these applications leads to better performance in the the message [24]. aspects that the feedback was given on (e.g. maintaining Schneider et al. [25] developed a hybrid approach an open posture) [15]. However, technology support for where learners could self-reflect on whether their nondeliberate practice needs to enable authentic practice in verbal communication had meaning and fit what they order for the learners to reach the goal of mastering the wanted to communicate. However, while the approach trained skill [16]. In the context of presentation skills of Schneider et al. [25] helps with aligning verbal and training, authentic practice means that the communica- non-verbal communication and the approach of Trinh tion between the presenter and the audience needs to et al. [19] helps with not forgetting important topics, an happen in both directions. Therefore, in order to reach important piece of the puzzle is still missing. This missing the goal of authentic practice, the communication of the piece is the quality of the composed message. Aspects audience towards the presenter needs to be simulated as like the coherence of the presentation, logical structure authentic as possible. Multiple approaches for simulating or whether the message is composed in an interesting the audience in presentation training software have been way are neglected. explored. Schneider et al. [17] and van Ginkel et al. [18] Furthermore, Ochoa [13] pointed out that most premade use of Virtual Reality, Trinh et al. [19] created an sentation skills training applications were not adapted by application with a robotic head as an audience and Ochoa users. Only one of these applications ([20]) has been used et al. [20] used a screen with a pre-recorded audience. in real-world educational settings. This is puzzling since However, in most cases, the audience was quite static, these applications were generally scoring well in user
The main objective of this work is to find out how to train students’ MCSPS by means of computer-based tools. An overview of the context, the tackled research gap, the main research question, objectives, planned explorations, planned contributions and planned evaluations can be found in figure 1. Furthermore, a timeline overview is given in figure 2. The planned steps are described in more detail in the following. evaluations [13]. However, in most cases, the questions in the user evaluations were about whether students thought the system was useful, whether they learned with it and whether they would use it again [13]. Having systems with good usability, but low actual usage might point to not having addressed the actual needs and problems of the user base so that the perceived usefulness is low. Key to addressing the users’ needs and problems is to involve all relevant stakeholders already in the design phase [26]. However, to the best of our knowledge, none of the existing presentation training applications involved users already when designing their technology but only in the evaluation of their prototype.
The most common way to train and assess communication skills is to train and assess presentation skills [10]. Therefore, this PhD project will focus on training presentation skills with computer-based tools in or- 4.1. Collecting requirements - What do der to increase communication skills. However, presen- teachers and students expect, want tation skills training is a relatively large field. As dis- and need from an application for cussed in section 2, there is already research on computersupported applications for training non-verbal communi- training MCSPS? cation in presentations as well as slide design. Neverthe- In contrast to existing research in computer-supported less, the content is still neglected which is why the focus presentation skills training, we will follow a participatory of this PhD project will be on training message compo- design process and incorporate presentation teachers as sition skills for public speaking (MCSPS). In contrast to well as students in higher education already before startgiving feedback on the message delivery, feedback on ing to design or implement a solution. The first part of message composition should mainly be given before the the participatory design process aims to work out the rehearsal [12]. The reason for this is that giving feed- requirements for the application based on best practices back on the message composition before the rehearsal and recommendations of the presentation teachers as allows students to first focus on composing the message well as the needs and the everyday lives of the users. and then, when rehearsing, to focus mainly on how to Based on this, requirements for an application for traindeliver that message. Consequently, a focus on MCSPS ing MCSPS will be retrieved. also means focusing on the preparation phase happening For incorporating the perspective of presentation before the rehearsal. teachers, expert interviews will be conducted. The fo
Based on these initial considerations the main research cus of these semi-structured interviews will be on the question will be as following: preparation of a presentation in order to find out which Main RQ How to train students’ MCSPS by means of steps are needed to prepare an efective presentation, how computer-based tools? these steps should be done and where students struggle with, according to the teachers.
This main research question is then further divided into To incorporate the perspective of students, we will confour sub research questions: duct semi-structured interviews with students in higher RQ1 Which aspects of students’ MCSPS are most suited education as well. The focus of these interviews should to be trained by means of computer-based tools be on their experiences with presentations and especially in order to increase students’ MCSPS? message composition. It is important to find out where an MCSPS training application might help to solve actual RQ2 What has been done by related work to support problems the students have. Furthermore, the perceived training the aspect of students’ MCSPS identified relevance of those problems needs to be determined. This in RQ1 with computer-based tools? is important, in order to be able to better evaluate for which problems they actually would take the additional efort of using a special training application. Apart from
this, it should be identified how students prepare a presentation normally, especially in contrast to how the experts suggest preparing a presentation.
To evaluate what has been done previously, a scoping review is planned to be conducted. Scoping reviews are a kind of literature review suited to create a map of the existing research of a broad topic [28]. Similarly as in systematic literature reviews, the literature is searched systematically, however, the scope is broader and the goal is diferent [ 29]. A systematic literature review intends to answer very specific research questions by aggregating the results of existing studies while a scoping review primarily intends to give an overview of the existing literature in a broader research area. In our case, we want to examine what research has been previously done in the intersection of MCSPS training and technologyenhanced learning. Therefore, we plan to do a scoping we will determine which aspects the software should give review. This scoping review will be executed following feedback upon. the PRISMA checklist [30]. In order to include the exist- In the second iteration, we will augment the prototype ing knowledge in MCSPS training in general as well as with automatic feedback on the chosen aspect. However, the existing knowledge on how to support this process it’s important to note that traditional feedback is not apwith technology, the scoping review will consist of two plicable in the context of presentations [24], as there is parts. no clear ”right or wrong”. The reasons for this are on the
In the first part, existing knowledge on presentation one hand that there are numerous efective approaches skills training, with a special focus on message composi- to delivering a message and on the other hand that the tion, will be reviewed. Based on this, a conceptual model efectiveness of specific presentation behaviors depends will be developed. on various factors (e.g. the content, the audience, the pre
Afterwards, based on this conceptual model, a review senter). As a result, we must find an alternative approach of existing technology-enhanced learning applications to inform students about behaviors that are frequently will be conducted. This part of the review will give an inefective for message delivery, while also clearly comoverview of which parts of the conceptual model have municating that these behaviors can be valid when used been addressed with computer-based tools and how. intentionally and with a specific purpose in mind. This second prototype will then also be tested with users.
prototype - How to design and implement a software for training oral communication content preparation skills? When the requirements are clear, a design for a first prototype should be created. In the first iteration, the prototype will only provide guidance on composing a message for a presentation and will not provide automatic feedback. This prototype will then be implemented and tested with students. Based on the results of these tests as well as the results of the teacher and student interviews
the prototype on presentation content preparation skills - How is using this presentation content preparation skills training application afecting the skills of users? The primary objective of the proposed application is to facilitate the training of students’ MCSPS. Consequently, it is important to evaluate how using this application affects students’ MCSPS. Ideally, such an evaluation should be a randomized controlled trial. However, for ethical study on the centrality of the discipline’s content reasons, we can not split up students of the same class and pedagogy, Communication Education 66 (2017) into two groups and provide an intervention potentially 402–422. doi:10.1080/03634523.2016.1265136. influencing their grades in only one of the groups. As a [2] N. E. Dunbar, C. F. Brooks, T. Kubicka-Miller, result, we will do a quasi-experiment. Oral Communication Skills in Higher Education:
The quasi-experiment will be conducted within the Using a Performance-Based Evaluation Rubric same course, with diferent groups of students in three to Assess Communication Skills, Innovative distinct semesters. This will result in the following three Higher Education 31 (2006) 115–128. doi:10.1007/ groups: one control group and two experimental groups. s10755- 006- 9012- x.
The course structure will remain consistent across all [3] J. Trevelyan, The Making of an Expert Engineer, three semesters. In each semester, every student will be zeroth ed., CRC Press, 2014. doi:10.1201/b17434. required to deliver three presentations. The first and third [4] J. S. Hinton, M. W. Kramer, The impact of presentations will serve as pre- and posttests, prepared self-directed videotape feedback on students’ selfwithout the use of our tool. For the second presentation, reported levels of communication competence and students in the experimental groups will use our tool. apprehension, Communication Education 47 (1998) One experimental group will use the version of the tool 151–161. doi:10.1080/03634529809379119. providing only guidance, while the other will use the [5] S. Živković, The Importance Of Oral Presentations version ofering guidance and individual feedback. The For University Students, Mediterranean Journal plan for the quasi-experiment is also visualized in figure of Social Sciences (2014). doi:10.5901/mjss.2014. 2. v5n19p468. [6] JQI. Joint Quality Initiative, et al., Shared ‘Dublin’descriptors for short cycle, first cycle, sec5. Conclusion ond cycle and third cycle awards, Dublin, Joint Quality Initiative (2004).
This paper presented an overview of a PhD project on [7] P. C. Kyllonen, Measurement of 21st century skills exploring how students could be supported in training within the common core state standards, in: Invimessage composition skills for public speaking (MCSPS) tational Research Symposium on Technology Enby means of computer-based tools. To accomplish this hanced Assessments, 2012, pp. 7–8. goal, the project will use participatory design within the [8] V. Chan, Teaching oral communication in undercontext of design-based research. graduate science: Are we doing enough and doing
The research process starts with interviews with pre- it right?, Journal of Learning Design 4 (2011) 71–79. sentation teachers as well as students in order to collect requirements for such a tool. In parallel, a scoping review [9] dNo.i:E10..B5a2r0r4e/ttj, lGd..-vZ4.i3L.iu8,2.H.-C. Wang, Seamon MCSPS training as well as the support of this train- less learning for oral presentations: Designing ing by means of computer-based tools will be conducted. for performance needs, Computer Assisted LanAfter the requirements are formalized, two versions of a guage Learning 35 (2022) 551–576. doi:10.1080/ prototype will be designed, implemented and tested. The ifrst version will provide only guidance on composing [10] 0L9.L58eF8e2b2v1r.e2, 0L2. 0E.. 1L7e2F0e2b5v4re., The introductory comthe message of a presentation, while the second one will munication course from 1956 to 2016: A metaprovide guidance and additionally give individual auto- synthesis, Communication Education 69 (2020) matic feedback. Finally, the efect of using these variants of the tool on students’ MCSPS will be evaluated in a [11] 1S9.9S–ta2w23is.kdio, iA:1.0G.e1r0m80u/th03,P63.Y45ar2b3o.r2o0u1g9h.,1V6.7A9 3lf8o0r.d, quasi-experiment. L. Parrish, Infusing Twenty-First-Century Skills into Engineering Education, Journal of Business Acknowledgments and Psychology 32 (2017) 335–346. doi:10.1007/ s10869- 016- 9477- 2.
This PhD project is part of the HyTea-project (FKZ: [12] L. LeFebvre, Team-based learning for the basic 01|S22075 A) funded by the German Federal Ministry communication course: A transformative pedagogof Education and Research (BMBF). ical approach, Review of Communication 16 (2016) 192–212. doi:10.1080/15358593.2016.1187454. [13] X. Ochoa, Multimodal Systems for Automated References Oral Presentation Feedback: A Comparative Analysis, in: M. Giannakos, D. Spikol, D. Di Mitri, [1] S. P. Morreale, J. M. Valenzano, J. A. Bauer, Why K. Sharma, X. Ochoa, R. Hammad (Eds.), The Mulcommunication education is important: A third timodal Learning Analytics Handbook, Springer