Eleventh International Workshop Modelling and Reasoning in Context (MRC) @ECAI 2020 1 Observation of Communicative Behaviour when Learning a Movement Sequence: Prequel to a Case Study Julian Blohm1 and Jörg Cassens2 and Rebekah Wegener3 Abstract. When trying to improve human-machine communi- behaviour will perhaps enable us to build systems for better commu- cation it can be helpful to better understand human thinking and nication between humans and machines [39]. behaviour. In some cases, it is not only feasible, but also helpful Communication is so much more than spoken or written language. to transfer recognised communicative patterns to machine interac- Natural language is inherently multimodal in nature [36]. Because of tion. The benefits of multimodal interfaces have been explored for this, the classic transmitter-receiver model of information processing quite some time, arguably starting with the famous “put that there!” that is often used in computer science is typically extended to include demonstration system [4], leading to a variety of theoretical works other modalities depending on the needs of the research [27, 32]. In and application systems [14]. However, there is still a lot of work to natural interaction, the progression and the success or failure of the be done before non-verbal elements of communication can challenge interaction can be shaped by many different factors including be- the predominant paradigms for human-computer interaction [18, 35]. havioural or contextual elements [27, 29]. We have previously worked on multimodal behaviour in specific con- The underlying research program of which this paper is a part aims texts of interaction [17] and on explanation-aware systems [16] as to investigate whether the consideration of behavioural and contex- well as a combination thereof [8]. In order to better understand which tual elements can provide insights that can be used for the optimiza- aspects of human-to-human communicative behaviour can (at least) tion of future explanation-aware systems. To this end, an experimen- be mimicked by computational systems, we perform empirical re- tal setup was developed in a pre-structured explanatory situation. In search with humans in this area. In this paper, we present a pre-study this experimental domain, the test subjects’ goal is to learn a be- for an experimental setup that looks at human-to-human communica- havioural sequence that is indicative of complex motor skill learning, tive behaviour during movement sequence learning. This will enable in this instance a Judo technique. The aim is to design the instruc- us to better understand the role of different features in explanatory tional material in such a way that it is relatively self explanatory, behaviour. In the end, a better understanding of this behaviour will making verbal communication superfluous. Non-verbal behaviours hopefully enable us to optimize human-machine interaction as it per- such as gestures, facial expressions and body movement are observed tains to explainable AI and might aid the development of better train- and the communicative behaviour is recorded as accurately and un- ing systems for learning complex motor skills in high risk environ- obtrusively as possible. This allows for the analysis of the timing and ments such as surgery or emergency medicine. potential motivation for additional communication and the consider- ation of how this might relate to contextual and individual factors. 1 Research Questions An increasing number of tasks in all walks of life are being taken 2 Human-Machine Communication on or supported at least partially by technology e.g. learning in high risk environments like surgery, where learning a new complex motor Human-machine communication (HMC) refers to the mutual Infor- skill can be essential to saving life, but where learning by doing is mation exchange between human and machine [41]. This means life threatening [21, 40]. An important aspect here is the notion of the “intuitive”4 , natural, and therefore multimodal interaction be- cooperative systems, mixed-initiative systems or, more general, the tween people and information processing systems. Early textual chat- notion of “human-in-the-loop” [34, 42]. bots such as Eliza [38] mainly responded to keywords or phrases For the often implied sharing of tasks between humans and ma- and answered with canned responses. This was then amended using chines to be effective, it is necessary that the exchange of informa- template-based systems [5]. tion between human and machine runs smoothly. While it has been By now, systems using spoken natural language and learned mod- (and in some cases still is) common to model humans as information els have become mainstream. Technologies such as Google Duplex, processing systems [6, 19], which means that they perceive signals Alexa (Amazon) and Siri (Apple) respond to questions and answer from stimuli through the sensory perception system, process that in- them appropriately, even mimicking non-task oriented aspects of hu- formation through the cognitive processing system and finally act man communication. For example, Google Duplex uses typical hu- on that information; human information processing is quite distinct man behaviours like a short pause for reflection between sentences from machine data processing. Despite the different capabilities and or uttering “hm” [22]. potentials, however, a better understanding of human communicative Turning to other modalities than spoken (or written) language, modern sensor technology in principle opens up the potential for 1 University of Hildesheim, Germany, email: blohmj@uni-hildesheim.de 2 University of Hildesheim, Germany, email: cassens@cs.uni-hildesheim.de 4 Intuitive is used here in a cultural-historic sense and is not referring to an 3 University of Salzburg, Austria, email: rebekah.wegener@sbg.ac.at assumed inherent property. Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Eleventh International Workshop Modelling and Reasoning in Context (MRC) @ECAI 2020 2 simulation of communication that is comparatively close to human- 4.1 Method to-human communication [5]. Despite these improvements, commu- nication does not always running smoothly. 4.1.1 Participatory observation Participatory observation was chosen as the method of data collec- tion as this is a standard method of field research and thus offers 3 Human-Human Communication a point of comparison [11, 23, 28]. During the procedure, two ob- servers recorded the behaviour of the participants and the trainer. They used pre-formulated observation sheets with the option to note Interpersonal communication can be described by various linguistic, down individual remarks. In addition, the trainer wrote down their semiotic or communication models [20]. In the pre-study described observations after the exercise using a memory protocol. here, we focused on the characteristics of communication approach The multi-perspective data collection (trainer, respondent, ob- [33] as well as as an integrative model of communication [27]. Com- servers) served to achieve comprehensive observations by relating munication, therefore, is here understood as a process that arises data points to each other and allowing them to be corrected if nec- through interactions. Verbal and non-verbal elements such as ges- essary. In the run-up to the project, the aim was to take into account tures, facial expressions and body language are used. (and where possible control) all factors that could plausibly have an Decisive for the unfolding of the communication process is the re- influence on the result and thus on the reliability of the data to be spective context, especially personal and situational context. Besides collected. For example, the context (place, time, atmosphere), the observable elements, non-visible activities determine communica- behaviour of the trainer and the observers, and the structuring of the tion behaviour (communication rules, sympathy, tenor, prejudices). execution was precisely defined. By pre-structuring the observation Basic prerequisite for successful communication is the use of a com- sheets, the focus of the observers was specifically directed to essen- mon repertoire of signs and symbols by the communication partners. tial aspects in contrast to free wording (validity). Elements of move- Nevertheless, misunderstandings and errors can occur when coding ment, verbal expressions and observations on the execution of the and decoding a message. The overall course of events is influenced Judo technique were recorded. by contextual and psychological factors. The objectives of a commu- All observations were made with the same observers and in the nication, the response and feedback behaviour, and the mutual per- same room. Those carrying out the observations kept an unobtru- ception also influence the course [27]. These factors should be taken sive external appearance. The test persons were addressed randomly into account when planning the empirical study. and did not have any personal relationship to the persons performing According to Watzlawick, humans will always communicate even the observations. In order to achieve reliable results, the test persons if they don’t intend to communicate [37]. Thus every behaviour has had no prior knowledge or reservations. To ensure this, a preparatory a communicative character. Part of the non-verbal side of communi- questionnaire was used. Using teaching material that was produced cation pertains to affect. Body language is related to individual vari- specifically for the task, the observation can be repeated reliably. ation and the situation. However, it is not possible to draw conclu- Since the trainer was also part of the exercise, various safeguards sions about the emotions of the communication partner by interpret- were put it place to ensure consistency over the course of the exper- ing a single body language expression. Not only are they not unique iment. The behaviour of the trainer was precisely defined and was in themselves, but we will always only see the expression of affect, also checked by an observer. With the help of the reflection sheet, and not the underlying emotion. Facial expressions vary individually, observations made in different runs could then be compared. contextually and culturally, therefore other elements are usefully in- cluded [3], for instance, our voice contains important and surpris- ingly reliable information about our emotional state [25]. 4.1.2 Selection of the object of explanation: Learning a Judo technique For the analysis of the non-verbal communication elements, learning 4 Planning of a movement sequence was chosen. In contrast to a purely cogni- tive learning situation, it can be clearly seen whether the respondent In a random sample, test subjects are to learn a Judo technique, i.e. has understood the given explanations by following the exercise in a complex motor skill in the form of a motion sequence. Instruction action. The fact that understanding and learning has taken place can on how to perform this motion sequence is given via text, video, and be demonstrated by the action itself [26]. photo sequences. The respective learning steps are evaluated when While the guidelines for the correct execution of Judo technique the motion sequence is enacted afterwards. The study consisted of by the German Judo Association (DJB) [10] were taken into account, two phases, a small pilot to test the experimental protocol, and the they were applied in a modified form because participants in this case study itself. For the remainder of this article, we will focus on study were complete novices. The use of the DJB guidelines however the small pilot phase and the process leading up to the experiment. gave a consistent and detailed measure for evaluating. The didactic In the preparatory phase, the focus lies on reflections on the structure of a training unit is familiar to the first author of this article method, the context of situation, the explanatory materials, and the who takes part in the experiment as a trainer. He has been active in evaluation strategy. Influencing factors and barriers which may com- Judo himself for about 20 years and has been active as a trainer for 5 plicate the course of communication are to be considered. These pre- years. In his role as a Judo trainer, he has to be able to teach the Judo liminary considerations are then evaluated in test runs checked and techniques in an understandable way. corrected. The trainer is part of the communication process and since His personal experience that the exclusive use of simple statements the test subject and the trainer together determine the course of com- (verbalizations), pictures (visualizations of throwing techniques), or munication, the behaviour of the trainer has to be taken into consid- even throwing descriptions in text form are not sufficient is consistent eration as well. with the literature [13, 21, 24] and translates to complex motor skill Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Eleventh International Workshop Modelling and Reasoning in Context (MRC) @ECAI 2020 3 learning in other disciplines than sports [9, 40]. Often, a combina- of a student. Stress elements contained in this situation were alle- tion of different explanation strategies are used and Judo instructors viated by the surrounding atmosphere. The behaviour of the trainer generally teach the technique using the following steps: played an important role in creating a pleasant and open atmosphere. In order to enable the participants to act as relaxed and pressure-free • Verbal explanation, as possible, the appearance, clothing, language style, posture etc. of • Demonstration both the trainer and the observers were prescribed before running the • In sequences with explanations experiments [2, 12]. • Clarifying demands These considerations were confirmed in the test runs where all • Practice phase with individual help participants noted that they felt comfortable in the situation and even enjoyed it. Even if the underlying mechanics and movements are understood in principle, when learning a new complex motor skill it is not un- 4.2.3 Observation usual to initially have difficulties in performing it correctly. If nec- essary, the technique should be explained again or shown repeatedly. As the experiment aimed for a relaxed atmosphere as close to ev- The motor skill chosen for this study was the “joint lock” because eryday life as possible, observers were used for both external and it does not require any previous knowledge or additional equipment. self-observation. They went directly into the setting, actively partici- With an arm joint lock it is important to fix the elbow joint of the pating and writing notes which are then evaluated. They were briefed partner and then overstretch it. and trained beforehand [30]. The use of cameras was deliberately avoided because the awareness of being under observation can lead to changes in behaviour (Hawthorne Effect) [1]. 4.1.3 Multimodality, sequential explanations and action In practice runs before the small pre-test, the observers were The instructional material was presented to the participants in digi- trained in the handling of the different observation sheets. It turned tal form using the keynote presentation software. It consisted of 10 out that the observers were able to follow the practice runs well and pages, 5 photos and 5 videos. It was designed to be self-explanatory that the pre-defined structure of the observation sheets was helpful. so that the verbal communication components were reduced. The fo- The overall impression and individual peculiarities could be easily cus of the observation was on non-verbal behaviour and movement recognized and recorded. elements. Nevertheless, some details were missed. In contrast to the planned The sequence of movements to be learned was broken down into setup, a recording device was deemed necessary in order to record individual learning sequences, which are modelled on the normal verbal utterances instead of transcribing them on-the-fly. This was training situation in Judo practice session. The acquisition phase was done by using a mobile phone during the later runs. According to the followed by an action phase, in which participants act and practise test subjects, this small, inconspicuous camera was not noticeable or what they have learned. The training texts were written such that the even perceived as disturbing. participants were directly addressed and could identify more easily with their role. The texts were kept simple and were developed as 4.2.4 Selection of participants an instruction manual. The written description of the movement se- quences is supplemented with photos and videos. The photos show The test subjects were recruited directly and invited to participate af- the current state or the initial situation and details. The video pro- ter a short eligibility interview. The following selection criteria were vides the movement sequence. All pages are structured identically established: to provide the participants with an easier orientation of the learning • Age group 18 and above (legal adult). path. • Body height, approx. between 1.70-1.90m. • The potential participants should have an average physical fitness. 4.2 Setting – A movement exercise is carried out with the persons addressed in order to test their coordination and movement skills (oppo- 4.2.1 Place and Time site windmill arm movement). The location of the study has an influence on the mood and motiva- • Good German language skills are necessary, as texts must be read tion of the participants [15]. For this reason, a room on the premises and understood. of the University of Applied Sciences and Arts Hanover was chosen • No previous knowledge of Judo or wrestling, determined by as it is a simple, small working room that is located on the fifth floor means of a questionnaire. with little disturbance from outside noise. Students of the university • Persons who do not wish to be touched or who do not agree with are familiar with this type of room and the choice of a workspace the general conditions of the experiment are also excluded. as opposed to a private room or training facility provided the exper- iment with a quiet, neutral space. A clock was not visible so that no For organisational and technical reasons, the participants were re- time pressure was built up and sessions were scheduled in the early cruited at the university campus in Hanover. The total of 10 partici- evening or on weekends, so that the participants arrived relatively pants were young adults. rested. 4.3 Execution 4.2.2 Atmosphere 4.3.1 Procedure The explanatory situation tested was a learning situation with clearly The trainer invited the participants and gave an initial overview of defined roles. The trainer is the instructor, the participant has the role the goals and progress of the experiment. The participants filled in Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Eleventh International Workshop Modelling and Reasoning in Context (MRC) @ECAI 2020 4 questionnaires I (personal) before, and II (feedback) after the exper- transition from acquisition to practice were recorded. The focus here iment. The explanatory material alternates between acquisition and was on the manner of implementation, i.e. whether the test subject action phases. The participant could scroll forward or backward and starts hesitantly or actively. The trainer describes the non-verbal or repeat individual parts at any time. The trainer was available as a verbal communication behaviour during the action phase. contact person for questions and interaction during the entire process The study uses a semi-standardised procedure, since it works with and operated the PC. Two observers filled out observation sheets of pre-formulated criteria, but it also leaves room for the recording of the communication partners A1 and A2 (participants) and B1 and new aspects. In addition, the video recordings were available for B2 (trainer). In parallel, video recordings were made using a mobile comparison. phone. Immediately after completion of the experiment, the trainer completed a memory protocol C on their own perceptions. 4.3.4 Reflection sheet 4.3.2 Questionnaires The reflection sheet C was filled in by the trainer directly after the The questionnaire I (personal questions), was handed out to the par- execution. The questions were answered spontaneously and reflect ticipants before the movement task was performed. The exclusion the first impression. The first questions refer to the execution of the criteria for the selection of participants were checked and personal judo technique. From the perspective of the experienced judo trainer, data was queried. Following to the integrative communication model the extent to which the technique is executed correctly was assessed [27], potential influencing factors such as previous knowledge, mo- and the process of learning was also examined. Afterwards, the re- tives, age, gender, etc. were taken into account. Volunteers were lationship between subject and trainer was described, especially its asked about their ability to understand instruction manuals because subjective impact. Attention was paid to the application of additional the judo technique is essentially developed step by step, similar to an help, when and why was this necessary, how help was given and instruction manual. whether it was successful. Questionnaire II (feedback) was given to the participants immedi- ately after the practice task had been executed in order to record the immediate experience. Questions were asked about the Judo tech- 4.4 Evaluation nique, the instructional material and the general conditions. When filling out the questionnaire, the test persons had the materials at The evaluation was derived from the observers‘ notes, the video their disposal. The feedback was intended to point out possible re- recordings and the trainer‘s reflection sheet C. The questionnaires structuring necessities for later follow-up studies. For example, the filled in by the test persons supplemented the data collected. Sim- test runs performed showed that some changes in the design had to ilarities and differences in the observations were interpreted and be made in order to achieve clarity. Also the detailed demand for pre- analysed. In this way the observations on handling of material, the vious knowledge of certain martial arts was reformulated into a more method and the course of communication can be viewed from dif- general question. ferent perspectives. This is intended to achieve the highest possible In addition, Questionnaire II asked for a self-assessment and in- degree of coverage. quires whether additional help was necessary both in terms of un- When describing the course of communication, the verbal and derstanding the material and performing the movement. Implicit in non-verbal remarks were recorded. The focus here is on the ques- the answers given is a distinction between whether the respondent tions of when, what and how it was communicated. The verbal com- asked for help of their own accord or whether the trainer intervened ments are clearly recognizable and can be written down. The non- proactively. Since the trainer is an essential part of the exercise, their verbal communication results from the context and the behavioural behaviour was described from the test person‘s perspective. In Ques- elements. Every “additional communication” was recorded. First of tionnaire II, the test person also gave a self-assessment of the degree all, a time stamp is noted, i.e. when the communication took place. In of difficulty and whether they needed help with the exercise. a second step the cause was examined. This resulted in the following areas for the evaluation: 4.3.3 Observation sheets The behaviour of the participant and of the trainer was recorded in 1. general personal data for the classification of the test subject separate observation sheets A and B. There is one observation sheet 2. recording of personality and behavioural characteristics (situa- each for the acquisition phase (A1 and B1) and a second for the ac- tional and context-related) tion phase (A2 and B2). 3. situation/atmosphere The observation sheets were pre-formulated according to selected 4. time, an average value is calculated criteria (verbal language, gestures, facial expressions, movement) 5. information part: handling of the materials/method and serve as an aid for the observer. They follow the chronologi- 6. linguistic comments cal sequence and repeat the fields of observation for the individual 7. body language sections in the same way. The pre-formulated fields of observation 8. action part should enable the observer to note many aspects in as short a time as possible. There is room for individual remarks so that the observers Each test subject was described individually. The self-reported can record unforeseen events. Nevertheless, the pre-formulated as- aspects and the observed behaviour were related to observed non- pects ensure a structured approach, especially for later evaluation. verbal communication behaviour. Hypotheses could then be formed The action part was mainly recorded using observation sheets A2 as to whether the non-verbal additional communication was due to and B2. In A2 the observer recorded descriptions in general form the inter-personal differences, the material, or the situation. The self- for implementation of the movement. In addition, aspects about the assessments were always related to the observed data. Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Eleventh International Workshop Modelling and Reasoning in Context (MRC) @ECAI 2020 5 5 First observations from trial runs orities themselves and this added discrepancies in the evaluations. It is difficult to counter this effect, but it may be necessary to improve Two trial runs were carried out and these test runs were intended to the training cycle for the observers. After all, ethnographic recording familiarise the observers and the trainer with the use of the observa- is a skill that itself requires a lot of practice. While the observation tion sheets and with the flow of the test. can not be considered representative due to the small number of test Overall, it was found that the planned procedure was reasonable subjects (10), the pre-study provided crucial learning for the revision and practicable. Materials offered proved to be sufficient for the par- of the larger study and it was also possible to obtain results that were ticipants. The test subjects were able to understand them and imple- useful and indicative in nature. ment the motions correctly. In the end, the participants were able to In the test runs of the designed study, non-verbal behaviour of the successfully perform the judo technique. They were satisfied with test subjects was transcribed in addition to verbal comments. Test their results and considered this learning path an acceptable alterna- subjects showed different communication needs, which could be read tive to classical Judo training. from behavioural cues. The non-verbal behaviour could also be seen Additionally, the multimodal explanation strategy, the decomposi- as expressions of inter-personal difference. Thus, for example, un- tion of the movement sequence to be learned into sequences, and the certainties that are shown through behaviour could be reacted to ac- alternation of acquisition and action phases, has proven to be useful. cordingly. Participants confirmed that they felt comfortable and enjoyed it. Even although participants worked independently with the training This indicates an overall relaxed atmosphere. After the introduction material and did not verbally ask for help, situations were identified in the first action part, participants wanted to perform the whole in which they signalled a need to communicate, e.g. by eye contact movement sequence immediately. or waiting [31]. It is helpful for explanatory systems, be they human The trainer had to intervene at this point and point out that only or technological, to react to this behaviour. the sequences shown should be practised. Here, the instructions by The prevalence and diversity of situations where eye contact the trainer had to be optimized so that the sequence would be clearer. played a crucial role in the interaction is indicative that a richer model Participants had to be encouraged to switch to the first action phase. of gaze might be beneficial for upcoming studies, particularly since Hesitation was signalled by eye contact and by waiting, indicating gaze is a feature that can readily be captured by existing sensors. that the test subjects needed some form of interactive response. This In the initial development of a situation where cooperation of mul- despite the fact that transition from acquisition to action is signalled tiple entities is central (collaborative or team work situations), the in the training material in such a way that execution could in principle need for additional communication is higher, so that the process and take place without any intervention by the trainer. the procedure are understood and mutual trust is created. The need for interactive response could indicate that there is a spe- In failure situations where corrective action and explanations are cific need for communication and information, especially in the ini- necessary, an appropriate communication strategy that includes mul- tial phase of becoming familiar with the learning path. Although the timodal feedback should be developed so that users do not give up explanatory material and the trainer‘s presentations contain a lot of in frustration or fail to recognize the error at all. An incidental find- relevant information, this did not seem to be sufficient for the partic- ing is that it appears from this experiment that impending frustration ipants during the acquisition phase. and possible abandonment of learning can be predicted from the be- In contrast, the need for eye contact during the action phase is haviour before it occurs so that an intervention might be possible. likely a result of the setup, as the technique is a partner task and it is This is consistent with findings in other work we have been done on necessary to respond to each other. So the search for eye contact can multimodal markers of importance [7]. here be interpreted as coordination during the execution. The trainer It was very clear from the experiment that explanations should reported that participants tend to react affirmatively to the search for be offered multimodally and, depending on the subject, also sequen- eye contact. Furthermore, eye contact was a frequently occurring be- tially. Repetitions and some redundancy, if necessary also in varia- havioural signal. tions, help participants to habituate to working methods and provide The evaluation of the different body signals, which of course are security and ultimately build a trust relationship. to be understood contextually, already suggest that the trainer should Within a human-centred approach to intelligent systems develop- respond adaptively to different test subjects. For example, test sub- ment, the better a system knows its user, the better it can potentially jects that are very cautious and reserved in the execution phase would respond to them and their individual needs. The experiment showed need encouragement in the action phase for a more courageous and that by taking behavioural elements into account, it is possible to get powerful execution. It is important to note that this encouragement to know the user or participant better. From the recognition of the in- can be shared using non-verbal cues. dividual needs for assistance, appropriate communication strategies can be designed. 6 Conclusions and Further Work References The goal of this experiment was two fold: to test the experimental protocol for the larger study and to see what aspects of human-to- [1] Renee L. Allen and Andrew S. Davis, Hawthorne Effect, 731–732, human interaction might be useful for designing and developing for Springer US, Boston, MA, 2011. [2] Michael Argyle, Adrian Furnham, Jean Ann Graham, et al., Social sit- human-to-machine interaction, particularly for explainable AI and uations, Cambridge University Press, 1981. training systems for high risk environments. In terms of testing the [3] Thomas Bøgevald Bjørnsten and Mette-Marie Zacher Sørensen, ‘Un- protocol, a number of aspects of the pre-study are being revised for certainties of facial emotion recognition technologies and the automa- the larger study. It should be noted that the pre-study showed defi- tion of emotional labour’, Digital Creativity, 28(4), 297–307, (2017). [4] Richard A. Bolt, ‘“put-that-there”: Voice and gesture at the graphics in- ciencies in our experiment protocol that will be rectified. Individual terface’, in Proceedings of the 7th Annual Conference on Computer behaviour of the participants could not be fully recorded and tran- Graphics and Interactive Techniques, SIGGRAPH ’80, p. 262–270, scribed. The observers also made individual judgements and set pri- New York, NY, USA, (1980). Association for Computing Machinery. Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Eleventh International Workshop Modelling and Reasoning in Context (MRC) @ECAI 2020 6 [5] Luka Bradeško and Dunja Mladenić, ‘A survey of chatbot systems [24] Karl M Newell, ‘Motor skill acquisition’, Annual review of psychology, through a loebner prize competition’, in Proceedings of Slovenian Lan- 42(1), 213–237, (1991). guage Technologies Society Eighth Conference of Language Technolo- [25] Virginia P Richmond, James C McCroskey, and Mark Hickson, Non- gies, pp. 34–37, (2012). verbal behavior in interpersonal relations, Allyn & Bacon, 2008. [6] Stuart Card, Thomas P. Moran, and Allen Newell, ‘The model human [26] Ulrike Rockmann, ‘Bewegungen verstehen und beherrschen’, Vom Ler- processor - an engineering model of human performance’, Handbook nen zum Lehren: Lern-und Lehrforschung für die Weiterbildung, 159, of perception and human performance., 2(45–1), (1986). (2006). [7] Jörg Cassens and Rebekah Wegener, ‘Supporting students through no- [27] Jessica Röhner and Astrid Schütz, Psychologie der Kommunikation, tifications about importance in academic lectures’, in Proceedings of Springer-Verlag, 2015. AmI 2018 – International Joint Conference on Ambient Intelligence, [28] Gabriele Rosenthal. Interpretative sozialforschung. eine einführung. eds., Achilleas Kameas and Kostas Stathis, volume LNCS, pp. 227– juventa, 2005. 232, Larnaca, Cyprus, (November 2018). Springer. ISBN: 978-3-030- [29] N. Santiano, L. Young, L.S. Baramy, R. Cabrera, E. May, Rebekah We- 03061-2. gener, David Butt, M. Parr, and Clinical Analysis Group, ‘The impact [8] Jörg Cassens and Rebekah Wegener, ‘Ambient explanations: Ambient of the medical emergency team on the resuscitation practice of critical intelligence and explainable ai’, in Proceedings of AmI 2019 – Eu- care nurses’, BMJ Quality and Safety, 20(2), 115–120, (2011). ropean Conference on Ambient Intelligence, eds., Ioannis Chatzigian- [30] Helmar Schöne, ‘Participant observation in political science: Method- nakis, Boris De Ruyter, and Irene Mavrommati, volume LNCS, Rome, ological reflection and field report’, in Forum Qualitative Sozial- Italy, (November 2019). Springer. forschung/Forum: Qualitative Social Research, volume 4, (2003). [9] Ligia Cordovani and Daniel Cordovani, ‘A literature review on obser- [31] Beatrix Schönherr, Syntax-Prosodie-nonverbale Kommunikation: em- vational learning for medical motor skills and anesthesia teaching’, Ad- pirische Untersuchungen zur Interaktion sprachlicher und parasprach- vances in Health Sciences Education, 21(5), 1113–1121, (2016). licher Ausdrucksmittel im Gespräch, volume 182, Walter de Gruyter, [10] Hannes Daxbacher, Klaus Hanelt, Roman Jäger, Klaus Keßler, Ulrich 2013. Klocke, Ralf Lippmann, Rudi Mieth, Hans Müller-Deck, Jan Schröder, [32] Claude Elwood Shannon, The Mathematical Theory of Communication, Mario Staller, Hans-Jürgen Ulbricht, and Franz Zeiser. Begleitmaterial by CE Shannon (and Recent Contributions to the Mathematical Theory zum dan-prüfungsprogramm - ein nachschlagewerk zu verschiedenen of Communication), W. Weaver, University of illinois Press, 1949. themen der dan-prüfungsordnung im deutschen judo bund e.v., 4. über- [33] Kommunikationspsychologie—Medienpsychologie, eds., Ulrike Six, arbeitete auflage, 2011. [Online; Stand 28.06.2020]. Uli Gleich, and Roland Gimmler, Beltz, 2007. [11] Uwe Flick, ‘Qualitative sozialforschung: Eine einführung (4. aufl., [34] Norbert Streitz, ‘Reconciling humans and technology: The role of vollst. überarb. und erw. neuausg.)’, Rororo Rowohlts Enzyklopädie, ambient intelligence’, in Ambient Intelligence, eds., Andreas Braun, 55654, (2006). Reiner Wichert, and Antonio Maña, pp. 1–16, Cham, (2017). Springer [12] Owen Hargie, ‘Interpersonal communication: A theoretical frame- International Publishing. work.’, in The handbook of communication skills, 29–63, Psychology [35] Matthew Turk, ‘Multimodal interaction: A review’, Pattern Recogni- Press, (1997). tion Letters, 36, 189–195, (2014). [13] Avi Karni, Gundela Meyer, Christine Rey-Hipolito, Peter Jezzard, [36] Gabriella Vigliocco, Pamela Perniss, and David Vinson, ‘Language as a Michelle M. Adams, Robert Turner, and Leslie G. Ungerleider, ‘The multimodal phenomenon: Implications for language learning, process- acquisition of skilled motor performance: Fast and slow experience- ing and evolution’, Philosophical transactions of the Royal Society of driven changes in primary motor cortex’, Proceedings of the National London. Series B, Biological sciences, 369, (09 2014). Academy of Sciences, 95(3), 861–868, (1998). [37] Paul Watzlawick, ‘Die axiome von paul watzlawick’, Unter: [14] Athanasios Katsamanis, Vassilis Pitsikalis, Stavros Theodorakis, and http://www. paulwatzlawick. de/axiome. html [26.06. 2011] S, 38, 60, Petros Maragos, Multimodal Gesture Recognition, 449–487, Associa- (2016). tion for Computing Machinery and Morgan & Claypool, 2017. [38] Joseph Weizenbaum, ‘Eliza—a computer program for the study of nat- [15] Naz Kaya and Feyzan Erkip, ‘Invasion of personal space under the con- ural language communication between man and machine’, Communi- dition of short-term crowding: A case study on an automatic teller ma- cations of the ACM, 9(1), 36–45, (1966). chine’, Journal of Environmental Psychology, 19(2), 183–189, (1999). [39] Dorsch Lexikon der Psychologie, ed., Markus Antonius Wirtz, Verlag [16] Anders Kofod-Petersen and Jörg Cassens, ‘Explanations and context in Hans Huber, 2020. [Online; Stand 13.06.2020]. ambient intelligent systems’, in Modeling and Using Context – CON- [40] Gabriele Wulf, Charles Shea, and Rebecca Lewthwaite, ‘Motor skill TEXT 2007, eds., Boicho Kokinov, Daniel C. Richardson, Thomas R. learning and performance: a review of influential factors’, Medical ed- Roth-Berghofer, and Laure Vieu, volume 4635 of LNCS, pp. 303–316, ucation, 44(1), 75–84, (2010). Roskilde, Denmark, (2007). Springer. [41] Tang Yuan Yan, Wang Patrick SP, and Yuen Pong Chi, Multimodal in- [17] Anders Kofod-Petersen, Rebekah Wegener, and Jörg Cassens, ‘Closed terface for human-machine communication, volume 48, World Scien- doors – modelling intention in behavioural interfaces’, in Proceed- tific, 2002. ings of the Norwegian Artificial Intelligence Society Symposium (NAIS [42] Fabio Massimo Zanzotto, ‘Human-in-the-loop artificial intelligence’, 2009), eds., Anders Kofod-Petersen, Helge Langseth, and Odd Erik Journal of Artificial Intelligence Research, 64, 243–252, (2019). Gundersen, pp. 93–102, Trondheim, Norway, (November 2009). Tapir Akademiske Forlag. ISBN: 978-8-2519-2519-8. [18] Stefan Kopp, ‘Giving interaction a hand: Deep models of co-speech gesture in multimodal systems’, in Proceedings of the 15th ACM on International Conference on Multimodal Interaction, ICMI ’13, p. 245–246, New York, NY, USA, (2013). Association for Computing Ma- chinery. [19] Karl-Friedrich Kraiss, ‘Der mensch als kognitive systemkomponente’, in Fahrzeug-und Prozeßführung, 26–70, Springer, (1985). [20] Robert M Krauss and Susan R Fussell, ‘Social psychological models of interpersonal communication’, Social psychology: Handbook of basic principles, 655–701, (1996). [21] Danielle E Levac, Meghan E Huber, and Dagmar Sternad, ‘Learning and transfer of complex motor skills in virtual reality: a perspective review’, Journal of NeuroEngineering and Rehabilitation, 16(1), 121, (2019). [22] Yaniv Leviathan and Yossi Matias, ‘Google duplex: an ai system for accomplishing real-world tasks over the phone’, Google AI blog, 8, (2018). [23] Philipp Mayring, Einführung in die qualitative Sozialforschung, Beltz Verlag, 2002. Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).