=Paper=
{{Paper
|id=Vol-2787/paper1
|storemode=property
|title=Observation of Communicative Behaviour when Learning a Movement Sequence: Prequel to a Case Study
|pdfUrl=https://ceur-ws.org/Vol-2787/paper1.pdf
|volume=Vol-2787
|authors=Julian Blohm,Jörg Cassens,Rebekah Wegener
|dblpUrl=https://dblp.org/rec/conf/ecai/BlohmCW20
}}
==Observation of Communicative Behaviour when Learning a Movement Sequence: Prequel to a Case Study==
https://ceur-ws.org/Vol-2787/paper1.pdf
Eleventh International Workshop Modelling and Reasoning in Context (MRC) @ECAI 2020 1
Observation of Communicative Behaviour when
Learning a Movement Sequence: Prequel to a Case Study
Julian Blohm1 and Jörg Cassens2 and Rebekah Wegener3
Abstract. When trying to improve human-machine communi- behaviour will perhaps enable us to build systems for better commu-
cation it can be helpful to better understand human thinking and nication between humans and machines [39].
behaviour. In some cases, it is not only feasible, but also helpful Communication is so much more than spoken or written language.
to transfer recognised communicative patterns to machine interac- Natural language is inherently multimodal in nature [36]. Because of
tion. The benefits of multimodal interfaces have been explored for this, the classic transmitter-receiver model of information processing
quite some time, arguably starting with the famous “put that there!” that is often used in computer science is typically extended to include
demonstration system [4], leading to a variety of theoretical works other modalities depending on the needs of the research [27, 32]. In
and application systems [14]. However, there is still a lot of work to natural interaction, the progression and the success or failure of the
be done before non-verbal elements of communication can challenge interaction can be shaped by many different factors including be-
the predominant paradigms for human-computer interaction [18, 35]. havioural or contextual elements [27, 29].
We have previously worked on multimodal behaviour in specific con- The underlying research program of which this paper is a part aims
texts of interaction [17] and on explanation-aware systems [16] as to investigate whether the consideration of behavioural and contex-
well as a combination thereof [8]. In order to better understand which tual elements can provide insights that can be used for the optimiza-
aspects of human-to-human communicative behaviour can (at least) tion of future explanation-aware systems. To this end, an experimen-
be mimicked by computational systems, we perform empirical re- tal setup was developed in a pre-structured explanatory situation. In
search with humans in this area. In this paper, we present a pre-study this experimental domain, the test subjects’ goal is to learn a be-
for an experimental setup that looks at human-to-human communica- havioural sequence that is indicative of complex motor skill learning,
tive behaviour during movement sequence learning. This will enable in this instance a Judo technique. The aim is to design the instruc-
us to better understand the role of different features in explanatory tional material in such a way that it is relatively self explanatory,
behaviour. In the end, a better understanding of this behaviour will making verbal communication superfluous. Non-verbal behaviours
hopefully enable us to optimize human-machine interaction as it per- such as gestures, facial expressions and body movement are observed
tains to explainable AI and might aid the development of better train- and the communicative behaviour is recorded as accurately and un-
ing systems for learning complex motor skills in high risk environ- obtrusively as possible. This allows for the analysis of the timing and
ments such as surgery or emergency medicine. potential motivation for additional communication and the consider-
ation of how this might relate to contextual and individual factors.
1 Research Questions
An increasing number of tasks in all walks of life are being taken 2 Human-Machine Communication
on or supported at least partially by technology e.g. learning in high
risk environments like surgery, where learning a new complex motor Human-machine communication (HMC) refers to the mutual Infor-
skill can be essential to saving life, but where learning by doing is mation exchange between human and machine [41]. This means
life threatening [21, 40]. An important aspect here is the notion of the “intuitive”4 , natural, and therefore multimodal interaction be-
cooperative systems, mixed-initiative systems or, more general, the tween people and information processing systems. Early textual chat-
notion of “human-in-the-loop” [34, 42]. bots such as Eliza [38] mainly responded to keywords or phrases
For the often implied sharing of tasks between humans and ma- and answered with canned responses. This was then amended using
chines to be effective, it is necessary that the exchange of informa- template-based systems [5].
tion between human and machine runs smoothly. While it has been By now, systems using spoken natural language and learned mod-
(and in some cases still is) common to model humans as information els have become mainstream. Technologies such as Google Duplex,
processing systems [6, 19], which means that they perceive signals Alexa (Amazon) and Siri (Apple) respond to questions and answer
from stimuli through the sensory perception system, process that in- them appropriately, even mimicking non-task oriented aspects of hu-
formation through the cognitive processing system and finally act man communication. For example, Google Duplex uses typical hu-
on that information; human information processing is quite distinct man behaviours like a short pause for reflection between sentences
from machine data processing. Despite the different capabilities and or uttering “hm” [22].
potentials, however, a better understanding of human communicative Turning to other modalities than spoken (or written) language,
modern sensor technology in principle opens up the potential for
1 University of Hildesheim, Germany, email: blohmj@uni-hildesheim.de
2 University of Hildesheim, Germany, email: cassens@cs.uni-hildesheim.de 4 Intuitive is used here in a cultural-historic sense and is not referring to an
3 University of Salzburg, Austria, email: rebekah.wegener@sbg.ac.at assumed inherent property.
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Eleventh International Workshop Modelling and Reasoning in Context (MRC) @ECAI 2020 2
simulation of communication that is comparatively close to human- 4.1 Method
to-human communication [5]. Despite these improvements, commu-
nication does not always running smoothly. 4.1.1 Participatory observation
Participatory observation was chosen as the method of data collec-
tion as this is a standard method of field research and thus offers
3 Human-Human Communication a point of comparison [11, 23, 28]. During the procedure, two ob-
servers recorded the behaviour of the participants and the trainer.
They used pre-formulated observation sheets with the option to note
Interpersonal communication can be described by various linguistic,
down individual remarks. In addition, the trainer wrote down their
semiotic or communication models [20]. In the pre-study described
observations after the exercise using a memory protocol.
here, we focused on the characteristics of communication approach
The multi-perspective data collection (trainer, respondent, ob-
[33] as well as as an integrative model of communication [27]. Com-
servers) served to achieve comprehensive observations by relating
munication, therefore, is here understood as a process that arises
data points to each other and allowing them to be corrected if nec-
through interactions. Verbal and non-verbal elements such as ges-
essary. In the run-up to the project, the aim was to take into account
tures, facial expressions and body language are used.
(and where possible control) all factors that could plausibly have an
Decisive for the unfolding of the communication process is the re-
influence on the result and thus on the reliability of the data to be
spective context, especially personal and situational context. Besides
collected. For example, the context (place, time, atmosphere), the
observable elements, non-visible activities determine communica-
behaviour of the trainer and the observers, and the structuring of the
tion behaviour (communication rules, sympathy, tenor, prejudices).
execution was precisely defined. By pre-structuring the observation
Basic prerequisite for successful communication is the use of a com-
sheets, the focus of the observers was specifically directed to essen-
mon repertoire of signs and symbols by the communication partners.
tial aspects in contrast to free wording (validity). Elements of move-
Nevertheless, misunderstandings and errors can occur when coding
ment, verbal expressions and observations on the execution of the
and decoding a message. The overall course of events is influenced
Judo technique were recorded.
by contextual and psychological factors. The objectives of a commu-
All observations were made with the same observers and in the
nication, the response and feedback behaviour, and the mutual per-
same room. Those carrying out the observations kept an unobtru-
ception also influence the course [27]. These factors should be taken
sive external appearance. The test persons were addressed randomly
into account when planning the empirical study.
and did not have any personal relationship to the persons performing
According to Watzlawick, humans will always communicate even
the observations. In order to achieve reliable results, the test persons
if they don’t intend to communicate [37]. Thus every behaviour has
had no prior knowledge or reservations. To ensure this, a preparatory
a communicative character. Part of the non-verbal side of communi-
questionnaire was used. Using teaching material that was produced
cation pertains to affect. Body language is related to individual vari-
specifically for the task, the observation can be repeated reliably.
ation and the situation. However, it is not possible to draw conclu-
Since the trainer was also part of the exercise, various safeguards
sions about the emotions of the communication partner by interpret-
were put it place to ensure consistency over the course of the exper-
ing a single body language expression. Not only are they not unique
iment. The behaviour of the trainer was precisely defined and was
in themselves, but we will always only see the expression of affect,
also checked by an observer. With the help of the reflection sheet,
and not the underlying emotion. Facial expressions vary individually,
observations made in different runs could then be compared.
contextually and culturally, therefore other elements are usefully in-
cluded [3], for instance, our voice contains important and surpris-
ingly reliable information about our emotional state [25]. 4.1.2 Selection of the object of explanation: Learning a
Judo technique
For the analysis of the non-verbal communication elements, learning
4 Planning of a movement sequence was chosen. In contrast to a purely cogni-
tive learning situation, it can be clearly seen whether the respondent
In a random sample, test subjects are to learn a Judo technique, i.e. has understood the given explanations by following the exercise in
a complex motor skill in the form of a motion sequence. Instruction action. The fact that understanding and learning has taken place can
on how to perform this motion sequence is given via text, video, and be demonstrated by the action itself [26].
photo sequences. The respective learning steps are evaluated when While the guidelines for the correct execution of Judo technique
the motion sequence is enacted afterwards. The study consisted of by the German Judo Association (DJB) [10] were taken into account,
two phases, a small pilot to test the experimental protocol, and the they were applied in a modified form because participants in this
case study itself. For the remainder of this article, we will focus on study were complete novices. The use of the DJB guidelines however
the small pilot phase and the process leading up to the experiment. gave a consistent and detailed measure for evaluating. The didactic
In the preparatory phase, the focus lies on reflections on the structure of a training unit is familiar to the first author of this article
method, the context of situation, the explanatory materials, and the who takes part in the experiment as a trainer. He has been active in
evaluation strategy. Influencing factors and barriers which may com- Judo himself for about 20 years and has been active as a trainer for 5
plicate the course of communication are to be considered. These pre- years. In his role as a Judo trainer, he has to be able to teach the Judo
liminary considerations are then evaluated in test runs checked and techniques in an understandable way.
corrected. The trainer is part of the communication process and since His personal experience that the exclusive use of simple statements
the test subject and the trainer together determine the course of com- (verbalizations), pictures (visualizations of throwing techniques), or
munication, the behaviour of the trainer has to be taken into consid- even throwing descriptions in text form are not sufficient is consistent
eration as well. with the literature [13, 21, 24] and translates to complex motor skill
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Eleventh International Workshop Modelling and Reasoning in Context (MRC) @ECAI 2020 3
learning in other disciplines than sports [9, 40]. Often, a combina- of a student. Stress elements contained in this situation were alle-
tion of different explanation strategies are used and Judo instructors viated by the surrounding atmosphere. The behaviour of the trainer
generally teach the technique using the following steps: played an important role in creating a pleasant and open atmosphere.
In order to enable the participants to act as relaxed and pressure-free
• Verbal explanation, as possible, the appearance, clothing, language style, posture etc. of
• Demonstration both the trainer and the observers were prescribed before running the
• In sequences with explanations experiments [2, 12].
• Clarifying demands These considerations were confirmed in the test runs where all
• Practice phase with individual help participants noted that they felt comfortable in the situation and even
enjoyed it.
Even if the underlying mechanics and movements are understood
in principle, when learning a new complex motor skill it is not un-
4.2.3 Observation
usual to initially have difficulties in performing it correctly. If nec-
essary, the technique should be explained again or shown repeatedly. As the experiment aimed for a relaxed atmosphere as close to ev-
The motor skill chosen for this study was the “joint lock” because eryday life as possible, observers were used for both external and
it does not require any previous knowledge or additional equipment. self-observation. They went directly into the setting, actively partici-
With an arm joint lock it is important to fix the elbow joint of the pating and writing notes which are then evaluated. They were briefed
partner and then overstretch it. and trained beforehand [30]. The use of cameras was deliberately
avoided because the awareness of being under observation can lead
to changes in behaviour (Hawthorne Effect) [1].
4.1.3 Multimodality, sequential explanations and action In practice runs before the small pre-test, the observers were
The instructional material was presented to the participants in digi- trained in the handling of the different observation sheets. It turned
tal form using the keynote presentation software. It consisted of 10 out that the observers were able to follow the practice runs well and
pages, 5 photos and 5 videos. It was designed to be self-explanatory that the pre-defined structure of the observation sheets was helpful.
so that the verbal communication components were reduced. The fo- The overall impression and individual peculiarities could be easily
cus of the observation was on non-verbal behaviour and movement recognized and recorded.
elements. Nevertheless, some details were missed. In contrast to the planned
The sequence of movements to be learned was broken down into setup, a recording device was deemed necessary in order to record
individual learning sequences, which are modelled on the normal verbal utterances instead of transcribing them on-the-fly. This was
training situation in Judo practice session. The acquisition phase was done by using a mobile phone during the later runs. According to the
followed by an action phase, in which participants act and practise test subjects, this small, inconspicuous camera was not noticeable or
what they have learned. The training texts were written such that the even perceived as disturbing.
participants were directly addressed and could identify more easily
with their role. The texts were kept simple and were developed as 4.2.4 Selection of participants
an instruction manual. The written description of the movement se-
quences is supplemented with photos and videos. The photos show The test subjects were recruited directly and invited to participate af-
the current state or the initial situation and details. The video pro- ter a short eligibility interview. The following selection criteria were
vides the movement sequence. All pages are structured identically established:
to provide the participants with an easier orientation of the learning • Age group 18 and above (legal adult).
path. • Body height, approx. between 1.70-1.90m.
• The potential participants should have an average physical fitness.
4.2 Setting – A movement exercise is carried out with the persons addressed
in order to test their coordination and movement skills (oppo-
4.2.1 Place and Time site windmill arm movement).
The location of the study has an influence on the mood and motiva- • Good German language skills are necessary, as texts must be read
tion of the participants [15]. For this reason, a room on the premises and understood.
of the University of Applied Sciences and Arts Hanover was chosen • No previous knowledge of Judo or wrestling, determined by
as it is a simple, small working room that is located on the fifth floor means of a questionnaire.
with little disturbance from outside noise. Students of the university • Persons who do not wish to be touched or who do not agree with
are familiar with this type of room and the choice of a workspace the general conditions of the experiment are also excluded.
as opposed to a private room or training facility provided the exper-
iment with a quiet, neutral space. A clock was not visible so that no For organisational and technical reasons, the participants were re-
time pressure was built up and sessions were scheduled in the early cruited at the university campus in Hanover. The total of 10 partici-
evening or on weekends, so that the participants arrived relatively pants were young adults.
rested.
4.3 Execution
4.2.2 Atmosphere 4.3.1 Procedure
The explanatory situation tested was a learning situation with clearly The trainer invited the participants and gave an initial overview of
defined roles. The trainer is the instructor, the participant has the role the goals and progress of the experiment. The participants filled in
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Eleventh International Workshop Modelling and Reasoning in Context (MRC) @ECAI 2020 4
questionnaires I (personal) before, and II (feedback) after the exper- transition from acquisition to practice were recorded. The focus here
iment. The explanatory material alternates between acquisition and was on the manner of implementation, i.e. whether the test subject
action phases. The participant could scroll forward or backward and starts hesitantly or actively. The trainer describes the non-verbal or
repeat individual parts at any time. The trainer was available as a verbal communication behaviour during the action phase.
contact person for questions and interaction during the entire process The study uses a semi-standardised procedure, since it works with
and operated the PC. Two observers filled out observation sheets of pre-formulated criteria, but it also leaves room for the recording of
the communication partners A1 and A2 (participants) and B1 and new aspects. In addition, the video recordings were available for
B2 (trainer). In parallel, video recordings were made using a mobile comparison.
phone. Immediately after completion of the experiment, the trainer
completed a memory protocol C on their own perceptions.
4.3.4 Reflection sheet
4.3.2 Questionnaires
The reflection sheet C was filled in by the trainer directly after the
The questionnaire I (personal questions), was handed out to the par- execution. The questions were answered spontaneously and reflect
ticipants before the movement task was performed. The exclusion the first impression. The first questions refer to the execution of the
criteria for the selection of participants were checked and personal judo technique. From the perspective of the experienced judo trainer,
data was queried. Following to the integrative communication model the extent to which the technique is executed correctly was assessed
[27], potential influencing factors such as previous knowledge, mo- and the process of learning was also examined. Afterwards, the re-
tives, age, gender, etc. were taken into account. Volunteers were lationship between subject and trainer was described, especially its
asked about their ability to understand instruction manuals because subjective impact. Attention was paid to the application of additional
the judo technique is essentially developed step by step, similar to an help, when and why was this necessary, how help was given and
instruction manual. whether it was successful.
Questionnaire II (feedback) was given to the participants immedi-
ately after the practice task had been executed in order to record the
immediate experience. Questions were asked about the Judo tech- 4.4 Evaluation
nique, the instructional material and the general conditions. When
filling out the questionnaire, the test persons had the materials at
The evaluation was derived from the observers‘ notes, the video
their disposal. The feedback was intended to point out possible re-
recordings and the trainer‘s reflection sheet C. The questionnaires
structuring necessities for later follow-up studies. For example, the
filled in by the test persons supplemented the data collected. Sim-
test runs performed showed that some changes in the design had to
ilarities and differences in the observations were interpreted and
be made in order to achieve clarity. Also the detailed demand for pre-
analysed. In this way the observations on handling of material, the
vious knowledge of certain martial arts was reformulated into a more
method and the course of communication can be viewed from dif-
general question.
ferent perspectives. This is intended to achieve the highest possible
In addition, Questionnaire II asked for a self-assessment and in-
degree of coverage.
quires whether additional help was necessary both in terms of un-
When describing the course of communication, the verbal and
derstanding the material and performing the movement. Implicit in
non-verbal remarks were recorded. The focus here is on the ques-
the answers given is a distinction between whether the respondent
tions of when, what and how it was communicated. The verbal com-
asked for help of their own accord or whether the trainer intervened
ments are clearly recognizable and can be written down. The non-
proactively. Since the trainer is an essential part of the exercise, their
verbal communication results from the context and the behavioural
behaviour was described from the test person‘s perspective. In Ques-
elements. Every “additional communication” was recorded. First of
tionnaire II, the test person also gave a self-assessment of the degree
all, a time stamp is noted, i.e. when the communication took place. In
of difficulty and whether they needed help with the exercise.
a second step the cause was examined. This resulted in the following
areas for the evaluation:
4.3.3 Observation sheets
The behaviour of the participant and of the trainer was recorded in 1. general personal data for the classification of the test subject
separate observation sheets A and B. There is one observation sheet 2. recording of personality and behavioural characteristics (situa-
each for the acquisition phase (A1 and B1) and a second for the ac- tional and context-related)
tion phase (A2 and B2). 3. situation/atmosphere
The observation sheets were pre-formulated according to selected 4. time, an average value is calculated
criteria (verbal language, gestures, facial expressions, movement) 5. information part: handling of the materials/method
and serve as an aid for the observer. They follow the chronologi- 6. linguistic comments
cal sequence and repeat the fields of observation for the individual 7. body language
sections in the same way. The pre-formulated fields of observation 8. action part
should enable the observer to note many aspects in as short a time as
possible. There is room for individual remarks so that the observers Each test subject was described individually. The self-reported
can record unforeseen events. Nevertheless, the pre-formulated as- aspects and the observed behaviour were related to observed non-
pects ensure a structured approach, especially for later evaluation. verbal communication behaviour. Hypotheses could then be formed
The action part was mainly recorded using observation sheets A2 as to whether the non-verbal additional communication was due to
and B2. In A2 the observer recorded descriptions in general form the inter-personal differences, the material, or the situation. The self-
for implementation of the movement. In addition, aspects about the assessments were always related to the observed data.
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Eleventh International Workshop Modelling and Reasoning in Context (MRC) @ECAI 2020 5
5 First observations from trial runs orities themselves and this added discrepancies in the evaluations. It
is difficult to counter this effect, but it may be necessary to improve
Two trial runs were carried out and these test runs were intended to the training cycle for the observers. After all, ethnographic recording
familiarise the observers and the trainer with the use of the observa- is a skill that itself requires a lot of practice. While the observation
tion sheets and with the flow of the test. can not be considered representative due to the small number of test
Overall, it was found that the planned procedure was reasonable subjects (10), the pre-study provided crucial learning for the revision
and practicable. Materials offered proved to be sufficient for the par- of the larger study and it was also possible to obtain results that were
ticipants. The test subjects were able to understand them and imple- useful and indicative in nature.
ment the motions correctly. In the end, the participants were able to In the test runs of the designed study, non-verbal behaviour of the
successfully perform the judo technique. They were satisfied with test subjects was transcribed in addition to verbal comments. Test
their results and considered this learning path an acceptable alterna- subjects showed different communication needs, which could be read
tive to classical Judo training. from behavioural cues. The non-verbal behaviour could also be seen
Additionally, the multimodal explanation strategy, the decomposi- as expressions of inter-personal difference. Thus, for example, un-
tion of the movement sequence to be learned into sequences, and the certainties that are shown through behaviour could be reacted to ac-
alternation of acquisition and action phases, has proven to be useful. cordingly.
Participants confirmed that they felt comfortable and enjoyed it. Even although participants worked independently with the training
This indicates an overall relaxed atmosphere. After the introduction material and did not verbally ask for help, situations were identified
in the first action part, participants wanted to perform the whole in which they signalled a need to communicate, e.g. by eye contact
movement sequence immediately. or waiting [31]. It is helpful for explanatory systems, be they human
The trainer had to intervene at this point and point out that only or technological, to react to this behaviour.
the sequences shown should be practised. Here, the instructions by The prevalence and diversity of situations where eye contact
the trainer had to be optimized so that the sequence would be clearer. played a crucial role in the interaction is indicative that a richer model
Participants had to be encouraged to switch to the first action phase. of gaze might be beneficial for upcoming studies, particularly since
Hesitation was signalled by eye contact and by waiting, indicating gaze is a feature that can readily be captured by existing sensors.
that the test subjects needed some form of interactive response. This In the initial development of a situation where cooperation of mul-
despite the fact that transition from acquisition to action is signalled tiple entities is central (collaborative or team work situations), the
in the training material in such a way that execution could in principle need for additional communication is higher, so that the process and
take place without any intervention by the trainer. the procedure are understood and mutual trust is created.
The need for interactive response could indicate that there is a spe- In failure situations where corrective action and explanations are
cific need for communication and information, especially in the ini- necessary, an appropriate communication strategy that includes mul-
tial phase of becoming familiar with the learning path. Although the timodal feedback should be developed so that users do not give up
explanatory material and the trainer‘s presentations contain a lot of in frustration or fail to recognize the error at all. An incidental find-
relevant information, this did not seem to be sufficient for the partic- ing is that it appears from this experiment that impending frustration
ipants during the acquisition phase. and possible abandonment of learning can be predicted from the be-
In contrast, the need for eye contact during the action phase is haviour before it occurs so that an intervention might be possible.
likely a result of the setup, as the technique is a partner task and it is This is consistent with findings in other work we have been done on
necessary to respond to each other. So the search for eye contact can multimodal markers of importance [7].
here be interpreted as coordination during the execution. The trainer It was very clear from the experiment that explanations should
reported that participants tend to react affirmatively to the search for be offered multimodally and, depending on the subject, also sequen-
eye contact. Furthermore, eye contact was a frequently occurring be- tially. Repetitions and some redundancy, if necessary also in varia-
havioural signal. tions, help participants to habituate to working methods and provide
The evaluation of the different body signals, which of course are security and ultimately build a trust relationship.
to be understood contextually, already suggest that the trainer should Within a human-centred approach to intelligent systems develop-
respond adaptively to different test subjects. For example, test sub- ment, the better a system knows its user, the better it can potentially
jects that are very cautious and reserved in the execution phase would respond to them and their individual needs. The experiment showed
need encouragement in the action phase for a more courageous and that by taking behavioural elements into account, it is possible to get
powerful execution. It is important to note that this encouragement to know the user or participant better. From the recognition of the in-
can be shared using non-verbal cues. dividual needs for assistance, appropriate communication strategies
can be designed.
6 Conclusions and Further Work
References
The goal of this experiment was two fold: to test the experimental
protocol for the larger study and to see what aspects of human-to- [1] Renee L. Allen and Andrew S. Davis, Hawthorne Effect, 731–732,
human interaction might be useful for designing and developing for Springer US, Boston, MA, 2011.
[2] Michael Argyle, Adrian Furnham, Jean Ann Graham, et al., Social sit-
human-to-machine interaction, particularly for explainable AI and uations, Cambridge University Press, 1981.
training systems for high risk environments. In terms of testing the [3] Thomas Bøgevald Bjørnsten and Mette-Marie Zacher Sørensen, ‘Un-
protocol, a number of aspects of the pre-study are being revised for certainties of facial emotion recognition technologies and the automa-
the larger study. It should be noted that the pre-study showed defi- tion of emotional labour’, Digital Creativity, 28(4), 297–307, (2017).
[4] Richard A. Bolt, ‘“put-that-there”: Voice and gesture at the graphics in-
ciencies in our experiment protocol that will be rectified. Individual terface’, in Proceedings of the 7th Annual Conference on Computer
behaviour of the participants could not be fully recorded and tran- Graphics and Interactive Techniques, SIGGRAPH ’80, p. 262–270,
scribed. The observers also made individual judgements and set pri- New York, NY, USA, (1980). Association for Computing Machinery.
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Eleventh International Workshop Modelling and Reasoning in Context (MRC) @ECAI 2020 6
[5] Luka Bradeško and Dunja Mladenić, ‘A survey of chatbot systems [24] Karl M Newell, ‘Motor skill acquisition’, Annual review of psychology,
through a loebner prize competition’, in Proceedings of Slovenian Lan- 42(1), 213–237, (1991).
guage Technologies Society Eighth Conference of Language Technolo- [25] Virginia P Richmond, James C McCroskey, and Mark Hickson, Non-
gies, pp. 34–37, (2012). verbal behavior in interpersonal relations, Allyn & Bacon, 2008.
[6] Stuart Card, Thomas P. Moran, and Allen Newell, ‘The model human [26] Ulrike Rockmann, ‘Bewegungen verstehen und beherrschen’, Vom Ler-
processor - an engineering model of human performance’, Handbook nen zum Lehren: Lern-und Lehrforschung für die Weiterbildung, 159,
of perception and human performance., 2(45–1), (1986). (2006).
[7] Jörg Cassens and Rebekah Wegener, ‘Supporting students through no- [27] Jessica Röhner and Astrid Schütz, Psychologie der Kommunikation,
tifications about importance in academic lectures’, in Proceedings of Springer-Verlag, 2015.
AmI 2018 – International Joint Conference on Ambient Intelligence, [28] Gabriele Rosenthal. Interpretative sozialforschung. eine einführung.
eds., Achilleas Kameas and Kostas Stathis, volume LNCS, pp. 227– juventa, 2005.
232, Larnaca, Cyprus, (November 2018). Springer. ISBN: 978-3-030- [29] N. Santiano, L. Young, L.S. Baramy, R. Cabrera, E. May, Rebekah We-
03061-2. gener, David Butt, M. Parr, and Clinical Analysis Group, ‘The impact
[8] Jörg Cassens and Rebekah Wegener, ‘Ambient explanations: Ambient of the medical emergency team on the resuscitation practice of critical
intelligence and explainable ai’, in Proceedings of AmI 2019 – Eu- care nurses’, BMJ Quality and Safety, 20(2), 115–120, (2011).
ropean Conference on Ambient Intelligence, eds., Ioannis Chatzigian- [30] Helmar Schöne, ‘Participant observation in political science: Method-
nakis, Boris De Ruyter, and Irene Mavrommati, volume LNCS, Rome, ological reflection and field report’, in Forum Qualitative Sozial-
Italy, (November 2019). Springer. forschung/Forum: Qualitative Social Research, volume 4, (2003).
[9] Ligia Cordovani and Daniel Cordovani, ‘A literature review on obser- [31] Beatrix Schönherr, Syntax-Prosodie-nonverbale Kommunikation: em-
vational learning for medical motor skills and anesthesia teaching’, Ad- pirische Untersuchungen zur Interaktion sprachlicher und parasprach-
vances in Health Sciences Education, 21(5), 1113–1121, (2016). licher Ausdrucksmittel im Gespräch, volume 182, Walter de Gruyter,
[10] Hannes Daxbacher, Klaus Hanelt, Roman Jäger, Klaus Keßler, Ulrich 2013.
Klocke, Ralf Lippmann, Rudi Mieth, Hans Müller-Deck, Jan Schröder, [32] Claude Elwood Shannon, The Mathematical Theory of Communication,
Mario Staller, Hans-Jürgen Ulbricht, and Franz Zeiser. Begleitmaterial by CE Shannon (and Recent Contributions to the Mathematical Theory
zum dan-prüfungsprogramm - ein nachschlagewerk zu verschiedenen of Communication), W. Weaver, University of illinois Press, 1949.
themen der dan-prüfungsordnung im deutschen judo bund e.v., 4. über- [33] Kommunikationspsychologie—Medienpsychologie, eds., Ulrike Six,
arbeitete auflage, 2011. [Online; Stand 28.06.2020]. Uli Gleich, and Roland Gimmler, Beltz, 2007.
[11] Uwe Flick, ‘Qualitative sozialforschung: Eine einführung (4. aufl., [34] Norbert Streitz, ‘Reconciling humans and technology: The role of
vollst. überarb. und erw. neuausg.)’, Rororo Rowohlts Enzyklopädie, ambient intelligence’, in Ambient Intelligence, eds., Andreas Braun,
55654, (2006). Reiner Wichert, and Antonio Maña, pp. 1–16, Cham, (2017). Springer
[12] Owen Hargie, ‘Interpersonal communication: A theoretical frame- International Publishing.
work.’, in The handbook of communication skills, 29–63, Psychology [35] Matthew Turk, ‘Multimodal interaction: A review’, Pattern Recogni-
Press, (1997). tion Letters, 36, 189–195, (2014).
[13] Avi Karni, Gundela Meyer, Christine Rey-Hipolito, Peter Jezzard, [36] Gabriella Vigliocco, Pamela Perniss, and David Vinson, ‘Language as a
Michelle M. Adams, Robert Turner, and Leslie G. Ungerleider, ‘The multimodal phenomenon: Implications for language learning, process-
acquisition of skilled motor performance: Fast and slow experience- ing and evolution’, Philosophical transactions of the Royal Society of
driven changes in primary motor cortex’, Proceedings of the National London. Series B, Biological sciences, 369, (09 2014).
Academy of Sciences, 95(3), 861–868, (1998). [37] Paul Watzlawick, ‘Die axiome von paul watzlawick’, Unter:
[14] Athanasios Katsamanis, Vassilis Pitsikalis, Stavros Theodorakis, and http://www. paulwatzlawick. de/axiome. html [26.06. 2011] S, 38, 60,
Petros Maragos, Multimodal Gesture Recognition, 449–487, Associa- (2016).
tion for Computing Machinery and Morgan & Claypool, 2017. [38] Joseph Weizenbaum, ‘Eliza—a computer program for the study of nat-
[15] Naz Kaya and Feyzan Erkip, ‘Invasion of personal space under the con- ural language communication between man and machine’, Communi-
dition of short-term crowding: A case study on an automatic teller ma- cations of the ACM, 9(1), 36–45, (1966).
chine’, Journal of Environmental Psychology, 19(2), 183–189, (1999). [39] Dorsch Lexikon der Psychologie, ed., Markus Antonius Wirtz, Verlag
[16] Anders Kofod-Petersen and Jörg Cassens, ‘Explanations and context in Hans Huber, 2020. [Online; Stand 13.06.2020].
ambient intelligent systems’, in Modeling and Using Context – CON- [40] Gabriele Wulf, Charles Shea, and Rebecca Lewthwaite, ‘Motor skill
TEXT 2007, eds., Boicho Kokinov, Daniel C. Richardson, Thomas R. learning and performance: a review of influential factors’, Medical ed-
Roth-Berghofer, and Laure Vieu, volume 4635 of LNCS, pp. 303–316, ucation, 44(1), 75–84, (2010).
Roskilde, Denmark, (2007). Springer. [41] Tang Yuan Yan, Wang Patrick SP, and Yuen Pong Chi, Multimodal in-
[17] Anders Kofod-Petersen, Rebekah Wegener, and Jörg Cassens, ‘Closed terface for human-machine communication, volume 48, World Scien-
doors – modelling intention in behavioural interfaces’, in Proceed- tific, 2002.
ings of the Norwegian Artificial Intelligence Society Symposium (NAIS [42] Fabio Massimo Zanzotto, ‘Human-in-the-loop artificial intelligence’,
2009), eds., Anders Kofod-Petersen, Helge Langseth, and Odd Erik Journal of Artificial Intelligence Research, 64, 243–252, (2019).
Gundersen, pp. 93–102, Trondheim, Norway, (November 2009). Tapir
Akademiske Forlag. ISBN: 978-8-2519-2519-8.
[18] Stefan Kopp, ‘Giving interaction a hand: Deep models of co-speech
gesture in multimodal systems’, in Proceedings of the 15th ACM on
International Conference on Multimodal Interaction, ICMI ’13, p.
245–246, New York, NY, USA, (2013). Association for Computing Ma-
chinery.
[19] Karl-Friedrich Kraiss, ‘Der mensch als kognitive systemkomponente’,
in Fahrzeug-und Prozeßführung, 26–70, Springer, (1985).
[20] Robert M Krauss and Susan R Fussell, ‘Social psychological models of
interpersonal communication’, Social psychology: Handbook of basic
principles, 655–701, (1996).
[21] Danielle E Levac, Meghan E Huber, and Dagmar Sternad, ‘Learning
and transfer of complex motor skills in virtual reality: a perspective
review’, Journal of NeuroEngineering and Rehabilitation, 16(1), 121,
(2019).
[22] Yaniv Leviathan and Yossi Matias, ‘Google duplex: an ai system for
accomplishing real-world tasks over the phone’, Google AI blog, 8,
(2018).
[23] Philipp Mayring, Einführung in die qualitative Sozialforschung, Beltz
Verlag, 2002.
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).