When trying to improve human-machine communication it can be helpful to better understand human thinking and behaviour. In some cases, it is not only feasible, but also helpful to transfer recognised communicative patterns to machine interaction. The benefits of multimodal interfaces have been explored for quite some time, arguably starting with the famous “put that there!” demonstration system [4], leading to a variety of theoretical works and application systems [14]. However, there is still a lot of work to be done before non-verbal elements of communication can challenge the predominant paradigms for human-computer interaction [18, 35]. We have previously worked on multimodal behaviour in specific contexts of interaction [17] and on explanation-aware systems [16] as well as a combination thereof [8]. In order to better understand which aspects of human-to-human communicative behaviour can (at least) be mimicked by computational systems, we perform empirical research with humans in this area. In this paper, we present a pre-study for an experimental setup that looks at human-to-human communicative behaviour during movement sequence learning. This will enable us to better understand the role of different features in explanatory behaviour. In the end, a better understanding of this behaviour will hopefully enable us to optimize human-machine interaction as it pertains to explainable AI and might aid the development of better training systems for learning complex motor skills in high risk environments such as surgery or emergency medicine.
An increasing number of tasks in all walks of life are being taken
on or supported at least partially by technology e.g. learning in high
risk environments like surgery, where learning a new complex motor
skill can be essential to saving life, but where learning by doing is
life threatening [
For the often implied sharing of tasks between humans and
machines to be effective, it is necessary that the exchange of
information between human and machine runs smoothly. While it has been
(and in some cases still is) common to model humans as information
processing systems [
Communication is so much more than spoken or written language.
Natural language is inherently multimodal in nature [
The underlying research program of which this paper is a part aims to investigate whether the consideration of behavioural and contextual elements can provide insights that can be used for the optimization of future explanation-aware systems. To this end, an experimental setup was developed in a pre-structured explanatory situation. In this experimental domain, the test subjects’ goal is to learn a behavioural sequence that is indicative of complex motor skill learning, in this instance a Judo technique. The aim is to design the instructional material in such a way that it is relatively self explanatory, making verbal communication superfluous. Non-verbal behaviours such as gestures, facial expressions and body movement are observed and the communicative behaviour is recorded as accurately and unobtrusively as possible. This allows for the analysis of the timing and potential motivation for additional communication and the consideration of how this might relate to contextual and individual factors. 2
Human-machine communication (HMC) refers to the mutual
Information exchange between human and machine [
By now, systems using spoken natural language and learned
models have become mainstream. Technologies such as Google Duplex,
Alexa (Amazon) and Siri (Apple) respond to questions and answer
them appropriately, even mimicking non-task oriented aspects of
human communication. For example, Google Duplex uses typical
human behaviours like a short pause for reflection between sentences
or uttering “hm” [
Turning to other modalities than spoken (or written) language,
modern sensor technology in principle opens up the potential for
4 Intuitive is used here in a cultural-historic sense and is not referring to an
assumed inherent property.
simulation of communication that is comparatively close to
humanto-human communication [
3
Interpersonal communication can be described by various linguistic,
semiotic or communication models [
Decisive for the unfolding of the communication process is the
respective context, especially personal and situational context. Besides
observable elements, non-visible activities determine
communication behaviour (communication rules, sympathy, tenor, prejudices).
Basic prerequisite for successful communication is the use of a
common repertoire of signs and symbols by the communication partners.
Nevertheless, misunderstandings and errors can occur when coding
and decoding a message. The overall course of events is influenced
by contextual and psychological factors. The objectives of a
communication, the response and feedback behaviour, and the mutual
perception also influence the course [
According to Watzlawick, humans will always communicate even
if they don’t intend to communicate [
In a random sample, test subjects are to learn a Judo technique, i.e. a complex motor skill in the form of a motion sequence. Instruction on how to perform this motion sequence is given via text, video, and photo sequences. The respective learning steps are evaluated when the motion sequence is enacted afterwards. The study consisted of two phases, a small pilot to test the experimental protocol, and the case study itself. For the remainder of this article, we will focus on the small pilot phase and the process leading up to the experiment.
In the preparatory phase, the focus lies on reflections on the method, the context of situation, the explanatory materials, and the evaluation strategy. Influencing factors and barriers which may complicate the course of communication are to be considered. These preliminary considerations are then evaluated in test runs checked and corrected. The trainer is part of the communication process and since the test subject and the trainer together determine the course of communication, the behaviour of the trainer has to be taken into consideration as well.
Participatory observation was chosen as the method of data
collection as this is a standard method of field research and thus offers
a point of comparison [
The multi-perspective data collection (trainer, respondent, observers) served to achieve comprehensive observations by relating data points to each other and allowing them to be corrected if necessary. In the run-up to the project, the aim was to take into account (and where possible control) all factors that could plausibly have an influence on the result and thus on the reliability of the data to be collected. For example, the context (place, time, atmosphere), the behaviour of the trainer and the observers, and the structuring of the execution was precisely defined. By pre-structuring the observation sheets, the focus of the observers was specifically directed to essential aspects in contrast to free wording (validity). Elements of movement, verbal expressions and observations on the execution of the Judo technique were recorded.
All observations were made with the same observers and in the same room. Those carrying out the observations kept an unobtrusive external appearance. The test persons were addressed randomly and did not have any personal relationship to the persons performing the observations. In order to achieve reliable results, the test persons had no prior knowledge or reservations. To ensure this, a preparatory questionnaire was used. Using teaching material that was produced specifically for the task, the observation can be repeated reliably. Since the trainer was also part of the exercise, various safeguards were put it place to ensure consistency over the course of the experiment. The behaviour of the trainer was precisely defined and was also checked by an observer. With the help of the reflection sheet, observations made in different runs could then be compared. 4.1.2
For the analysis of the non-verbal communication elements, learning
of a movement sequence was chosen. In contrast to a purely
cognitive learning situation, it can be clearly seen whether the respondent
has understood the given explanations by following the exercise in
action. The fact that understanding and learning has taken place can
be demonstrated by the action itself [
While the guidelines for the correct execution of Judo technique
by the German Judo Association (DJB) [
His personal experience that the exclusive use of simple statements
(verbalizations), pictures (visualizations of throwing techniques), or
even throwing descriptions in text form are not sufficient is consistent
with the literature [
Even if the underlying mechanics and movements are understood in principle, when learning a new complex motor skill it is not unusual to initially have difficulties in performing it correctly. If necessary, the technique should be explained again or shown repeatedly. The motor skill chosen for this study was the “joint lock” because it does not require any previous knowledge or additional equipment. With an arm joint lock it is important to fix the elbow joint of the partner and then overstretch it. 4.1.3
The instructional material was presented to the participants in digital form using the keynote presentation software. It consisted of 10 pages, 5 photos and 5 videos. It was designed to be self-explanatory so that the verbal communication components were reduced. The focus of the observation was on non-verbal behaviour and movement elements.
The sequence of movements to be learned was broken down into individual learning sequences, which are modelled on the normal training situation in Judo practice session. The acquisition phase was followed by an action phase, in which participants act and practise what they have learned. The training texts were written such that the participants were directly addressed and could identify more easily with their role. The texts were kept simple and were developed as an instruction manual. The written description of the movement sequences is supplemented with photos and videos. The photos show the current state or the initial situation and details. The video provides the movement sequence. All pages are structured identically to provide the participants with an easier orientation of the learning path. 4.2 4.2.1
The location of the study has an influence on the mood and
motivation of the participants [
of a student. Stress elements contained in this situation were
alleviated by the surrounding atmosphere. The behaviour of the trainer
played an important role in creating a pleasant and open atmosphere.
In order to enable the participants to act as relaxed and pressure-free
as possible, the appearance, clothing, language style, posture etc. of
both the trainer and the observers were prescribed before running the
experiments [
These considerations were confirmed in the test runs where all participants noted that they felt comfortable in the situation and even enjoyed it. 4.2.3
As the experiment aimed for a relaxed atmosphere as close to
everyday life as possible, observers were used for both external and
self-observation. They went directly into the setting, actively
participating and writing notes which are then evaluated. They were briefed
and trained beforehand [
In practice runs before the small pre-test, the observers were trained in the handling of the different observation sheets. It turned out that the observers were able to follow the practice runs well and that the pre-defined structure of the observation sheets was helpful. The overall impression and individual peculiarities could be easily recognized and recorded.
Nevertheless, some details were missed. In contrast to the planned setup, a recording device was deemed necessary in order to record verbal utterances instead of transcribing them on-the-fly. This was done by using a mobile phone during the later runs. According to the test subjects, this small, inconspicuous camera was not noticeable or even perceived as disturbing. 4.2.4
The test subjects were recruited directly and invited to participate after a short eligibility interview. The following selection criteria were established: • Age group 18 and above (legal adult). • Body height, approx. between 1.70-1.90m. • The potential participants should have an average physical fitness. – A movement exercise is carried out with the persons addressed in order to test their coordination and movement skills (opposite windmill arm movement). • Good German language skills are necessary, as texts must be read and understood. • No previous knowledge of Judo or wrestling, determined by means of a questionnaire. • Persons who do not wish to be touched or who do not agree with the general conditions of the experiment are also excluded. For organisational and technical reasons, the participants were recruited at the university campus in Hanover. The total of 10 participants were young adults. 4.3 4.3.1
The explanatory situation tested was a learning situation with clearly defined roles. The trainer is the instructor, the participant has the role The trainer invited the participants and gave an initial overview of the goals and progress of the experiment. The participants filled in questionnaires I (personal) before, and II (feedback) after the experiment. The explanatory material alternates between acquisition and action phases. The participant could scroll forward or backward and repeat individual parts at any time. The trainer was available as a contact person for questions and interaction during the entire process and operated the PC. Two observers filled out observation sheets of the communication partners A1 and A2 (participants) and B1 and B2 (trainer). In parallel, video recordings were made using a mobile phone. Immediately after completion of the experiment, the trainer completed a memory protocol C on their own perceptions. 4.3.2
The questionnaire I (personal questions), was handed out to the
participants before the movement task was performed. The exclusion
criteria for the selection of participants were checked and personal
data was queried. Following to the integrative communication model
[
Questionnaire II (feedback) was given to the participants immediately after the practice task had been executed in order to record the immediate experience. Questions were asked about the Judo technique, the instructional material and the general conditions. When filling out the questionnaire, the test persons had the materials at their disposal. The feedback was intended to point out possible restructuring necessities for later follow-up studies. For example, the test runs performed showed that some changes in the design had to be made in order to achieve clarity. Also the detailed demand for previous knowledge of certain martial arts was reformulated into a more general question.
In addition, Questionnaire II asked for a self-assessment and inquires whether additional help was necessary both in terms of understanding the material and performing the movement. Implicit in the answers given is a distinction between whether the respondent asked for help of their own accord or whether the trainer intervened proactively. Since the trainer is an essential part of the exercise, their behaviour was described from the test person‘s perspective. In Questionnaire II, the test person also gave a self-assessment of the degree of difficulty and whether they needed help with the exercise. 4.3.3
The behaviour of the participant and of the trainer was recorded in separate observation sheets A and B. There is one observation sheet each for the acquisition phase (A1 and B1) and a second for the action phase (A2 and B2).
The observation sheets were pre-formulated according to selected criteria (verbal language, gestures, facial expressions, movement) and serve as an aid for the observer. They follow the chronological sequence and repeat the fields of observation for the individual sections in the same way. The pre-formulated fields of observation should enable the observer to note many aspects in as short a time as possible. There is room for individual remarks so that the observers can record unforeseen events. Nevertheless, the pre-formulated aspects ensure a structured approach, especially for later evaluation.
The action part was mainly recorded using observation sheets A2 and B2. In A2 the observer recorded descriptions in general form for implementation of the movement. In addition, aspects about the transition from acquisition to practice were recorded. The focus here was on the manner of implementation, i.e. whether the test subject starts hesitantly or actively. The trainer describes the non-verbal or verbal communication behaviour during the action phase.
The study uses a semi-standardised procedure, since it works with pre-formulated criteria, but it also leaves room for the recording of new aspects. In addition, the video recordings were available for comparison. 4.3.4
The reflection sheet C was filled in by the trainer directly after the execution. The questions were answered spontaneously and reflect the first impression. The first questions refer to the execution of the judo technique. From the perspective of the experienced judo trainer, the extent to which the technique is executed correctly was assessed and the process of learning was also examined. Afterwards, the relationship between subject and trainer was described, especially its subjective impact. Attention was paid to the application of additional help, when and why was this necessary, how help was given and whether it was successful. 4.4
The evaluation was derived from the observers‘ notes, the video recordings and the trainer‘s reflection sheet C. The questionnaires filled in by the test persons supplemented the data collected. Similarities and differences in the observations were interpreted and analysed. In this way the observations on handling of material, the method and the course of communication can be viewed from different perspectives. This is intended to achieve the highest possible degree of coverage.
When describing the course of communication, the verbal and non-verbal remarks were recorded. The focus here is on the questions of when, what and how it was communicated. The verbal comments are clearly recognizable and can be written down. The nonverbal communication results from the context and the behavioural elements. Every “additional communication” was recorded. First of all, a time stamp is noted, i.e. when the communication took place. In a second step the cause was examined. This resulted in the following areas for the evaluation: 1. general personal data for the classification of the test subject 2. recording of personality and behavioural characteristics (situational and context-related) 3. situation/atmosphere 4. time, an average value is calculated 5. information part: handling of the materials/method 6. linguistic comments 7. body language 8. action part
Each test subject was described individually. The self-reported aspects and the observed behaviour were related to observed nonverbal communication behaviour. Hypotheses could then be formed as to whether the non-verbal additional communication was due to the inter-personal differences, the material, or the situation. The selfassessments were always related to the observed data.
Two trial runs were carried out and these test runs were intended to familiarise the observers and the trainer with the use of the observation sheets and with the flow of the test.
Overall, it was found that the planned procedure was reasonable and practicable. Materials offered proved to be sufficient for the participants. The test subjects were able to understand them and implement the motions correctly. In the end, the participants were able to successfully perform the judo technique. They were satisfied with their results and considered this learning path an acceptable alternative to classical Judo training.
Additionally, the multimodal explanation strategy, the decomposition of the movement sequence to be learned into sequences, and the alternation of acquisition and action phases, has proven to be useful.
Participants confirmed that they felt comfortable and enjoyed it. This indicates an overall relaxed atmosphere. After the introduction in the first action part, participants wanted to perform the whole movement sequence immediately.
The trainer had to intervene at this point and point out that only the sequences shown should be practised. Here, the instructions by the trainer had to be optimized so that the sequence would be clearer. Participants had to be encouraged to switch to the first action phase. Hesitation was signalled by eye contact and by waiting, indicating that the test subjects needed some form of interactive response. This despite the fact that transition from acquisition to action is signalled in the training material in such a way that execution could in principle take place without any intervention by the trainer.
The need for interactive response could indicate that there is a specific need for communication and information, especially in the initial phase of becoming familiar with the learning path. Although the explanatory material and the trainer‘s presentations contain a lot of relevant information, this did not seem to be sufficient for the participants during the acquisition phase.
In contrast, the need for eye contact during the action phase is likely a result of the setup, as the technique is a partner task and it is necessary to respond to each other. So the search for eye contact can here be interpreted as coordination during the execution. The trainer reported that participants tend to react affirmatively to the search for eye contact. Furthermore, eye contact was a frequently occurring behavioural signal.
The evaluation of the different body signals, which of course are to be understood contextually, already suggest that the trainer should respond adaptively to different test subjects. For example, test subjects that are very cautious and reserved in the execution phase would need encouragement in the action phase for a more courageous and powerful execution. It is important to note that this encouragement can be shared using non-verbal cues. 6
The goal of this experiment was two fold: to test the experimental protocol for the larger study and to see what aspects of human-tohuman interaction might be useful for designing and developing for human-to-machine interaction, particularly for explainable AI and training systems for high risk environments. In terms of testing the protocol, a number of aspects of the pre-study are being revised for the larger study. It should be noted that the pre-study showed deficiencies in our experiment protocol that will be rectified. Individual behaviour of the participants could not be fully recorded and transcribed. The observers also made individual judgements and set priorities themselves and this added discrepancies in the evaluations. It is difficult to counter this effect, but it may be necessary to improve the training cycle for the observers. After all, ethnographic recording is a skill that itself requires a lot of practice. While the observation can not be considered representative due to the small number of test subjects (10), the pre-study provided crucial learning for the revision of the larger study and it was also possible to obtain results that were useful and indicative in nature.
In the test runs of the designed study, non-verbal behaviour of the test subjects was transcribed in addition to verbal comments. Test subjects showed different communication needs, which could be read from behavioural cues. The non-verbal behaviour could also be seen as expressions of inter-personal difference. Thus, for example, uncertainties that are shown through behaviour could be reacted to accordingly.
Even although participants worked independently with the training
material and did not verbally ask for help, situations were identified
in which they signalled a need to communicate, e.g. by eye contact
or waiting [
The prevalence and diversity of situations where eye contact played a crucial role in the interaction is indicative that a richer model of gaze might be beneficial for upcoming studies, particularly since gaze is a feature that can readily be captured by existing sensors.
In the initial development of a situation where cooperation of multiple entities is central (collaborative or team work situations), the need for additional communication is higher, so that the process and the procedure are understood and mutual trust is created.
In failure situations where corrective action and explanations are
necessary, an appropriate communication strategy that includes
multimodal feedback should be developed so that users do not give up
in frustration or fail to recognize the error at all. An incidental
finding is that it appears from this experiment that impending frustration
and possible abandonment of learning can be predicted from the
behaviour before it occurs so that an intervention might be possible.
This is consistent with findings in other work we have been done on
multimodal markers of importance [
It was very clear from the experiment that explanations should be offered multimodally and, depending on the subject, also sequentially. Repetitions and some redundancy, if necessary also in variations, help participants to habituate to working methods and provide security and ultimately build a trust relationship.
Within a human-centred approach to intelligent systems development, the better a system knows its user, the better it can potentially respond to them and their individual needs. The experiment showed that by taking behavioural elements into account, it is possible to get to know the user or participant better. From the recognition of the individual needs for assistance, appropriate communication strategies can be designed.