The Council of Coaches project has developed a platform that aims at providing tailored and personalized virtual coaching for ageing people to support them in improving their health and well-being. This paper presents the results of the user evaluations of the technical prototype that we conducted.
The initial prototype used for this evaluation implements a small dialogue manager that is able to steer the scripted
dialogue and control the possible user input and user interface [
(1) Francois (Diet Coach): proposed a healthy recipe based on the user’s dietary preferences that were collected during the interaction. (2) Olivia (Physical activity Coach): recommended the user go for a walk around the block once a day around meal time. (3) Emma (Social Coach): suggested the user to have a friend or a family member accompany them during the walk. (4) Carlos (Peer): provided supportive dialogue emphasizing the expertise of the coaches and the eficacy of their coaching.
We made use of the Godspeed questionnaire to measure the Animacy, Anthropomorphism, Likeability and the Perceived intelligence of our coaching agents. We did not utilise the Perceived safety questionnaire, as we did not expect the discussed subject to have a strong emotional impact with regards to anxiety, agitation, or surprise. Further, we modified the System Usability Scale questionnaire to suit our study. We removed the questions related to use of the product in terms of a new technology since we used a simple computer interface. However, we still retained some questions related to the ease of use. Finally, we asked two general open-ended questions about the participant’s opinion of the system and the agents to capture the overall impression, and to find out if they would recommend the system to others. 2.4
The experiment involved one user interaction that lasted approximately five minutes. The experiment was conducted in English. The scenario consisted of four coaches, where two coaches are ASAP agents and two Greta agents. Every participant interacted with the system individually. The participants were first asked to read the information letter and sign the informed consent form, as well as ask any questions they had. A brief description of the experiment and the tasks to be performed by the participant was explained. During the interaction the virtual coaches introduced themselves and provided an interactive coaching session on healthy weight management. After the interaction, the participants were asked to answer the questionnaire. 2.5
The prototype was setup at Sorbonne University and the University of Twente. We collected responses from 7 and 13 participants respectively. In total we had 20 participants, with 40% being female (n=8) and 60% being male (n=12). A total of 30% of the participants were below 55 years old while 70% were aged 55 years or older, thus falling within the primary target population for Council of Coaches. A total of 75% of the participants had never interacted with a virtual agent prior to this study. 2.6
The results indicate that participants rated the agents high on likability (m = 3.67) and perceived intelligence (m = 3.63), but did not have high score on anthropomorphism (m = 2.76) and animacy (m = 2.93). Since 70% of population (a) Study 1
(b) Study 2 were above 55 years, their expectations of agents were probably a bit technically unrealistic. Furthermore, we need to consider the fact that 75% of them had never interacted with a virtual agent before and their perception was probably influenced by media (films).
The opinion of participants about the system and agents was mixed. While several participants mentioned enjoying using the system, and felt comfortable, few participants remarked that the gaze behaviour of agents was not genuine. This could be related to the remarks about the interaction not being human enough yet. Thus, we aim to develop a model to improve the gaze behaviour of agents during the interaction.
The agents were particularly perceived to be very friendly (m = 4.05) and kind (m = 3.85). 70% of the participants thought that the system was well integrated. 55% of the participants indicated that they would use the system frequently and 50% felt confident using the system. However, 35% of the participants reported that they would need technical assistance. Even though the interface was simple (clicking buttons), the overall system might have looked complicated for the older participants. If it was just launching an app or a browser, the user might have not felt that they required assistance. Overall, 80% of the participants said that they would recommend the system to their friends and family. 3
Group cohesion is prominent when the main goal of the group is decision making or problem solving. Cohesion
describes the tendency of group members’ shared bond/attraction that drives the members to stay together and to want
to work together [
The main aim of a group behaviour model is to generate the non-verbal behaviours for all the agents present in the
discussion based on their roles i.e., listener or speaker as the conversation proceeds. In this work, the goal of this
component is to enable the agents to display cohesive group behaviour. In order to model group cohesion in multi-party
interactions, we first annotated non-verbal behaviours i. e., gaze behaviour, facial expressions, head movement and
laughter and perceived cohesion level on 2-min video segments of the Patient Consultation Corpus (PCC) [
The scenario consists of the same four coaches (virtual agents)(see Sec. 2.2) interacting with each other with a diferent
appearance. Three diferent dialogue samples were developed on weight loss, stress management and sleep [
We make use of two pre-study questionnaires and three post-study questionnaires. The pre-study questionnaires measure the Negative Attitude towards Robots Scale (NARS) adapted to virtual agents (4-items) and persuadability of the user (5-items). The post-questionnaire measures the cohesiveness (4-items), the credibility (3-items) and the persuasiveness (3-items) of the group. 3.4
The goal of the study was to understand the impact of cohesive group of agents. We have developed a between-group
study with two groups. The first group interacted with agents that display behaviours generated by a random behaviour
generator. The second group interact with agents that display cohesive group behaviours generated by our model. The
behaviours we focused on are gaze, smile and head nods. The agent appearance and dialogue content remain the same
for both the groups. Based on the results from our previous study on persuasiveness, we assigned the role of providing
advice to an older authoritative agent while the supportive coach was assigned to a younger peer coach [
The study was initially planned to take place in a laboratory setting. However due to the recent health regulations, respecting the social distancing, we have modified the experimental setup. Since the technical setup is not on-the-fly and requires several software platforms to be installed along with their licensing, asking the participants (age group above 50) would have been a challenging task. Therefore, we modified our setup to present pre-recorded videos to the participant to imitate interactions in real-time. We used a survey platform to generate the flow of the conversation, provide options for the user to provide their response and based on the response selected the next video to be played. 3.5
The participant read a general instruction form and provided their consent to take part in the study. The participants iflled in the pre-study questionnaire. An introductory video of a virtual agent was presented to familiarise the user to the virtual agent, their behavioural capabilities and the type of interaction. We then started the session where the user is asked to imagine a situation and then interact with the group of agents. The user is presented with a recording of the interaction and prompted for a response when required. Once the user selects an option, the video interaction continues. Once the interaction is complete, the user is notified and the post-study questionnaire is displayed and we collected basic demographic information. 3.6
The participants were recruited online using a survey hosting platform named Prolific. We had set three specifications to recruit participants, i.e., aged above 50, proficient in English and has been diagnosed with chronic disease. In total we had 32 participants taking part in our evaluation study where 10 participants were in the age group of 51-60 and 22 were in the age group above 60. 36% of the participants were male while 64% were female. 3.7
A one-way Anova for used for the analysis of the responses. The perceived level of cohesion was slightly higher for the condition using our model in comparison to random behaviour model for all the participants (n=30). However, the diference was not statistically significant (p > 0.05). We calculated the persuadability score of each participant and retained those with a score higher than three. In total we had 16 participants equally distributed between the two conditions who reported to be persuadable. Results indicate that the perceived level of cohesion was higher for the videos generated by our model (m=4.03) than the random behaviour model (m=3.53) and the diference was slightly significant (p=0.1). There was no statistically significant diference between the two conditions for the perceived level of trust. We computed the mean score of trust for only persuadable participants in both conditions. Even though the rating was higher for the condition using our model, the diference was not statistically significant. We further grouped the participants based on NARS questionnaire, and we did not find any significant results. Finally, the perceived level of persuasiveness was rated equally for both the conditions with no diference.
In this evaluation study we tried to measure the perceived level of cohesion and how this in turn aspects the trust in the agents and their persuasiveness. In order to to do this we designed an online evaluation study with two conditions. We used our model to generate cohesive behaviours for one condition and for the other we used a random behaviour model. We found there was no significant diference in the perceived level of cohesion for both the condition for all the participants. However, when we filtered out participants based on their persuadability score we found a slightly significant diference where participants found the condition using our model to be highly cohesive group of virtual agents. The study had to be done online with pre-recorded videos which hindered the quality of videos. Even though we tried our best to record high-quality videos, we are not sure whether the participants were able to watch them in the same setting. Since the diferences in a listener executing a smile or nod is very subtle the participants might have missed it. Also, the environmental conditions could afect the results which we were not able to control. Regarding the perceived persuasiveness, we found there was no significant diference. This could be attributed to the fact that we used the same dialogue content and agents for both the conditions and only the non-verbal behaviours were diferent. Some participants found the automatic text-to-speech generated audio to be very artificial which could have afected their rating. Overall, the participants found the study to be quite interesting and an enjoyable experience. We described two evaluation studies conducted in the context of the Council of Coaches project. The first study focuses on the evaluation of the technical prototype. The results indicate that participants rated the agents high on likability and perceived intelligence, however, the gaze model for group interaction of the agents need to be developed. Further, the users enjoyed the interaction and said they would recommend the system to others. Using the feed back received from this study, we developed a group behaviour model that handles cohesive non-verbal behaviour generation for the group of agents with focus on gaze, smile and head nods. Results indicated that the perceived level of cohesion was slightly higher than the random behaviour model. Overall, the participants found the study to be quite interesting and an enjoyable experience. In future, we aim to expand the cohesive behaviour generation model to include other non-verbal behaviours.
This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement number 769553. This result only reflects the authors’ views and the EU is not responsible for any use that may be made of the information it contains. We are grateful to our project collaborators at University of Dundee, University of Twente and Universitat Politècnica de València who were mainly responsible for development of the technical prototype.