The use of intelligent virtual agents in language learning has increased in recent years. Studies into several aspects of personalisation aiming to increase user engagement are an ongoing research topic with avatar personality being one such aspect. As a step towards our development of intelligent virtual avatars, we present two of our initial experiments to explore di erences in user interaction with two contrasting avatar personalities { P1: open-minded, friendly and sociable and P2: closed-o , curt and distant. Each user interacted with a single personality in a video-call setting and gave feedback on the interaction. Our expectations, that P1 would be rated more enjoyable and induce participants to talk more, were only partially con rmed. While P1 did induce longer conversations in the participants, we found that interactions with both personalities were enjoyed and that user perception of P1 and P2 di ered, but less than intended. Several possible causes for these results are discussed, and we outline impacts for follow on intelligent system design.
Learning a foreign language is often di cult for several reasons, the availability
of conversational partners to practice spoken interaction being one. Ideally, a
language learner would immerse themselves in the language by spending some
time in a foreign country and interacting with native speakers regularly. As
this opportunity is not open to every student, one alternative for conversational
practice is Computer Assisted Language Learning (CALL), speci cally, virtual
tutors: They are constantly available, nancially more accessible, have in nite
patience and can minimise the student's anxiety or embarrassment. A number of
these systems for conversational practice have already been examined, embedded
in video games, on their own or as part of a larger CALL system [
For these CALL applications, the automated tutor is usually embodied in some way, for example as a virtual avatar, to keep the student focused on the learning activities. In order to increase user engagement with a virtual avatar, Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) many avenues are being explored, among them customisation of physical characteristics or the use of gestures and facial expressions. Anything that distinguishes a virtual character could be useful. As such, assigning speci c personality traits to a virtual tutor, expressed through personal preferences, appearance and behaviour, and perhaps tailored to the student's own personality, may increase engagement. To investigate this hypothesis, we rst need to nd out whether there are any observable variations in the interaction and feedback when students are confronted with di erent tutor personalities. To this end, we designed an interactive experiment with two opposing tutor personalities to explore that question and a separate survey to validate the two personalities we designed.
Chatbots are increasingly used in areas of day-to-day life, e.g. as personal
assistants or in customer service. So far, they are often text-based as speech
integration is still an area of active research [
To embody an interaction partner, virtual humans or avatars are often used in
a CALL system. These can range from static images all the way to animated 3D
characters, possibly even embedded within a VR environment. Computer games,
for instance, can incorporate gami cation in the language learning process, which
is done frequently in CALL for student motivation and engagement. Some recent
examples of CALL chatbots and/or virtual avatars being used commercially or
in scienti c research are Mondly [
Personalities can be described through numerous models in psychology. One
widely used model to categorise di erent personalities by broad behavioural
traits is the OCEAN model. This empirical model identi es ve dimensions {
openness, conscientousness, extroversion, agreeableness and neuroticism { that
describe a person's character [
The in uence of personality on di erent aspects of language learning has
been studied extensively. However, more studies can be found focussing on the
student's personality [
In virtual characters on the other hand, various studies can be found
discussing personality, for instance the expression of personality traits in virtual
humans [
For our experiment, we chose English as the foreign language, due to the wealth of available speech synthesisers and recognisers as well as it being an easy language for which to nd participants.
To investigate whether the interaction with and perception of the virtual
tutor by language learners would change with di erent personality traits in the
tutor, we designed the main experiment so that the participant would talk to
a virtual avatar for a few minutes and then rate the avatar personality and
give feedback on the conversation itself. We used an expressive avatar [
Set up as a Wizard of Oz Study [
We designed two personalities to di er along three of the OCEAN model's ve dimensions. As our context is language teaching, we chose to focus on Agreeableness, Extroversion and Openness as these can be expressed best in a dialogue setting. Due to the avatar's limited range of facial expressions and a lack of emotionally charged situations in the interaction, we left out Neuroticism. Conscientousness was also not included as a personality trait, as it did not apply to the topics discussed and would be better expressed in actions rather than pure conversation. { Empathises with the partici- { Set in her own opinions, insistent pant on not changing them { Always friendly { Will express her opinions if they { Tries to understand dialogue di er from the dialogue partner partner's point of view { Formal and curt wording, lack{ Patient with the participant ing most ller words { Curious about other people's { Appears closed o
ideas and experiences { Dislikes new experiences, espe{ Ready to try new things cially travel { Interested in opinions and pref- { Not interested in new places or erences di erent from her own experiences { Smiling facial expression Audio-Visual { Looks straight at the viewer Features { Emphatic lip movements { Fast talking speed { Sad facial expression { Head slightly tilted, not looking
directly at viewer { Little emphasis in lip movements { Slow talking speed
As we only worked with one avatar, both characters have a few things in common: They have the same physical characteristics, are female and named Saoirse. The di erent personalities were designed to represent two extremes expressed by creating dialogue scripts where the avatar exhibited speci c personality traits. In addition, posture, facial expression and speech characteristics were tted to support the script (see Figure 1). As detailed in Table 1, Personality 1 (P1) represents the higher end of our 3 scales, being open, friendly and sociable, while Personality 2 (P2) exhibits low scores along the 3 dimensions, behaving in a more closed o , curt and distant manner.
Personality Veri cation Survey In order to verify the two personalities we
had designed, we ran an online survey where participants were shown a video
recording of each avatar in random order and asked to rate the avatar's
personality on a standardised personality test [
Main Experiment As in-person experiments with local participants and xed technical equipment were not feasible at the time of writing due to the 2020/2021 Coronavirus pandemic, an experiment website 2 was built using primarily ReactJS, NodeJS and WebRTC with separate views for the researcher and the participant (Figure 2). On signing up for the experiment, a participant was randomly assigned an avatar personality to interact with. Once a participant had signed in, they could start the experiment in their own time and were presented with an initial survey to collect some meta information (native language, age group, ...). Next, in the main part of the experiment, the participant entered a video chat with the avatar. On the experimenter's side, the software OBS Studio 3 was used to assemble the video clips of each personality into an OBS scene and replay them as needed in the video call via virtual camera. Each video call was recorded via screen recorder and saved locally. After the video call, the participant was presented with another questionnaire rating the avatar's personality and giving feedback on the interaction.
To assess the avatar's perceived personality after the interaction, we decided to use adjectives symbolising the 3 personality dimensions, with 4 items per dimension, 2 positive, 2 negative each (see Table 2). Participants were asked to rate on a 5-point scale how well each description t the avatar they had just talked to. Next to each adjective, a de nition of the word was available in case the user had not encountered the word before.
The participants of this experiment were adult English learners, 18 years or older with normal hearing, as the main mode of interaction was spoken dialogue.
We expected P1 to be more pleasant and enjoyable to converse with, which would show itself in markedly positive user feedback and high scores on the
3 https://obsproject.com/
J. Dobbriner et al.
Table 2. Descriptors used in the avatar's personality evaluation by the participant Dimension High Low Extroversion assertive shy
outgoing distant Agreeableness patient annoyed
caring demanding Openness curious bored
imaginative conservative avatar personality survey. In contrast, we anticipated low personality scores and for fewer participants to enjoy the interaction with P2. Regarding the dialogue, we also hypothesised that P1 freely sharing much of herself would animate participants to talk more, and therefore result in longer interactions than with P2. 4
Personality Veri cation Survey In this survey, with only native English speakers, we recruited 33 participants, 67% female and 33% male, aged between 18 and 54 years with 33% in the 18-24 and 48% in the 25-34 age groups.
Extroversion
Agreeableness
Openness
Conscientousness
Neuroticism 1.0 0.8 0.6 0.4 0.2 0.0 1 2 Personality 1 2 Personality 1 2 Personality 1 2 Personality 1 2 Personality
Our personality design was veri ed, as participants consistently rated P1 signi cantly higher than P2 on the Extroversion, Agreeableness and Openness scales just as we had intended. Figure 3 shows the box plots of these scores for both avatars on all 5 personality scales. For better readability, we normalised the scores between 0 and 1, i.e. 1 on the Extroversion scale would signify extreme extroversion while 0 is extreme introversion. A T-test at a signi cance threshold of 0:05 con rms these results with Openness (T (18) = 10:651, p = 1:857e 15) the most signi cant, followed by Extroversion (T (18) = 9:704, p = 3:889e 14) and Agreeableness (T (18) = 7:693, p = 1:620e 10). Even on the remaining two scales the di erences were signi cant, more so regarding Neuroticism (T (18) = 6:436, p = 3:001e 8) { P1 scored much lower with a median under 20% which translates to greater emotional stability compared to P2 { than Conscientousness (T (18) = 3:810, p = 3:404e 4) where both personalities were rated relatively high but P1 still received signi cantly higher scores, also shown in Figure 3.
In terms of preference, participant responses also con rmed our expectations with only a single participant indicating they would rather talk to P2 and the remaining 97% stating a preference for P1.
Main Experiment Over the course of this pilot experiment, 18 participants completed the main study, 44% male and 56% female, aged 18 - 45 years, predominately under 35, who had been learning English between 7 and 22 years (M = 12:5, SD = 4:33). A majority of 78% reported some experience with virtual characters or environments and the remaining 22% noted no experience at all. At 83%, most participants spoke German as their native language with another 11% Italian and 5.6% (one person) Chinese. Due to the automated random assignment of experiment groups { 1 for P1, 2 for P2 { and the even number of participants, there were 9 participants per avatar personality.
Going forward, it must be stated that with a sample size as low as this, any statistics computed on the collected data cannot be very robust and all results are to be taken as indicative.
Extroversion
Agreeableness Avatar personality scores To answer the question, whether participants were generally able to tell apart the di erent personalities they interacted with, let us rst look at the scores the avatar achieved in the 3 personality dimensions.
Figure 4 shows box plots of the 3 personality dimensions comparing both avatar personalities. At rst glance, the plots look mostly as expected, with P1 generally scoring higher than P2, but a closer look reveals a few unexpected results.
While P1 generally achieved higher scores, the box plots for Agreeableness (Figure 4, middle) stretch over a larger interval and overlap more than they di er. For Extroversion, while the 25th and 75th percentile are higher in the rst personality, the median is actually the same at 0:625 in both groups and P2 achieved scores to just under 0:7 which is surprising for an introverted character. The clearest distinction is found for Openness with far less overlap between the box plots (Figure 4, right).
A T-test at a signi cance threshold of 0:05 con rms these results as shown in Table 3, with Openness (T (18) = 2:166, p = 0:046) showing the only signi cant di erence between P1 and P2, whereas Extroversion (T (18) = 1:816, p = 0:088) is marginally signi cant and may prove distinct with more participants. Participants still rated P1 as slightly more agreeable, but the mean comparison in
Audio Analysis Aside from the direct personality survey, we also analysed the audio track recorded from each participant, computing speaking duration, interruptions and pauses within a participant's turn as well as between turns.
The total duration of the participant's interaction with the avatar varied signi cantly between avatar personalities: While P1 interactions lasted an average of 12:15 minutes (SD = 1:81, M in = 8:61, M ax = 14:25), P2 conversations were markedly shorter at 7:55 minutes on average (SD = 1:54, M in = 5:77, M ax = 10:26). However, this marked di erence was at least partly due to differences in the avatar script as the P1 script included overall longer utterances.
Since learners' speaking practice is the main goal of the application and to fairly measure how much the avatar induced the participant to talk, we compared the ratio of the the participant's total speaking time divided by the avatar's total speaking time for each conversation. The distributions of this ratio between the two groups are very di erent: P1 participants talked between 1:17 and 3:14 times as much as the avatar with the 25th percentile at 1:80 and the 75th percentile at 2.39, whereas participants in P2 range between talking just over half as much as the avatar's at 0:59 to 2:52 times as much, the latter of which is an outlier. A mean comparison of this measure (Table 3) has a signi cant result at a p value of 0:022, thus con rming our hypothesis that P1 would animate learners to talk more than P2.
Another useful measure are interruptions that inevitably occur during video chats due to network latency or misjudging when the other party is going to speak. An automated dialogue system especially may need to adjust to a user's individual response times and minimum pause duration for a speci c speaker to adequately mimic natural conversations. During the experiment, the number of involuntary interruptions by the avatar occurred independent of the avatar personality due to connectivity variations and human error of the researcher controlling the avatar, but the frequency of interruptions may in uence the participant's perception of the avatar's personality. As such, we counted the number of interruptions for each conversation and found no signi cant di erence between the personalities (see Table 3).
The pauses in the conversation present a further aspect of analysis that may be of interest, particularly the participant's reaction time, i.e. the interval until the participant speaks after an utterance from the avatar and the participant's pauses while speaking. Analysis of participant response times across groups reveals no real di erence between groups for both measures, the participant silences within a speaker's turn having a median around 1 second and the reaction times with a median of 2 seconds across groups.
Qualitative Feedback As a nal part of the experiment, participant feedback was collected to determine enjoyment of the interaction, positive and negative aspects noted by the participants, their attentiveness during the dialogue and suggestions for further topics to discuss with the avatar.
All participants replied with Yes when asked whether they had enjoyed the interaction. When we gauged the participants' attentiveness by having them recall what they talked about, each participant recalled the main topics of conversation (hobbies, travel and pets).
In a question about positive and negative aspects of the avatar's conversation, frequent positive aspects of P1 were: patient (repeating words or phrases), interested, asking questions/introducing topics, kind and interesting to talk to. Negative aspects included: interruptions/impatience and topic changes without answering a user question. Showing too little emotion and talking too fast were also brought up, but only once. For P2, the main positive aspect was a good understanding of the user with matching responses. Unexpectedly, considering the intended lower agreeableness level, the avatar being nice, polite and having a sense of humor was also mentioned. A possible explanation might be that P2, while frequently disagreeing with the participant, was never openly rude, and polite disagreement may even be perceived as truer understanding than constant agreement with the learner. Frequent negative aspects were that the avatar was sometimes hard to understand and was sti or unnatural to talk to. The avatar looking sad and not answering some questions was also sporadically noted.
Topic suggestions for further conversation with the avatar were numerous for both groups and several people explicitly stated they would like to talk to the avatar again. Notably for P2, the avatar's general sadness and fear was repeatedly suggested, demonstrating an urge to help the avatar develop a more positive attitude. 5
While in our validation survey both personalities were clearly distinguishable, the results of the main experiment show less of a perceived di erence between the two. Part of that may be explained by having native English speakers for the validation and language learners in the main experiment along with the di erent number of participants, the mode of delivery and di erent focus: In the survey, 33 participants watched a video with instructions to focus on personality and had a direct comparison of both avatars, whereas the main experiment was interactive, with 18 language learners more focused on understanding the avatar and formulating a response and they only saw one avatar.
Aside from that, certain implications of the online interaction likely also a ected speci c personality dimensions: The overall high extroversion scores for P2 could be due to the avatar being perceived as more assertive than intended in its aim to keep the conversation alive. For participants who tend to give short replies and not ask many questions themselves, the avatar's continued questions may be recognised as outgoing, leading to a higher extroversion score.
Agreeableness, on the other hand seems to be highly varied independent of avatar personality, and generally rather high, which is further reinforced in the feedback survey where the P1 is repeatedly noted as kind or friendly { as intended { but P2 is unexpectedly deemed nice and polite by two participants. In the script, agreeableness was incorporated implicitly by a generally more informal speaking style for P1 but not in explicit statements, which would likely be more easily picked up by native speakers. Additionally, the avatar's involuntary interruptions certainly had an impact on perceived patience that was also noted explicitly as a negative in the P1 feedback. Countering that, the willingness to repeat utterances which was present in both avatars as this is a learner's environment, would be perceived as higher patience. Both interruptions and repetitions therefore likely a ected the agreeableness score as noise across personalities. Participants' direct comments also indicated that P2 was perceived as sad and depressed rather than disagreeable.
With the avatar's questions in both personalities, the openness score, particularly curiosity was always rated at least neutral or higher and thus elevated the score across groups. However, the explicitly conservative statements of P2, along with more in-depth questions and requests for recommendations in P1 appear to be su cient to be perceived as distinct from each other.
With regards to the speaking time ratio, P1 appears to encourage the
participant to speak more than P2 does, as expected if the participant is mirroring
the avatar's conversational style [
Regarding participant feedback, the universal enjoyment of interacting with the avatar was group-independent and is likely at least in part due to the novelty of the experience in contrast to conventional teaching. Gathering feedback and recording student progress over an extended period of time would be required for more valid results. However, the positive and negative aspects of the personalities will help us improve them in further iterations of this project.
As a nal aspect of this study, it should be noted that our participants were all volunteers and therefore bringing a high baseline of openness and agreeableness to the interaction. If used e.g. as an extension to traditional language teaching in a classroom or even just in a larger group of learners, the acceptance of and engagement with a virtual language tutor like this would likely di er. 6
In this pilot study we designed and veri ed two di erent personalities for a virtual language tutor, exploring through an interactive experiment whether participants interacted di erently with it depending on the tutor's personality. While the personality di erences appeared distinct in our validation survey, in our main study we found that only one out of three personality dimensions, Openness, was perceived signi cantly di erently between both groups. Extroversion, a second dimension came close to the signi cance threshold and the third dimension, Agreeableness was widely distributed with relatively high scores in both groups. However, the participant's speaking time relative to the avatar as a fourth measure, was signi cantly higher in the personality designed to be more pleasant overall, thus matching our expectations, and qualitative feedback for the application itself was encouraging.
While this initial study has limitations, we see it as an important step in validating our approach to data-driven customisable assistive agents. Such agents require the synthesis of a number of aspects of arti cial intelligence, machine learning, and cognitive science, but in so doing provide us a very real and benecial application domain for intelligent systems development. In current research we are now beginning to put our ndings to work in prototype development. With respect to the speci c future work building on our activities here, we will focus on making the personality di erences more apparent during the interaction, automating and extending the study by building a chatbot for each personality and possibly adding more personalities as well as exploring measures to automatically adapt to the student.
This publication has emanated from research conducted with the nancial support of Science Foundation Ireland Centre for Research Training in DigitallyEnhanced Reality (D-REAL) under Grant number [18/CRT/6224]. For the purpose of Open Access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission