Successful communication between citizens and decision makers - eParticipation, despite progressing from dedicated solutions to modern, social media-based approaches has been facing many challenges. We argue that Virtual Reality technologies through its sense of presence and embodiment for discussion participants can help in alleviating some of the major obstacles hindering effective communication and collaboration. In this paper, we propose a novel approach to building AI models to support effective dialog implementation in VR. VR platforms potentially afford studies on user behavior without the overhead of complicated sensor infrastructure required for data collection. In particular, we propose machine-learning-based approach for predictive log analytics to identify behavioral patterns that support or obstruct effective collaboration in the context of structured dialog conversation. We discuss the applicability of the models to e-Participation and possible broader application of the models created. We also argue that VR-interaction-data-based models have the potentials to be transferable to managing and improving real-life interactions.
e-Participation can be defined as technology-mediated dialogue between citizens and
decision makers [
The emerging immersive social Virtual Reality platforms offer new means of immersive communication that promises to overcome many of the challenges hindering effective e-Participation dialogue by offering strong social presence and the sense of embodiment in virtual avatars resulting in more honest interactions and sense of community [12][13].
In this sense, we argue that the emerging Virtual Reality (VR) technologies, which offer simulated collaborative environments, also often referred to as the “telepresence” [14], thanks to high-interactivity, strong immersion and increased presence capabilities, that gets close to real experience [15], create new opportunities for e-Participation inclusive communications. Specifically, we look at implementation of more effective VR-driven e-Participation – VR-Participation through specific adoption of dialogue protocol that has been argued in the literature to support constructive engagement [16, 17]. In particular we investigate how the principles of dialogue defined by Bohm [17] and reframed with four principles by Isaacs [18]: 1) Listening, 2) Respecting , 3) Suspending and 4) Voicing can be supported by VR technology. Those well-defined principles can be mapped to specific behavioral sequences.
Therefore, in our investigation we propose a novel approach leveraging trained AI models for validating and supporting dialog principles implementation in VR. Specifically, using online VR platforms for user-behavior-learning, unlike in real-world setting gives us more discrete view and comprehensive data on user behavior without a need for complicated sensor infrastructures. We aim at extracting basic behavioral items – events directly from the VR system via provided APIs and analytical plugins. We are then detecting sequences of specific behavioral items by applying an approach adapted from Lag Sequential Analysis and label them by interacting with e-Participation users. We label them from the perspective of supporting each of the four dialogue principles. Finally, the detected behavioral patterns feed to specific Machine Learning architectures for automated classification and behavioral predictions. We envisage automated post-session reporting helping VR event hosts, as well as VR environments designers to optimize their e-Participation dialogue-driven communication setup. We also envisage possible real-time recommendations (derived from predictive power of AI) to the event hosts, as the interaction happens, to ensure successful sessions. We discuss the applicability of the models to e-Participation and possible broader application of the models created. We argue that VR-interaction-data-based models have potential to be transferable to managing real-life interactions.
The research question we attempt to address in this work is as follows: How to identify behavioral patterns in Virtual-Reality-based group-communication that benefit specific four dialogue principles, hence supporting effective e-Participation? In this work, we build on top of our past studies including theoretical and early empirical work on using Virtual Reality for e-Participation [12, 13, 34, 35]. Since there is paucity of studies on maximizing effective group communication in VR, in particular in the context of user behavior in e-Participation discussions in Immersive Virtual Environments (vr-Participation) we had to start our study by building relevant approach to studying discussion participant behavior in VR.
We investigated the literature on user behavior analysis and elicited relevant method for identifying behavioral sequences in group interaction – Lag sequential analysis (LSA). Since that manual or semi-automated classic method is time and human-resource consuming we investigated ways of providing more contemporary, fully automated and possibly real time analysis for vr-Participation sessions. We argue that at technical level our problem can be considered a predictive log analytics problem. Therefore, we investigated relevant Machine Learning architectures to support automated behavioral logs analysis based on event sequences as training set and live-feed data. 3 3.1
In our work, by Virtual Reality, commonly referred to as VR we consider totally immersive simulated environments leveraging Head Mounted Displays (HMD) and manipulators as interface, offering a form of strong telepresence and co-presence, where users are isolated from their surroundings as defined by Steuer et al. [14]. Contemporary authors in the domain of e-Participation [19–22] relate to word virtual in a very different sense to the concept considered in this paper. The authors refer to Virtual as digital platforms in general, in particular social media, while our definition is in line with the one relating to Virtual Worlds definition given by Bell et al. [23] presented as: A synchronous, persistent network of people, represented as avatars, facilitated by networked computers. The immersive Virtual Reality platforms, hosting Virtual Venues and Worlds introduce new quality to human-to-human interactions via digital medium through three basic affordances: 1) Sense of Immersion, 2) Sense of Presence & Embodiment, 3) Sense of Community. The strong sense of immersion can significantly improve VR group interaction capacity to listening and participation due to strong isolation from “real world” and focus only on matters in VR unlike in teleconferencing or social media solutions where participants use screens to interact and get easily distracted and carried away due to the “screen barrier” effect [24]. In this context, Bricken stresses on significant difference between viewing (on screen) and inclusion (in VR) by stating that in virtual reality users interact directly with various information forms in an inclusive environment. The strong sense of presence & embodiment of participants in Virtual Reality environments, discussed by computer scientists [14, 15] has been also strongly corroborated by cross-domain works linking the computer science domain and neuroscience [25]. The presence is understood here as a mental state in which user feels physically present within the computer mediated environment [26]. Finally, the premise of VR contributing to stronger community that benefits the participants has been also corroborated in literature [47].
Those three pillars of VR contributed to emergence of so-called VR platforms that allow group communication and collaboration in simulated environments. The most popular emergent Social VR platforms include AltspaceVR1 (by Microsoft) and VRChat2. Social VR platforms offer thematic events and dedicated spaces & environments for specific types of communities such as common interest groups (music, arts, developers) or support groups (like LGBTIQ). In principle, those platforms simulate real-life interaction in a virtual environment with acknowledgment of basic physics and natural dynamics such as distance, directional and distance-dependent propagation of sound, head and arms gestures and other non-verbal communication. In our previous publications [12, 13, 34, 35] we discussed the use and the benefits of adopting that emerging channels as new venue for e-Participation – VR-Participation. In this paper we argue that social VR platforms, in fact can be effectively used to study participant behavior and help supporting effective communication in Virtual environments. Moreover, due to resemblance of the VR interaction to face-to-face events, in terms of basic parameters, we claim possible transferability of the patterns to real world participatory interactions. 3.2
In our work we draw from established methods for behavioral pattern analysis in particular we fashion our approach after the lag sequential analysis (LSA) as a base approach for our study. LSA has been widely used to study group activities by analyzing behavioral sequences at the individual level [27–29]. LSA enables investigation and understanding of the sequential relationships between each participant’s behavior. That method was particularly popular in the past 40 years in the era of limited computing resources, where most of the work was done manually or semi-automatically.
Typically, in the preparation stage for classic LSA processing, researchers would record a video of each of the subjects while performing some arranged tasks and then coders would “transcribe” the video into specifically coded text with a number of behavioral coded items that would be sent for subsequent sequential analysis. The events can be defined as discrete interaction elements – behavioral items, for example: A1 - Subject gazes at other subject for certain specified amount of time A2 - Subject gazes at specific object for certain specified amount of time B1 - Subject uses specific hand-tool for certain specified amount of time C1 - Subject uses a whiteboard for certain specified amount of time
D1 - Subject checks her/his phone for certain specified amount of time 1 https://altvr.com/ 2 https://www.vrchat.net
As in the approach presented by [29] the encoded data is tested by computing Zscore statistics to produce adjusted residuals tables for the subjects’ behaviors. Then, in the built tables where the Z-scores are greater than specific, set threshold, and where their sequential relationships achieved significance (p < 0.05); these values are arranged to form the sequential patterns. Those patterns are then visualized with relevant directed graphs. The graphs are aggregating repeatable behavior in the context of the studied group in specific context. The more common the behavior the thicker the graph edge (arrow) symbolizing specific sequence. We present an example graph based on the proposed example behavioral items in Figure 1.
A2
B1 A1
C1
D1 Our approach implements the same stages of LSA analytics while benefiting from the power of modern, machine learning methods. Similarly, as in the case of LSA we first need to collect the data representing the behavioral items – events. In our case we will detect sequences of events that support specific dialogue principles. However, in our approach, we replace the legacy video recording stage with direct data collection via available VR system APIs and analytics plugins libraries. Since the APIs provide the coded events in a sequential manner (including timestamps), the classic transcribing & coding is redundant. We envisage one or two possible separate architectures for behavioral analysis. In case of two architectures, one will be used for classification of behavioral patterns and one for live-predicting user behavior. Unlabeled data, in principle, can be used immediately with the architecture supporting predictive analysis explained further in this document. However, we first envisage additional stage that includes labeling the specific sequences of behavioral items (events) as supporting or hindering the dialogue stages. Finally, the labeled data instead of creating relevant sequence tables (like in LSA) it is used to train the AI models for automated behavioral patterns classification. Since, the sequential data coming from APIs has a form of data log, in our approach, we follow the best practice coming from log analysis domain. In particular we draw upon the works on machine learning- based predictive business processes monitoring [30]. 3.3
At the technical level a common contemporary way to automation via machine learning is through the use of neural networks. In our study we intend to apply well-established architectures. Extensive experimentation should help us to select the most effective architecture for the task of identifying the behavioral items. We have been considering several different architectures that may fit the task. We would like to start with the convolutional neural network (CNN) as a possible option to classify the sequences of behavioral items accordingly to previously identified and trained patterns. CNNs has been used successfully to image classification [31] including image time series as well as in audio [32] classification. Other architecture considered is (considered the most commercially successful3) is LSTM – Long short-term memory introduced by [33] which is a type of RNN – Recurrent neural network that leverages feedback connections and provides good predicting capabilities for time series data. As we believe our problem can be considered a predictive log analytics problem and based on our revision of the models especially in the predictive business process monitoring [30] LSTM seems like the best performing candidate architecture. In particular since that the architecture is insensitive to gaps between the events (again similarly to LSA method) in a time series it makes it particularly suitable for analysis of user behavior in VR. The set of architectures applicable is not limited to the ones discussed and further research should help us to clarify on the best fitting solution. We may also try to use LSTM for the classification stage instead of CNN as a potential solution. 4
First, the trained models, using CNN (but possibly also LSTM) are expected to provide essential capability to provide post-session and real-time communication recommendations both for the event-hosts and vr-Participation virtual environments designers. The second approach leveraging LSTMs is aimed at live-predicting user behavior for helping moderation of the communication as the dialog unfolds.
Our novel approach encompasses entire pipeline from e-Participation event, that hosts a session in immersive VR environment, through data acquisition, training models and using AI for classification and prediction of user behavior. We present the general overview of our approach in Figure 2. 4.1
As mentioned in the background section, in our study we leverage lag sequential analysis - LSA as a template for generating our training data for classifying user behavior patterns. However, in our novel approach we replace the legacy video recording with direct data collection via available VR system APIs and available analytics libraries. In our study we leverage the existing VR collaborative (social VR) environments to provide an experimental venue and sensing infrastructure to detect sequences of behavioral actions that support dialogue principles. Specifically, since the Virtual Reality environments that are subject of the study are created using popular Unity framework, 3 https://www.bloomberg.com/news/features/2018-05-15/google-amazon-and-facebook-owe-j-rgen-schmidhuber-a-fortune we are going to use available libraries such as: Unity3D Google Analytics4 or MixPanel Unity implementation5.
Evaluates Session
VR API
VR
Environment Hosts Sessions
E-Participation
Events Stream
Identified Behavior (negative) Warning or Notification (positive)
Event Sequence Sequence Labeling
Log Training
LSTM Real-time Behavior Prediction
Sequence Classification
Training Labeled Behavioral Sequence
CNN Post-Session and Real-Time
Behavior
Classification Prediction
Identified Behavior Identified Behavior (negative)
Warning or Notification (positive) 4 https://docs.unity3d.com/Manual/AssetStoreAnalytics.html 5 https://developer.mixpanel.com/docs/unity Dependable on specific sequences, the same behavioral items can have positive or negative impact on communication. For instance, if User A performs L2 and T2 – gazing and talking to another participant and then U2 (looking at personal browser) occurs followed again by L2 and T2 that is beneficial as user is checking some facts in discussion. In fact, that sequence support the Voicing principle of the dialogue. If user A performs L1, L2, L3, L4 without any T action in repeated manner that supports the Suspending principle of the dialogue.
However, if user continuously does U2 with seldom L events, that is an indication of disconnect especially if none of the T (talking) events follows and is corroborated by W3 (walking away). The event sequences require to be put in positive and negative scenarios as exemplified that then are labeled for training the models. The predictive element does not require labeling. Instead the stream of events can be feed the LSTMs live to provide valuable predictions and recommendations to e-Participation session users. 4.2
As we elicit the required data via provided APIs and 3D analytics plugins, the specific action reports provided by the APIs are put in data logs. The generated behavioral sequences are verified and labeled accordingly to supporting specific dialogue principles based on interactions with discussion participants and by providing relevant questionnaires. Once the sequences are labelled, we send for further processing for training the models. 4.3
In our AI-driven analytics, we want to first investigate two different architectures CNNs and LSTMs. Harnessing the ability of CNN models to learn feature representations exclusively from raw data, we plan to have convolutions on the temporal dimension of user behavior time series. In order to exploit the interaction between users, convolutions will include data from all users at the time n.
Convolution on 15 sec User_A E_1
E_3 User_B E_6 E_9 User_C E_7
E_3
E_7 E_8 E_1
E_8 E_3 E_3 … … …
E_7 E_1 E_7
E_8 E_5 E_3
E_3 E_8 E_6
E_3 E_8 E_3 5s 10s 15s
time
Therefore as in the example presented in Figure 3, we are going to feed the CNN with time series for instance 5 minutes for 3 users with Convolution on 15s that means the size of the input will be 3x20. Those five minutes of activity will be classified as to whether they support specific dialogue principles. The detected sequences can be used live to inform the participants about specific occurrences and suggesting relevant actions. They can also serve as a post-session report for organizers for further analysis. LSTMs are a special type of RNNs that are able to learn long-term dependencies in sequences of data. In other words, they can remember information for long periods of time. We plan to harness this ability of LSTM’s to predict users’ next actions on demand, giving us the advantage to make relevant suggestions and adapt the environment, based on the predicted future. The LSTMS will be fed and live trained directly from the APIs without a need for prior training.
This type of networks learns from sequences of data. In our context, we will train them using sequences of live user’s behavior in VR. Each user sequence will have the following shape: [event_1, event_2, event_3, … , event_n] where n is the number of events.
Input: [event_2, event_2, event_4, event_3], time period = [0,3] Output for time n+1: event_8 The training and prediction are recursive therefore the perditions “evolve” as the session goes. 5
e-Participation improvement through approach proposed The direct benefit for next-gen e-Participation from the presented approach is the possibility to go beyond the common, ubiquitous, user-questionnaire-based participation performance analysis in favor of more structured and more discrete methods based on precise measurements in a controlled, simulated environment. Specifically, the identification and explicit classification of specific participant behaviors supporting dialogue principles are of invaluable importance to future vr-Participation stakeholders. Moreover, we believe that the results obtained through VR experimentation, made using our approach, may help not only to understand better the dynamics of emerging next-gen e-Participation as well as classic, online e-Participation but is potentially transferable to real-world participatory events. Therefore, we argue that this research may contribute significantly to general citizen-participation paradigm. 6
In this paper we presented an innovative approach adapting the machine learning-based predictive log analysis to monitoring and analysing participant behavior in e-Participation in Virtual Reality. In particular we provided a pipeline proposal for automated behavioral sequences analytics for identification of behaviors supporting four principles of dialogue. The presented approach draws from past methods such as LSA – lag sequential analysis and augments it with the new approach involving AI models training and automated behavior classification and behavior predictions. We argue that more structured, automated participant evaluation is pivotal for effective communication and e-Participation interaction improvement. We cannot claim absolute completeness of the approach provided and proposed set of methods and architectures is rather exploratory and should be expanded. Specifically, the preliminarily selected CNN and LSTM architectures can be replaced by other architectures as experiments are going to be performed. Moreover, the manually annotated data can show to be insufficient and we may focus entirely on unsupervised approach.
However, we argue that, considering the general paucity of solutions to evaluate eParticipation user behavior in Virtual Reality, the framework proposed creates a good base for grassroots experimentation. In particular we set the scene for experimental setting allowing not only improve e-Participation by reporting on the effectiveness of communication but also to provide real-time recommendations to participants to ensure constructive results. The major limitation of this work is the lack of relevant experimentation commenced with the “in-the-field” use of the proposed pipeline. Therefore, the future work requires relevant implementation and experimentation with the proposed approach. In particular we intend to organise vr-Participation sessions and work with the stakeholders towards evaluating the proposed pipeline in terms of its effectiveness to benefit the e-Participation. 7
In this paper we provided a short overview of the challenges hindering the effective eParticipation. We elaborated upon the emerging Virtual Reality driven e-Participation – vr-Participation as a potential solution to classic e-Participation issues. Most importantly, drawing from established methods such as LSA, we proposed a novel, structured approach to ensuring that the emerging vr-Participation provides better and more effective communication by introducing a user-feedback loop designed around machine learning-driven post-session and live user-behavior log analysis. We argue that, by explicitly addressing the need for support for the four dialogue principles, the proposed approach contributes towards ensuring improved communications in next-gen vr-Participation as well as possibly supporting existing e-Participation. Moreover, we believe that the behavioral patterns identified through our approach can be potentially transferred and benefit the face-to-face participatory interactions. 17. 18. 19. 23. 24. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35.