Industrial and neutral environments bring multiple challenges in case of data analysis and cooperation while working with data visualization. This exploratory study aims to provide insights into user-3D data visualization interaction using Microsoft HoloLens 2 headset. The results were obtained in a 20-person study in a neutral environment - a classroom in a university building, and in a semi-industrial process tomography laboratory. Based on thematic analysis, this study presents initial guidelines for AR applications for working with 3D data visualizations.
Technological progress significantly influences the development of systems supporting access to
information and data visualisation. Augmented reality (AR) fulfills the function of supporting activities
performed physically by providing the user with a visual perception beyond the real world [
Over time, awareness, and the need to use gestures in AR has evolved, and research has been
developed to show which gestures are intuitive, in what cases do gestures work, and what other solutions
should be considered [
The dataset used in the prototype came from an electrical capacitance tomography (ECT) system. This industrial data visualization technique allows one to take a picture of the industrial processes in real time, which is essential for monitoring the process and understanding the phenomena inside an industrial silo. In the presented work, 3D tomography data is a visualization of granular material distribution in silo during process discharging. The observations and interviews were conducted during 20 experiment sessions, each with a different participant. They were not selected based on specific characteristics or personality types and were mostly recruited from university students at Lodz University of Technology. Data on gender, age, and prior experience with AR and ECT process tomography was collected from each participant through a demographic questionnaire.
Fifteen experiments were conducted in a neutral environment – a classroom in a university building.
In the remaining ones, participants would be observed using the AR system in the Process Tomography
Laboratory (TomoLab), a semi-industrial environment in a real silo used in the ECT process
tomography experiments [
Four tasks were planned for the study participants. Visual representation of the conception of implementation of tasks is shown in Figure 1, and more detailed descriptions are given in Tab. 1. Study participants were encouraged to speak out loud about what and how they wanted to perform, which allowed researchers to help them and gather more data by making notes of their comments.
The semi-structured interviews were recorded using a smartphone app and the observations were noted using pen and paper. Apart from that, each session was recorded from the point of view of the Microsoft HoloLens 2 headset using a built-in camera. The recordings were converted into transcripts for easier analysis using an automatic audio file to text software. Data collected from the three sources were then analysed using a thematic analysis approach. The first step was conducted by two researchers and consisted of going through all the written data (transcripts of sessions and interviews) and notes from the videos, highlighting similar phrases and ideas and giving names to those groups. This created two sets of codes which were then compiled into one set and finally, with input from all of the researchers, the codes were grouped into themes. The results of our analysis are presented in detail in section 3. This approach was used to highlight the most important issues presented in the interviews and actions of the participants and to draw conclusions directly from the user.
To better understand the study's results, it is important to know the participants' experience level with using the Augmented Based on the collected surveys, the participant group is 70% occasional users of virtual reality headsets, and 30% had never used such technology. Additionally, none of them had any prior knowledge of process tomography. Most of them, 95%, are aged between 18 and 35 years old, and most are students. After observations and interviews, we were able to categorize the data, using the thematic analysis method and identified several distinct themes. The observations identified are described below, categorized either by the individual or group theme.
With n=20 participants overall there were numerous different attitudes towards using gestures and generally using the app as well as strategies that the participant adopted in using it. Five participants expressed that they felt a disconnect between their expectations and the way the app was working and this led to discomfort and some frustrations. In some cases, users tried to use solutions from different apps or devices, for example, a mobile and VR game “Fruit Ninja” or comparing the gestures to using a touchpad on a computer.
Most users tried the same gesture multiple times as the system feedback was not easily understood by the user. Participants were not sure if their inconsistent results were caused by the problems of the software and hardware, or they were not good at using the application. There were two competing attitudes between participants to tasks involving moving and rotating the 3D cylinder. The first one was treating the immaterial 3D object in AR as a physical object and trying to come closer and touch it with open palms. This was especially noticeable when users tried different gestures for rotating. The second approach was treating the cylinder similar to a 2D object on a touchscreen and utilized gestures similar to those from a touchscreen. They would try to move by pointing at it or making a one-handed gesture.
A crucial theme present both in the interviews and actions of the participants was the role of physical and spatial comfort understood as providing a comfortable environment that would enable the user to focus on the data.
Users reported discomfort connected with wearing the headset. Even though this is something HoloLens addresses with the option to change the tightness of the grip in the headset, this mild frustration was present in the interviews after the session. Apart from this issue there were reports of problems with field of view, which was not big enough for some participants. This required them to move their head around and was especially problematic if the cylinder was placed high above the person or was in the full-size version of the silo in TomoLab.
Moreover, a topic found in the interviews and recordings from HoloLens was the importance of having enough space around the user and the placement of 3D data relative to the environment and the user. Some users would come close to the cylinder and move around it to look at all the sides while some preferred to move the dataset closer to them. This second option was especially noticeable in the sessions conducted at TomoLab, because of the equipment present around and sharp edges of some of the silo parts.
The cutting gesture that was used in the HoloLens application required the user to move their hand through the augmented reality object. This proved difficult for some of the participants at first and even more problematic during the research sessions at TomoLab because of the size of the silo. On the other hand, some users instinctively wanted to come closer to the 3D object and touch it as if it was a physical thing even in tasks that did not require it.
One important aim of this study was to find out if users will find thinking of gestures to use, without any prior instructions from the app or the researcher. Generally, this was not difficult for them, as they reported thinking of gestures was “easy” and “natural”. Several participants compared the gestures, both those that they thought of and those that were implemented in the app, to doing something in “real life”. However, four users felt that finding the gesture set in the application would be impossible or near impossible without the guidance of the researcher. Two other users also expressed the need for guidance and were glad that they could redo the gesture after getting a hint.
The biggest sources of frustration for the users were the problems and limitations connected with tested software and the hardware HoloLens HMD. The inconsistencies in gesture detection limited some users from freely trying to complete the tasks. There were also problems with the visibility of menus that were supposed to be anchored to the users’ hands. Besides the already mentioned comments, an important issue was the field of view. The issues were not as often in the case of tests in a classroom as in the TomoLab, because the cylinder was bigger, and the users wanted to see it all at once, which was not possible in some cases. The importance of the system running smoothly is further underlined by the fact that participants mention software and hardware problems even when speaking about different aspects of the experience. In the first task (called Task 0 in the app) the user was asked to look at the cylinder, with the option to walk around it to look at the other side and familiarize with the 3D object. That is also when the researcher would ask the user about some of the information given during the introduction to process tomography given before the tasks. Most of the answer about the overall understanding of what is happening were correct, but there were some when it was not clear for the user. The participants were also asked to comment on the colours of the visualization, with most of them commenting on the “inverted” colour scheme – the red colour indicated a lower value. There were also some hardware problems as it was hard to see some differences between colours because of the semi-transparent look of the 3D object. In tasks that required cutting the object in half some of the users expressed the desire to see the line where the cut will happen before they make the gesture.
Notably, on average the physical and temporal demand scores were not the highest. Furthermore, performance was the highly self-esteemed and weighted category, which indicates that most users were preoccupied with their performance during the tasks. The frustration is also highly weighted, which indicates that for a lot of users the frustration was important, which supports the findings from the thematic analysis.
The scope of this study was to investigate the behaviour of a group of users while interacting
gesturally with a virtual 3D object containing data. We conducted a task-based study using an AR
prototype app on HoloLens HMD, in which specific, industrial tomography measurement data was
visualized as a 3D object. Several types of interaction as well as some problems, were revealed based
on qualitative data analysis. Even though this study used a specific configuration, some issues were
noticed or outliers among the data overall, the analysis of this data allowed us to see that the gestures
used to interact with the 3D object were naturally similar for most users and mostly corresponded to
the gestures available on the HoloLens, with some differences depending on the testing settings. To
verify the results, our application should be tested on a larger scale with more complex tasks, especially
based on temporal and spatio-temporal data [
The study aimed to identify convenient gestures to analyse and view data in an industrial environment. Whereas there are already natural gestures used in augmented reality systems, it is crucial to note the different approach when interacting with data in the AR space in comparison to traditional interfaces. Based on the quantitative and qualitative data obtained during the survey of 20 people, it can be concluded that users tend to use gestures resembling interaction with real objects. Furthermore, the gestures they performed were similar but not identical. Future findings will be used for the development of more complicated data analysis actions and process complex data in other branches.
This work was partially supported by the project “Multimodal system supporting remote collaborative work in industrial settings” SKN/SP/535708/2022 funded by the Polish Ministry of Education and Science within programme “Student Engineering Clubs Innovate”.