Creation of a Hardware-Software Complex for the Transmission of Odor in Virtual Reality Environments* Artem Smolin1[0000-0001-7242-6471], Konstantin Malyshev2[0000-0002-8455-5876], Anna Spiri- donova3[0000-0003-0724-0405], Andrey Mironov4[0000-0001-5336-8800], Julia Konstantinova5[0000- 0001-8957-3566] 1,2,3,4,5 ITMO University, Saint-Petersburg, Russia smolin@itmo.ru, malyshevko@itmo.ru, spiranna@list.ru, mr_blum@mail.ru, konstantinova_yuliya@inbox.ru Abstract. One of the main tasks to enhance perception and deeper immersion in virtual reality environments is the use of different sensory channels. This article discusses the creation of a prototype device for the reproduction of smells, which is used in conjunction with virtual reality headsets. This device will allow the user to immerse in the virtual space more fully. Keywords: Virtual reality, Sensor channels, hardware and Software complex. 1 Introduction During the evolution mammals obtained the ability of the nervous system for multi- modal perception to combine different inputs of the sensory system, such as light, sound, smell and taste. This ability led to the enhanced detection or identification of a particular stimulus. A reflex caused by the stimulus is a stereotypical reaction of the mammalian organ- ism, to something that prompted this reflex. Based on this theory, it can be assumed that we can increase the degree of user’s involvement in the virtual reality environ- ments, by increasing the number of sensory inputs to enhance the stimulus. The sphere related to the degree of immersion of users in the virtual reality is now actively developing, a variety of technological devices are being created, ranging from various controllers to complex systems with climate change. All this allows to increase the degree of involvement of users into the virtual reality. Based on the above, within the framework of this work, it was decided to create a prototype of a smell reproduction device that can be connected to various virtual reality headsets and integrated into new or existing applications. * Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons Li- cense Attribution 4.0 International (CC BY 4.0). 2 2 The Use of different sensory channels in virtual reality environments At the moment, various technological prototypes are being created. They use differ-ent sensor channels in both unique and combined formats [1]. For example, experiments are being conducted on the effects on the taste buds of the person. The Nourished project is a VR simulator that can help people with allergies to per- ceive the full sensory experience of eating (recreating the full sensory experience of eating, without actual food). The main components of the development are: Oculus vir- tual reality headset, food detection devices, motion sensor and aroma diffusers (see Fig. 1). The system creates visual effects using virtual reality, as well as simulates sounds during meal. In this case, the user eats healthy food in the form of cubes printed on a 3D printer (algae, flour from shiitake mushrooms, etc.) [2]. Fig. 1. Project Nourished. Experiments on the symbiosis of smell and other ways of influencing perception be- gan in the 1960s, when watching a film was followed by special cards that a person could scrub and feel a certain smell. One of the developed commercial prototypes of the device for transmitting smells is multisensory VR mask Feelreal (see Fig. 2) [3]. The mask is attached to the viewer’s face and to one of the virtual reality headsets, and then a scent generator with seven removable cartridges releases odors into the nose area. 3 Feelreal SDK is attached to this method, which will allow developers to add various individual scents to their projects. In addition, using the special Feel Real Player, it is possible to use different flavors while watching movies [1]. Fig. 2. Multisensory VR mask Feelreal. 3 Creating a prototype device for transmitting smells in virtual reality environments For the first version of the prototype, experiments were conducted, on the basis of which we received data on changes in interest and involvement in the interaction pro- cess after the inclusion of another sensory channel. Before creating a prototype of this device, research was carried out, which resulted in a method that meets the following criteria:  The number of odors available for use in the system;  Control the strength and saturation of the smell;  Possibility of introducing additional solutions that affect the ability to perceive odors;  Possibility of combining odors to obtain a complex aroma;  Ability to easily integrate the game engine and prototype.  The research work was divided into three stages. At the first stage, methods for re- producing odors were investigated to enable the device user to fully evaluate it, and the following methods for the technical implementation of the prototype were also investigated and thought out:  Use of high-pressure cylinders;  Spraying scents and odors in a natural way;  Storing the smell in its natural form or using an analog (flavoring agent) and spraying with an odor injection system. 4 As a result, the third method was chosen, in which the spray was carried out using an odor injection system. The second stage of the study was the search for a technical solution to control the previously selected method of implementing odor reproduction. First of all, methods for controlling the injection of odors and regulating the amount of odors were investigated. As a result of this work, a large number of control methods were developed and the option of using a water pump and valves with a tap controlled by motors without gearboxes and tubes used in home aquariums was selected. After working on the next version of the prototype, a different method was chosen, due to the low reliability of the first one. The following technical solutions for the control of valve and pump were also inves- tigated:  Development of unique management Board;  Using a ready-made Arduino Board and writing unique firmware;  Using a ready-made solution, such as injecting water into the engine. As a result, the second method was chosen: using a ready-made Arduino-based Board and a raspberry Pi mini-computer as a web server to receive commands and execute them on the device. Fig.3 shows the final layout of the prototype, without interaction with the mini-com- puter. For this device the microcontroller Arduino Nano was chosen because of its simplic- ity and capabilities, as well as due to compatibility with other more powerful counter- parts. In addition, thanks to this microcontroller, it is possible to expand the function- ality without high costs. The control device was built on the ARM architecture, with the possibility of wireless connection methods. As a result of the undertaken work, we can highlight the following components, which should have interaction with the system and with each other:  Control device;  Microcontroller;  Engines;  Pump. On the Fig. 4 there is demonstrated the third iteration of the scheme, in which the chan- nels of exchange commands (black lines), power (red dotted lines) and the following components of the system are indicated:  The motor driver;  Pump driver;  Power supply;  Voltage converter;  Microcontroller;  Control device;  Pump; 5  Engine;  Valve. Fig. 3. First prototype diagram. Fig. 4. Device diagram. 6 The final version of the device diagram, which is shown in Fig. 5 represents a complete diagram of the interaction of the device components and external sources. Fig. 5. Component interaction scheme device and external sources. 4 Preparing the created prototype for operation To prepare the created prototype for operation, the following tasks were required:  PC setup;  Configuration of the control device;  Calibration of devices. To control and configure the controller and the control device from a personal com- puter, it was necessary to have the PuTTy program, which allowed connecting to the device via the Serial port and SSH. Also, to connect to the Serial port, a special con- verter was required, which was implemented as a USB device. When using Unix-like systems with a kernel version higher than 4.4, the driver for our controller was embedded in the kernel. To flash the Arduino controller, it was nec- essary to install the Arduino IDE software package for the required version of the op- erating system. 30 4.3.1.2 Since the control device is available without a pre-installed OS, we needed to use the Orange Pi portal [4] and find a suitable version of the operating system for us. In this case, it was Ubuntu Server, after using which we recorded the image on a MicroSD memory card using the Win32 Diskimager utility, which is distributed under the OpenSource license. After recording to the microSD card, it was required to connect it to the device via the SERIAL port, for this purpose, using connectors we joint the Converter in USB and outputs for the UART. 7 In the process of connecting at a speed of 115200 b / s, process of the first start appeared. Since the terms of reference already specified which language and libraries should be used to function the program code on the control device, the required pack- ages were installed:  NodeJS;  Git;  Python 3;  GCC;  Make. The last three points are required for successful Assembly of packages used in the Node Js project. The Serial package was built using the command: npm install -g serial. As a result of the first launch of the device, it was clear that the standard package manager that comes with NodeJS was not enough. It was decided to use the third-party package manager Yarn, which is installed using the command: npm install -g yarn. After successful installation, you it is possible to launch it in the folder with the already created project using the command: yarn install. For the voltage Converter to work, its initial calibration was required. For this pur- pose, a multimeter was used, which was switched to the voltage measurement mode. The input contacts were supplied with a voltage of 12V, the output was connected with the contacts of multimeter, and then the device was calibrated until the necessary 5V was obtained on the multimeter. 5 Final testing To test the developed prototype, there was chosen a VR location created by Borrowed Light Studios. It represents a journey into Vincent van Gogh's the Night Cafe. [5] To enhance the effect of immersion, the smells of coffee and brandy were used, those drinks were typical in the cafes of van Gogh's epoch (see Fig. 6). Further prospects for the modification of the project are to change the way the air supply is controlled; instead of engines and valves, it is supposed to use an electromag- netic nozzle, which will allow for a more accurate supply of odor and increase the speed of response to a request, since the reaction time to the task will be minimal. The new version of the prototype is supposed to be connected directly to the Oculus Quest helmet, using Bluetooth technology, thus abandoning the Pro-jutting device in the form of a web server. 8 Fig. 6. Prototype of the device with odor reproduction. 6 Conclusions Virtual reality technologies are actively used in society in all aspects of its existence: education, commerce, entertainment, etc. Nevertheless, there is a number of unsolved problems, one of which is the creation of a software and hardware complex for immersion in virtual reality environments with the full use of touch channels. As a result of the undertaken work, a prototype of a smell reproduction device for virtual reality environments was developed. The development of this device included the following stages:  Selection of technology of reproduction of smells;  Technical development of the device;  Selection of test applications;  Selection and preparation of smells. References 1. Papagiannis H. Augmented reality. Everything you wanted to know about future technology. – Moscow: Eksmo, 2019. – P. 123 9 2. Project Nourished, http://www.projectnourished.com/ 3. Multisensory VR mask Feelreal, https://feelreal.com/ 4. Orange Pi Portal, http://www.orangepi.org/downloadresources/ 5. The Night Cafe - An Immersive VR Tribute to Vincent van Gogh, 6. https://www.borrowedlightvr.com/the-night-cafe/