Thorough ventilation is one of the measures to prevent COVID-19 infection. CO2 concentration is frequently utilized as a ventilation guideline. In fact, o ce buildings, schools and factorys are increasingly introducing systems that display changes in CO2 concentration values. However, 2D map based visualizations are not enough to understand the progression of indoor air pollution and whether there is a need for ventilation. A 3-Dimensional (3D) measurement is needed to visualize CO2 concentration in space. However, conventional methods have various problems to achieve a 3D measurement, because a large number of sensors are needed to sense the entire room and explicit knowledge of their location. Therefore, an automated method to measure CO2 concentration in 3D is also proposed. We propose a three-dimensional (3D) visualization of CO2 concentration using a head mounted display (HMD). This indoor CO2 concentration method uses both xed sensors and mobile sensors with a CO2 gas sensor module. The visualization facilitates understanding of the temporal changes and spatial distribution of CO2 concentration. A prototype was developed using Microsoft HoloLens 2 as our Augmented Reality (AR) HMD; an iRobot Roomba 600 Series as our autonomous mobile robot at ground level; a William Mark Air Swimmer Shark as our airship robot to get measurements at higher positions; and a M5Stack Gray, a M5Stick C Plus, and TVOC/eCO2 gas sensor unit as CO2 gas sensor modules. Using the position coordinates and measured values of each sensor, a 3D distribution of CO2 concentration is automatically calculated and visualized using the AR-HMD.
most e ective infection control measures.
Clean indoor air is essential for human health. Indoor air CO2 concentration can be used as a standard for inaccounts for more than food and water intake and nearly door ventilation conditions and is advocated to be less 60% of the weight of materials consumed by humans in than 1000 [ppm] [7]. In addition, the magnitude of the a lifetime [1]. Indoor air quality has a signi cant impact risk of viral infection is correlated with the indoor CO2 on the human body. Proper ventilation behavior is im- concentration, and keeping the indoor CO2 concentraportant to maintain healthy and comfortable indoor air tion low is a good measure against infection [8]. In fact, quality. Inadequate ventilation causes high carbon diox- the use of CO2 concentration monitors is increasing in ide (CO2) concentrations, which have adverse e ects on spaces where many people gather, such as o ce buildthe human body, including headache, drowsiness, carbon ings, schools, and factories. However, 2D map based monoxide poisoning, and impaired concentration and visualizations are not enough to understand the progresmemory [2, 3, 4]. At the time of this writing, COVID-19 sion of indoor air pollution and whether there is a need is a worldwide pandemic, and one of the most common for ventilation. This is especially true for some gas such risk factors is “closed spaces with poor ventilation” [5]. as CO2 which is not perceivable by the human senses. COVID-19 spread is caused by inhalation of droplets [6]. Therefore, we propose a three-dimensional (3D) visualization of CO2 concentration using a head mounted display APMAR’22: The 14th Asia-Paci cWorkshop on Mixed and Augmented (HMD) to facilitate the understanding of CO2 concenReality, Dec. 02-03, 2022, Yokohama, Japan tration in space. Previously, Burgués et al. proposed a ⇤ Corresponding author. method to visualize the temporal variation of 3D gas dism.poetrsuuskqau. miaa@hios..onha2is@t.jips.n(Mai.stP.jepru(Msq.uOíat-sHukeran);ández); tributions generated by gas sources located at various isoyama@is.naist.jp (N. Isoyama); hideaki.uchiyama@is.naist.jp locations [9]. To measure the 3D gas distribution, they di(H. Uchiyama); kiyo@is.naist.jp (K. Kiyokawa) vided the room into 3D grids, and sensors were placed in 0000-0002-0486-1743 (M. Perusquía-Hernández); each grid. The results show that gas heavier than air does 0000-0002-6535-8439 (N. Isoyama); 0000-0002-6119-1184 not accumulate near the ground surface, and that the gas (H. Uchiyama); 0000-0003-2260-1707 (K. Kiyokawa)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License distribution changes drastically with time, regardless of CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) height. Additionally, Russel et al. proposed a method Autonomous Mobile Robot
Fixed CO2 Sensor Module
CO2 Sensor Module for measuring 3D gas distribution using a mobile robot Various methods have been proposed for indoor enviequipped with a telescopic sensor head [10]. The results ronmental measurement using mobile robots. For examshowed that the gas was concentrated near the ceiling, ple, Jin et al. used an autonomous mobile robot equipped even though the gas source was located on the oor. This with sensors for indoor environment monitoring [12]. supports the idea that the dispersion of gas heavier than However, it does not account for inaccessible areas to air is in uenced by convection. This indicates that 3D the autonomous mobile robot. Therefore, we propose a measurement is necessary even for CO2, which is heavier measurement method that combines xed sensors and than air. However, conventional methods require man- mobile robots with sensors. ual registration of each sensor’s position beforehand. In addition, conventional methods can only measure the 2.2. CO2 concentration indication system position of a xed sensor, and a large number of sensors are required to sense the entire room. Sone proposed a system that displays changes in indoor
To address these issues, we aim to achieve high-density CO2 concentration on a time-series graph [13]. The pur3D measurement and visualization without requiring la- pose of the system is to encourage users to voluntarily borious manual labor. Therefore, we propose an aug- ventilate their room by looking at the measured values mented reality (AR) visualization system for indoor CO2 on the display, and to close the windows when the venticoncentration (see Fig. 1) using both xed sensors and lation is su cient. However, as a result, no change in the sensors mounted on mobile robots that have the follow- user’s ventilation behavior was observed. We expect that ing characteristics: using AR to superimpose the gas distribution directly in the air will encourage users to take action.
2.3. AR visualization systems • Automatic localization of xed CO2 gas sensors
in 3D. • Automatic localization of CO2 gas sensors
mounted on mobile robots in 3D. • Interpolation of the measurement to estimate
dense distribution. • AR visualization of CO2 concentration distribution.
Duan et al. proposed a system that combines an optical see-through HMD, a motion capture camera, and a laser gas detector to visualize gas distribution in the real environment [14]. Their system visualizes in real-time by mapping the detected gas information on the oor surface to the color and size of the visualization model. Kataoka et al. proposed an AR system for real-time measurement and visualization of 3D sound elds, that are invisible 2. Related Work to the human eye, which combines SLAM (simultaneous localization and mapping), an optical see-through HMD, 2.1. CO2 concentration measurements and acoustic measurement technology [15]. By superNumerical uid dynamics models such as CFD simula- imposing the measured sound eld information on the tions are often used to analyze atmospheric gases. How- real environment, it is possible to present information ever, there is a large discrepancy between the predicted coherent with the environment. In this study, we use AR CO2 concentration by CFD simulation and the measured to superimpose objects that represent the observed CO2 value by sensors [11]. Therefore, we measure changes in concentration on the real environment. CO2 concentration in the real environment.
Display Acquisition of position coordinates for fixed sensors
CO2 Data
(OSC) Acquisition of position coordinates for mobile sensors
CO2 concentration distribution calculation Coordinate transformation matrix calculation QR Code Recognition
Interpolated
Data (OSC) QR ID Data Position Data (OSC)
Receive interpolated data Creation of visualization models Visualization of CO2 concentration
corresponding marker. While the user is not looking at the corresponding marker, the CO2 concentration disIn this study, we build an AR visualization system for tribution is calculated using the last measurement. An CO2 concentration using both xed and mobile sensors. external motion tracker can be attached on a mobile senFigure 2 shows the system con guration. The system sor if continous measurement is desired while the user is automatically calculates the distribution of CO2 concen- not looking at the marker. Automatic acquisition of the tration in a room using the position coordinates and mea- position coordinates of each sensor enables automatic sured values of xed and mobile sensors that patrol the calibration of sensor placement. The visualization space room. We use UDP-based OpenSound Control (OSC) to is divided into a voxel grid, and the CO2 concentration communicate the measurement. The visualization model at each grid point is calculated by linear interpolation corresponding to the calculated CO2 concentration is from the position coordinates and measurements of each displayed in the form of AR in real-time using an AR- sensor. We render a particle-based smoke-like visualizaHMD. We believe that many people will be wearing AR tion model based on the calculated CO2 concentration. glasses in the future. This system can be used for daily The color of the visualization model continuously corsensing of CO2 concentration in o ces and other indoor responds to the CO2 concentration value. For example, spaces. The system can also be used to stimulate ven- low CO2 concentrations can be represented by green and tilation behavior, to help optimize furniture placement, high concentrations by red. The visualization model is and to improve the layout of air conditioning equipment displayed in the real environment in real-time using the such as circulators and air puri ers. AR-HMD.
CO2 concentration is measured using a set of CO2 gas sensor modules, which consist of a microcomputer and a CO2 gas sensor. The CO2 concentration distribution 4. Implementation in a room can be estimated by linear interpolation from the position coordinates and measured values of xed 4.1. Sensing sensors (Fig. 1(b)) plased at various locations in the room We implemented a prototype system using several CO2 and the mobile sensors that patrol the room. As a mobile gas sensor modules those xed in various indoor locasensor, we can use a CO2 sensor module mounted on a tions and those mounted on an autonomous mobile robot moving object such as an airship robot (Fig. 1(c)) and an (iRobot Roomba 600 Series) and an airship robot (William autonomous mobile robot (Fig. 1(d)). The position of the Mark Air Swimmer Shark). The CO2 gas sensor modules sensors can be acquired either by an AR marker attached consist of a M5Stack Gray (120 [g]) and a TVOC/eCO2 gas on the sensor and a built-in camera of the AR-HMD, or by sensor unit (SGP30). The CO2 concentration is measured an external motion tracker. Assuming that the AR-HMD at a sampling rate of 1 Hz. The microcomputer for the can estimate its self position by SLAM, each marker’s CO2 gas sensor module mounted on the airship robot is position can easily be acquired by simply looking at its a M5Stick C Plus (21 [g]) to reduce weight. TVOC/eCO2 AR marker. One observation is enough for xed sen- gas sensors mainly measure volatile organic compounds sors. In the case of mobile sensors, the position can be and hydrogen (H2). The eCO2 concentration is calculated continuously acquired while the user is looking at the 2]/s icnagmtehreaQoRf tchoedeHaMttaDc.hFeodutro ttrhaectkrearcskewriwthitQh Rthecobdueislt-aint[nm tached are used for calibration. A transformation matrix ito is calculated from the position coordinates in the HMD lrea coordinate system and the position coordinates in the cce tracker coordinate system. The position coordinates of A the tracker in the coordinate system of the HMD are calculated using the transformation matrix and obtained automatically. based on the H2 concentration. The minimum measurement value of the sensor is 400 [ppm]. Figure 3 shows example CO2 concentration and acceleration of the microcomputer measured while the autonomous robot was running on the oor for ve minutes. The measured CO2 concentration did not change signi cantly even when the acceleration changed signi cantly due to collision.
modules are detected by the built-in camera of the HMD to obtain the position coordinates of xed sensors placed at various locations in the room. We use a Microsoft HoloLens 2 as an AR-HMD. Automatic acquisition of sensor position coordinates enables automatic calibration of sensor placement.
are required to achieve spatially high-density 3D measurements. We thought that high-density 3D measurement could be realized by having a mobile robot mounted with sensors patrol the room. The positions of the mobile robots are tracked either by a QR code or by an HTC VIVE Tracker (for the autonomous robot). In the case of QR codes, the position can be continuously acquired while the user is looking at the QR code attached to the mobile robot. It is discrete data, but can be measured over a wide range. If continuous measurement is desired while the user is not looking at the QR code, an external motion tracker can be attached on the autonomous mobile robot. CO2 concentration distribution can then be calculated with higher accuracy.
Regarding the calibration of the position sensing, rst, the coordinate system of the tracker is aligned with that of the HMD. The position coordinates of the tracker in the coordinate system of the HMD are obtained by read
distribution The visualization space is divided into a voxel grid, and the CO2 concentration at each grid point is calculated by linear interpolation from the position coordinates and measurements of each sensor. The weighted average method is used to estimate the value of each grid point from irregularly distributed measurements [16]. The weight wp, which is inversely proportional to the square of the distance rs from each grid point (xi, yj , zk) to the measurement point by the xed sensors (xs, ys, zs), is given by Eq. 1: ws =
1 (rs2 + ↵ 2)s where ↵ and s are parameters that specify the weights. The smaller ↵ , the higher the data su ciency, and the larger s, the more pronounced the di erence in weights between the far and near points. The weight wm, which is inversely proportional to the square of the distance rm from each grid point (xi, yj , zk) to the measurement point by the mobile sensors (xm, ym, zm), is decreased exponentially over time and is forgotten after a certain time. The exponential decay of the weights is given by Eq. 2: wm(t) =
1 (rm2 + ↵ 2)s e t where t is time and is a positive number called the decay constant. The larger is, the faster wm decreases. Finally, the CO2 concentration estimation at the grid point fijk is given by Eq. 3 where Fs is the measured value by the xed sensor and Fm is the measured value by the mobile sensor: fijk =
PN m=1 wm(t)Fm s=1 wsFs + PM PN m=1 wm(t)
s=1 ws + PM
distribution We render a particle-based smoke-like visualization model based on the calculated CO2 concentration. The visualization model is displayed in the real environment (1) (2) (3) in real-time using the AR-HMD as shown in Fig. 4. This method is considered semi-transparent volume rendering. It is similar to a traditional splatting technique, but instead of splatting, the smoke particles are assigned a color based on the CO2 concentration and opacity. The size of each voxel is 20 x 20 x 20 cm and the spatial resolution is coarse, so simply lling the image with a single color will result in noticeable jaggies. In addition, low transparency reduces the visibility of the real environment, while high transparency reduces the visibility of the visualization. For these reasons, we decided to use particles for visualization. The color of the smoke particles is continuously mapped to the CO2 concentration, for example, green for low and red for high. The transfer function can be changed by a con guration le depending on the user preference. The latency for the CO2 gas sensor measurements to be re ected in the visualization model is approximately 887 ms.
Ten participants in their twenties experienced the prototype system for about ve to ten minutes. Afterwards, a 10-question questionnaire was administered, consisting of four questions to investigate awareness of ventilation, ve questions to evaluate the system, and free-response questions. In a survey on ventilation awareness, 80% of respondents indicated that ventilation is important. However, 90% of the respondents did not know that CO2 concentration should be kept below 1000 [ppm] to maintain good indoor air quality. This suggests that a means of communicating the need for ventilation is needed. A Likert scale of 5-points was adopted for the evaluation of this system. The questions are shown in Table 1. The distribution of the survey results is shown in Fig 5. From the responses collected for Q1, Q2, Q3, and Q4, it can be concluded that all four categories of visual clarity of high concentration areas, visual clarity of di erences in concentration distribution, increased awareness of indoor air quality, and increased awareness of ventilation show positive trends. The results of Q5 suggest that AR visualization may be a more e ective means of communicating the need for ventilation than conventional methods such as displaying numerical values on a 2D display. The following are the main opinions and impressions obtained from the free writing. Most participants appreciated the system for ease of understanding the CO2 concentration distribution. Some participants also gave us an idea for better visualization and additional functions which encouraged further work.
• It is easy to see where the CO2 concentration is high or low, and to feel the gradual change. • I felt that the visual representation of CO2 concentration emphasized the dangerous state of my location more than the 2D display. a b c ] m p p [ 2 O C ] m p p [ 2 O
C
I was able to recognize the location and range of high CO2 concentrations.
I was able to visually recognize the di erence between areas of low and high CO2 concentrations.
I increased my interest in cleaner indoor air.
I increased my awareness of ventilation.
Which do you think is a more e ective means of indicating the need for ventilation compared to the method of displaying values on a 2D display? Responce Type 5-Point Likert (1:Strongly Disagree; 5:Strongly Agree) 5-Point Likert (1:Strongly Disagree; 5:Strongly Agree) 5-Point Likert (1:Strongly Disagree; 5:Strongly Agree) 5-Point Likert (1:Strongly Disagree; 5:Strongly Agree) 5-Point Likert (1:2D Display; 5:AR) sor placement. The CO2 concentration distribution in a room is estimated by linear interpolation from the position coordinates and measured values of every sensor.
The visualization model corresponding to the automatically calculated CO2 concentration is displayed in the real environment in real-time using the AR-HMD. We built a prototype using a Microsoft HoloLens 2 as an AR-HMD, a William Mark Air Swimmer Shark as an airship robot, an iRobot Roomba 600 Series as an autonomous mobile robot, and several CO2 gas sensor modules that consist of a M5Stack Gray, a M5Stick C Plus and a TVOC/eCO2 gas sensor unit. A preliminary evaluation revealed that the participants found the system useful and promising.
High-precision interpolation results can be obtained as the number of sensors used increases. We plan to conduct a scalability test to see how many sensors can be connected to our system. In the future, we plan to Figure 5: User evaluation results. evaluate the accuracy of linear interpolation under multiple conditions, such as the number of sensors used, and when measurements are taken with only xed sensors mended standard of the Ministry of Health, Labor and or only mobile sensors. In addition, we will evaluate the Welfare, Japan. However, since it is unclear whether this accuracy of the linear interpolation in the case of longis the optimal color mapping, we plan to experts for their term measurement. We use a William Mark Air Swimmer opinions. Another problem that is always present in vol- Shark as the airship robot and an iRobot Roomba 600 Seume rendering is that the depth information is di cult ries as the autonomous mobile robot, but we currently do to understand due to the information in the foreground. not control their movement. Our system is intended to We are considering performing thresholding to display be used in an environment where humans coexist with only the darker areas, or adding a fog e ect to display the robot, so it is necessary to implement an automatic only a certain amount of the nearby area. patrol method for mobile robots that is compatible with an environment where humans coexist.
In the future, we will improve the color mapping for 6. Conclusion and future work better visibility and encouraging ventilation. Our system can be used as a platform to visualize not only CO2 but also other gases by changing the sensors mounted on the measurement modules. In the future, we will also support other sensors, such as an anemometer to change the decay rate of mobile sensor readings, and to visualize human movement and involvement over time.
CO2 concentration distribution using CO2 sensor modules, mobile robots, and an AR-HMD. In our system, the position coordinates of xed sensors and mobile sensors that patrol the room are obtained using a built-in camera of the AR-HMD, enabling automatic calibration of senin carbon dioxide concentration in bedrooms, Procedia Engineering 57 (2013) 175–182. URL: [1] S. Murakami, Housing and the human body : An en- https://www.sciencedirect.com/science/article/ gineering perspective, Japanese journal of clinical pii/S1877705813007558. doi:https://doi.org/ ecology 9 (2000) 49–62. 10.1016/j.proeng.2013.04.025, modern [2] R. Mimura, T. Chikamoto, Relation of indoor en- Building Materials, Structures and Techniques. vironment and learning e ect of classroom (part [12] M. Jin, S. Liu, S. Schiavon, C. Spanos, Automated 9) in uence of carbon dioxide uctuation and ther- mobile sensing: Towards high-granularity agile inmal environment on workability, physiology and door environmental quality monitoring, Building psychology, Conference of the Japan Society of Air- and Environment 127 (2018) 268–276. Conditioning and Sanitation Engineers, Academic [13] N. Sone, Prototype carbon dioxide concentration Proceedings 2018.8 (2018) 169–172. doi:10.18948/ indicator to encourage ventilation, Naruto Univershasetaikai.2018.8.0_169. sity of Education Information Education Journal 18 [3] J. G. Allen, P. MacNaughton, U. Satish, S. Santanam, (2021) 61–64.
J. Vallarino, J. D. Spengler, Associations of cognitive [14] L. DUAN, D. Liyuan, H. MATSUKURA, P. PUNfunction scores with carbon dioxide, ventilation, PONGSANON, I. Daisuke, S. Kosuke, Superimposand volatile organic compound exposures in o ce ing methane gas visualization over a real environworkers: a controlled exposure study of green and ment using optical see-through augmented reality, conventional o ce environments, Environmental Proceedings of the Virtual Reality Society of Japan health perspectives 124 (2016) 805–812. Conference 26 (2021) ROMBUNNO.2D1–3. [4] U. Satish, M. J. Mendell, K. Shekhar, T. Hotchi, [15] Y. Kataoka, W. Teraoka, Y. Oikawa, Y. Ikeda, D. Sullivan, S. Streufert, W. J. Fisk, Is co2 an in- Real-time measurement and display system of 3d door pollutant? direct e ects of low-to-moderate sound intensity map using optical see-through co2 concentrations on human decision-making per- head mounted display (2018). URL: https://doi.org/ formance, Environmental health perspectives 120 10.1145/3283289.3283358. doi:10.1145/3283289. (2012) 1671–1677. 3283358. [5] Ministry of Health, Labour and Welfare, About [16] Y. INUI, S. MASUMOTO, K. SHIONO, Grivventilation to improve "closed spaces with iew3d: Three dimentional gridding and visualpoor ventilation" in commercial facilities, 2020. ization based on irregularly spaced observations, URL: https://www.mhlw.go.jp/content/10900000/ Geoinformatics 20 (2009) 197–210. doi:10.6010/ 000616069.pdf. geoinformatics.20.197. [6] World Health Organization (WHO), Coronavirus disease (covid-19): How is it transmitted?, 2021. URL: https://www.who.int/ news-room/questions-and-answers/item/ coronavirus-disease-covid-19-how-is-it-transmitted. [7] Ministry of Health, Labour and Welfare, About building environmental sanitation management standards, 2022. URL: https://www.mhlw.go.jp/ bunya/kenkou/seikatsu-eisei10/. [8] Z. Peng, J. L. Jimenez, Exhaled co2 as covid-19 infection risk proxy for di erent indoor environments and activities, medRxiv (2021). doi:10.1101/2020.
09.09.20191676. [9] J. Burgués, V. Hernández, A. J. Lilienthal, S. Marco,
Gas distribution mapping and source localization using a 3d grid of metal oxide semiconductor sensors, Sensors and Actuators B: Chemical 304 (2020) 127309. doi:https://doi.org/10.1016/ j.snb.2019.127309. [10] R. A. Russell, Tracking chemical plumes in 3dimensions, in: 2006 IEEE International Conference on Robotics and Biomimetics, 2006, pp. 31–36.
doi:10.1109/ROBIO.2006.340274. [11] P. Batog, M. Badura, Dynamic of changes