Technology and Scenarios for Objects 3D Models visualization using Augmented Reality Olga Pavlovaa, Andriy Bashtaa, Mariia Kostiuka and Houda El Bouhissi b a Khmelnytskyi National University, Institutska str., 11, Khmelnytskyi, 29016, Ukraine b LIMED Laboratory, Faculty of Exact Sciences,University of Bejaia, 06000, Bejaia, Algeria Abstract Current trends and prospects of using augmented reality technology for various industries are considered. Analysis of publications and the latest commercial developments that use AR showed that this direction is promising both from the scientific and business points of view. In this study, the concept of information technology for 3D models visualization in augmented reality is proposed. A mathematical model of object representation in three-dimensional space, a method and algorithm for objects 3D models visualization in augmented reality, as well as a structural diagram and principle of operation of the proposed information technology for objects 3D models visualization in augmented reality have been developed. A test three- dimensional model of the object was also developed and experiments were conducted on its visualization in AR using the proposed information technology for objects 3D models visualization in augmented reality. Keywords 1 Augmented reality (AR), Information Technology (IT), 3D model, visualization, Mobile Application 1. Introduction Today, the issue of using augmented reality in various areas of activity is of great importance both from a scientific and a practical point of view. According to the latest research on the market [1] augmented reality is an effective tool for business, as it allows to make purchases, namely trying on clothes, shoes, interior items and furniture without physically visiting the store. This is currently extremely relevant in a world facing the consequences of the global Covid-19 pandemic, where businesses have had to look for tools to be able to operate and communicate with customers remotely. It is also extremely relevant now, when Ukraine is at a state of war. Augmented reality provides visualizing objects and things that may not be physically nearby, but a person will see them in real time and in life size. This is useful in the development of virtual training equipment and simulators, for example, for the medical and military industries, since real equipment is expensive and in most cases quite large for the equipment of training centers. There is no doubt that augmented reality (AR) is already being integrated into almost all areas of industry. This technology is already available on almost every smartphone/tablet, and according to Statist's forecasts (Fig. 1), the industry will only grow (capitalization is expected to be 1.73 billion dollars by 2024). A lot of industrial areas (including retail, automotive, healthcare, education, entertainment and others) are implementing AR, there has never been a greater demand for AR-based applications development services than there is today [1]. IntelITSIS’2023: 4th International Workshop on Intelligent Information Technologies and Systems of Information Security, March 22–24, 2023, Khmelnytskyi, Ukraine EMAIL: olya1607pavlova@gmail.com (O. Pavlova); andreybashta@gmail.com (A. Bashta); maria@khnu.km.ua (M.Kostiuk) ORCID: 0000-0003-2905-0215 (O. Pavlova); 0000-0002-0775-1347(A. Bashta); 0000-0002-9559-4109(M.Kostiuk) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) Figure 1: Number of mobile AR active users worldwide from 2019 to 2024 according to [1] According to the analysis, the topics of the most frequently published scientific research works in augmented reality domain are devoted to AR application for educational purposes, medicine, E- commerce and advertising area and AR/VR-based user interfaces for different purposes. The results of the quantitative analysis of the publications number in each domain are presented in Fig.2. Figure 2: Analysis of industries where the most research work is conducted in the direction of augmented reality application Considering the abovementioned, development of a multipurpose information technology that could provide the visualization of any kind of object`s three-dimensional model in augmented reality in natural size and in real time currently would be relevant. Since smartphones or tablets are the most frequently used tools to operate with AR-based applications, the most convenient form of the proposed information technology implementation would be cross-platform mobile application. It would make it available for using on both Android and iOS based devices. The areas of such information technology application can also be various - learning purposes (for example, an atom or molecule structure visualization in Chemistry), in museums or archeology domain to visualize the historic artefacts, in medicine for internal organs structure visualization etc. 2. Related works In the course of the study, an analysis of the latest scientific publications devoted to the application of augmented reality for three-dimensional objects visualization was carried out. In [2] the method, algorithm and software-based approach of route paving using 3D markers and augmented reality technology are proposed. The authors of [3] propose three relatively nontechnical methods to produce 3D AR objects for chemistry courses and demonstrate their use as both quick in-lecture activities and as part of an extended laboratory. Thе paper [4] proposes the integration of photogrammetric reconstruction, 3D modelling and augmented reality application in order to achieve the complete visualization of a stone sculpture even if highly damaged or fragmentary. The paper [5] provides two demonstrations of how Augmented Reality (AR), which is the projection of virtual information onto a real-world object, can be applied in the classroom and in the special laboratory. The paper [6] applies Geographic Information Technologies (GIT) to the field of Cultural Heritage (CH), aiming to analyze patrimonial valuation through Digital Representations (DR). It approaches the main topics of 3D data acquisition, modelling and visualization, as well as cultural heritage applications, and presents 3D and Augmented Reality (AR) trials, together with their main results and discussion. In [7] a general review of using augmented reality as an efficient tool for software publications is given. However, any practical case of using AR is not highlighted. The authors of [8] present DesignAR, an augmented design workstation for creating 3D models. The authors of [9] provide radiological image visualization using virtual and augmented reality for better planning and monitoring of surgeries. In paper [10] reviews and advances existing literature concerning immersive employee experiences in the metaverse. In this research, previous findings were cumulated showing that virtual work environments as computer-generated reality spaces integrate digital twinning, remote collaboration tools, productivity software, and wearable self-tracking devices. The authors of [11] give an example of the application of augmented reality in medical education would be an augmented reality T-shirt that allows students to examine the inside of the human body as an anatomy lab. Also, since IT industry of Ukraine is developing rapidly and following the world IT trends, it was decided to conduct an analysis of Ukrainian scientists` research works in augmented reality domain. Thus, the authors of [12] conduct an analysis of the current state and prospects for the development of augmented reality in Ukraine in business and education. In [13] augmented reality is proposed to be used in university education of future IT specialists. The authors of [14] propose using AR technology for interactive Chemistry learning. Overall Ukrainian scientists mostly consider AR implementation for educational purposes. The conducted review of the literature sources showed that none of the known solutions are aimed for multi-purpose three-dimensional objects visualization in augmented reality. Also the analysis of already known software solutions which have AR-component and provide visualization of 3D models of objects has been conducted. The advantages and disadvantages of the reviewed tools are also given. The results of the analysis are presented in Table 1. Table 1 Review of the ready-made solution that use augmented reality for objects visualization Name Presenta- Operating Brief Advantages Disadvantages tion form System Description Fectar Mobile Android, IOS In app-AR- -variety of AR- -low quality of application model models AR-models gallery, -large users because of model can community moderation lack be added by -does not -overloaded user require device interface relative with AR to the support application`s functionality Assemblr Mobile Android, IOS An AR - laconic design -access to certain application model - a wide AR models is paid gallery app selection of AR that allows models users to - convenient create and search for AR share AR models by models dividing them from into categories mockups - a large number of AR models divided by categories - the possibility of using ready- made 3D models UniteAR Mobile Android, IOS An - a large - access to the application application number of AR application is designed to models divided granted only after create, by categories granting access to distribute -the possibility the camera and display of using ready- - low quality AR AR models made 3d models models - no model preview AR- Mobile Android, iOS The part of - detailed and - the number and component application the animated AR variety of AR of Google application models models is limited application designed to - availability of - users cannot display an a preview upload their own AR model before models related to a switching to search AR mode query TeamViewer Cross- Android, iOS, Designed - the product - paid application Assist AR platform Windows, for remote has a clear application macOs assistance practical - a small number by application of pointers in the transmitting form of AR a video models signal from the device and overlaying auxiliary symbols in the form of AR models on top of the video. ARt Website Cross- A website - preview for - minimalistic platform aimed at AR models interface rendering - expandable - AR support is AR models list of AR required from a for exterior models device design. ARvid Mobile iOS An AR- -over 500 -paid application application based realistic -in app purchases mobile augmented application reality 3D with 3D - most of the models to AR models are place and animated and control. have 4K textures. Currently, the only completely ready free of charge tool for visualizing three-dimensional models of objects is the AR component of Google mobile application [15]. Among the advantages of this product is that the applications are available for Android and iOS operating systems, and the models for visualization are animated. Among the disadvantages - the database contains only 32 models, these are mostly animals and birds, the function of uploading own models is absent. Taking into account the relevance of this issue, it was decided to develop an augmented-reality- based information system for objects three-dimensional models visualization in the form of cross- platform mobile application. Therefore, the aim of this study is: 1) to create a method and algorithm of objects 3D models visualization in augmented reality; 2) to develop the client-based part in the form of cross-platform mobile application which provides 3D models visualization; 3) to create the test models for visualization in AR and conduct the experiment on objects 3D models visualization in augmented reality in Khmelnytskyi National University campus. 3. Representing of a three-dimensional object in space. Mathematical model To present a 3D model of object in real world, it is necessary to take into account that we need to work in three-dimensional space. Since the ultimate goal is to develop a tool for visualizing models in the form of a mobile application, we will use the smartphone camera with AR function support as a tool to operate with images in augmented reality. First, we need to consider the coordinates of the external place, where the camera of mobile phone focuses. The phone itself will also conditionally be located in the three-dimensional Cartesian coordinate system at the intersection of the X, Y and Z axes. Figure 3 shows a scene with an object model image in the form of a polygon with a camera (a) and a view from the device's camera (b). а) b) Figure 3: The model of the object in the three-dimensional Cartesian coordinate system (a) the entire scene with the camera b) the view from the device's camera To develop an algorithm for representing an object in augmented reality, it is necessary to calculate the distance from the device's camera to the model of the object, which we will see on the smartphone screen according to formula (1): 𝑅𝑅 𝑙𝑙 = 𝛼𝛼𝛼𝛼 , (1) sin 360 where l is a distance from the device camera to the AR model; R is a radius of the virtual sphere attached to the camera; 𝛼𝛼 is an angle of view. The following formulae (2-4) are used to convert the coordinates of the represented model from the Cartesian coordinate system to the spherical one: 𝑟𝑟 = �𝑥𝑥 2 + 𝑦𝑦 2 + 𝑧𝑧 2 , (2) where r is a distance from the origin of the coordinates to the point. �𝑥𝑥 2 +𝑦𝑦 2 𝜃𝜃 = 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 � 𝑧𝑧 �, (3) where 𝜃𝜃 is a polar angle. 𝑦𝑦 𝜑𝜑 = 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 � �, (4) 𝑥𝑥 where 𝜑𝜑 is an azimuthal angle. Also we need to calculate the orientation of the model along the Y axis, around the center of rotation (formula (5)) and the distance between points in three-dimensional space (formula (6) : 𝑦𝑦 = 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎(−𝑥𝑥, −𝑧𝑧) − 𝜃𝜃. (5) 𝑙𝑙 = �(𝑥𝑥2 − 𝑥𝑥1 )2 + (𝑦𝑦2 − 𝑦𝑦1 )2 + (𝑧𝑧2 − 𝑧𝑧1 )2 , (6) where L is the distance between points or beacons in three-dimensional space. 4. Method of objects 3D models visualization in Augmented Reality To develop a proposed tool for 3D models visualization, first it is necessary to develop a method of its operation, taking into account the abovementioned mathematical model. The graphic representation of the Method of objects 3D models visualization in augmented reality is presented in Fig.4. The proposed method consists of the following steps: 1. Development of software for model visualization in augmented reality; 2. Construction of test models of 3D models of objects 3. Downloading test models to a software and technical tool, which is most convenient to present in the form of a mobile phone application; 4. Life-size, real-time augmented reality 3D models of objects in the environment using a smartphone camera. 5. Conducting experiments on the successful location of the object in space. Figure 4: Method of 3D objects models visualization in augmented reality As it can be seen from the Figure, software and technical toolkit will contain 3D models in its data storage, making them available for visualization using the smartphone camera. User also can upload own models in application`s data storage throuhg user interface. When developing applications with complex functionality, special attention should be paid to taking into account all possible bottlenecks at the early stages of the life cycle, that is, when designing the architecture of the software tool. That is why a special attention should also be paid to UX development. According to [16] user interface should follow the main Gestalt principles — Proximity, Similarity, Continuation, Closure, and Figure/Ground, since AR-based user interface design is not only about the aesthetics, it is about intuitiveness and comprehensibility of functionality in appearance, which is more a psychological component of user interface design. Such interface should be user friendly and clear. Also a special attention should be paid to 3D graphics design. The models for representing in AR must be of a high quality. Based on the method, an algorithm for objects 3D models visualization in augmented reality was developed, which is presented in Figure 5. As an environment for models visualisation any view from the device camera can be chosen. the image of the model appears on device screen in preview mode. When user selects the 3D real-time display option, the image is superimposed over the image of the surrounding space (environment) in the smartphone camera, allowing the user to see the model in the field of view of the device's camera. The user can move and scale the model. The principle of operation of the proposed in this work information technology for 3D objects models visualization in augmented reality is shown in Figure 6. Figure 5: Algorithm of 3D models of objects rendering in augmented reality Figure 6: The principle of the software and technical tool operation for the reproduction of objects 3D models in augmented reality. The proposed information technology for 3D objects models visualization in augmented reality consists of Data Storage and Software System. Data Storage is the part where the models are stored. Users can upload their own models there. Software System is divided by two parts - a user side part – i.e Frontend part where user can interact with program interface, choose and preview the model for visualization; and server-side part – i.e Backend part which is responsible for saving a new uploaded by user model into the data storage, validation of the chosen by user model in AR and its visualization using device camera. The structure of the proposed information technology for 3D objects models visualization in augmented reality is presented in Fig.7. Figure 7: The structure of the proposed information technology for 3D objects models visualization in augmented reality 5. Results & Discussion For conducting the experiments a test 3D model of “Solar Tree construction” was developed. For development Blender 3D modelling Environment was used. The model is presented in Figure 8.1. Also a script for visualization of the test model was developed. We can upload the model in software database in .glb or .obj extension and have it available for the visualization. The script for the model visualization was transferred to Android-based smartphone with AR function support for conducting the experiment. The experiment on the proposed model visualization in real time in augmented reality has been conducted. As an environment for the visualization of outdoor art object the Botanical Garden of Khmelnytskyi National University was chosen. During the experiment the test model was visualized in AR through smartphone camera in the external environment and a photo from camera view was taken. The photo with the experiment results is presented in Fig.8.2. As the experiment proved, the developed software works quite well and performs its functions. The test model was displayed on the smartphone screen and was superimposed over the image of the surrounding space using smartphone camera. The further efforts of the authors will be aimed at: 1) creating more test models and conduct the experiments on visualization 3D models for different purposes and in various environments; 2) developing user interface of the proposed information system for objects 3D models visualization in augmented reality in the form of cross-platform mobile application. 3) application of the developed tool for scientific and real-life needs, such as digitization and visualization of museum exhibits and archaeological artifacts. Figure 8.1: 3D model for Figure 8.2: The results of the experiments on 3D objects models visualization in AR visualization in augmented reality 6. Conclusions The current trends and the prospective of Augmented Reality technology application for various purposes are considered. The literature analysis and the analysis of the related works showed that the interest to AR technology is present both from scientific and business sides. The industries where augmented reality is currently being implemented the fastest and in the largest volume are science, medicine and e-commerce/advertising. The literature review provided the conclusion that currently there are no technical means that provide visualization of three-dimensional models of objects in augmented reality with the ability of the user to add their own models. Therefore, creating of such information technology is an urgent task. AR using trends research has shown that the most effective form for the proposed information technology implementation is the form of a cross-platform mobile application which is available both Android and iOS users. In this study, a mathematical model for visualizing an object in three-dimensional space, a method and algorithm for visualizing an object model in augmented reality, as well as a structural diagram and principle of operation of the proposed information technology were developed. A test three-dimensional model was also developed and experiments were conducted on its visualization in augmented reality using the proposed information technology. Experiments have shown that the prototype of the developed information technology works quite well. Therefore, directions for further research are developing user interface of the proposed information system for objects 3D models visualization in augmented reality in the form of cross-platform mobile application and its application for scientific and real-life needs, such as digitization and visualization of museum exhibits and archaeological artifacts. 7. References [1] Mobile augmented reality (AR) market revenue worldwide from 2021 to 2026 URL: https://www.statista.com/statistics/282453/mobile-augmented-reality-market-size/ (Last accessed 11.02.2023). [2] Pavlova O., Bashta A., Kravchuk S., Hnatchuk Y., Bouhissi H.E. Augmented Reality Based Technology and Scenarios for Route Planning and Visualization. CEUR Workshop Proceedings, 2022, 3156, pp. 613–623 [3] Sanii B. Creating Augmented Reality USDZ Files to Visualize 3D Objects on Student Phones in the Classroom. Journal of Chemical Education 2020 97 (1), 253-257 DOI: 10.1021/acs.jchemed.9b00577 [4] Gherardini F. 3D Virtual Reconstruction and Augmented Reality Visualization of Damaged Stone Sculptures. IOP Conference Series: Materials Science and Engineering, Volume 364, Florence Heri-Tech - The Future of Heritage Science and Technologies 16–18 May 2018, Florence, Italy DOI: 10.1088/1757-899X/364/1/012018 [5] Plunkett, K. N. A Simple and Practical Method for Incorporating Augmented Reality into the Classroom and Laboratory. Journal of Chemical Education 2019 DOI: 10.1021/acs.jchemed.9b00607 [6] Marques L. et al. Cultural Heritage 3D Modelling and visualisation within an Augmented Reality Environment, based on Geographic Information Technologies and mobile platforms. ACE: Architecture, City and Environment, 11 (33): 117-136, 2017. DOI:10.5821/ace.11.33.4686. ISSN: 1886-4805. [7] Wolle P., Müller P., Rauh D. Augmented Reality in Scientific Publications—Taking the Visualization of 3D Structures to the Next Level. ACS Chemical Biology 2018 13 (3), 496-499 DOI: 10.1021/acschembio.8b00153 [8] Reipschläger P. and Dachselt R. 2019 DesignAR: Immersive 3D-Modeling Combining Augmented Reality with Interactive Displays. In Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces (ISS '19). Association for Computing Machinery, New York, NY, USA, pp. 29–41. https://doi.org/10.1145/3343055.3359718 [9] González I., Méndez J.A., Palomera R. et al. Applications of Virtual and Augmented Reality in Biomedical Imaging. Journal of Medical Systems 43, 102 (2019). https://doi.org/10.1007/s10916- 019-1239-z [10] Carter D. Immersive Employee Experiences in the Metaverse: Virtual Work Environments, Augmented Analytics Tools, and Sensory and Tracking Technologies. Psychosociological Issues in Human Resource Management; Woodside10 (1), 2022, pp. 35-49. DOI:10.22381/pihrm10120223 [11] Kye1 B., Han N., Kim E., Park Y., Jo S. Educational applications of Metaverse: possibilities and limitations. J Educ Eval Health Prof 2021, 18(32). DOI: https://doi.org/10.3352/jeehp.2021.18.32 [12] Mitii I., & Soloviov V. (2018). Augmented Reality: Ukrainian Present Business and Future Education. Educational Dimension, 51, pp. 290-296. [13] Babkin V., et al. Using augmented reality in university education for future IT specialists: educational process and student research work. CEUR Workshop Proceedings, 2021. [14] Nechypurenko P., Stoliarenko V., Starova T., Selivanova T., Markova O., Modlo Y., & Shmeltser E. (2020). Development and implementation of educational resources in chemistry with elements of augmented reality. [15] Google AR/VR Components. URL:https://arvr.google.com/ (Last accessed 14.12.2022). [16] Hovorushchenko T., Pavlova O., & Kobel K. Method of Evaluating the User Interface of Software Systems for Compliance with the Gestalt Principles. In 2019 IEEE 14th International Conference on Computer Sciences and Information Technologies (CSIT) Vol. 2, pp. 138-141 [17] Augmented Reality Trends to Redefine Business Growth in 2023. URL: https://www.intelivita.com/blog/augmented-reality-trends/ (Last accessed 14.12.2022). [18] Augmented Reality Trends of 2023: New Milestones in Immersive Technology. URL: https://mobidev.biz/blog/augmented-reality-trends-future-ar-technologies (Last accessed 14.02.2022). [19] Sasaki R., Yamamoto K. A Sightseeing Support System Using Augmented Reality and Pictograms within Urban Tourist Areas in Japan. ISPRS Int. J. Geo-Inf. 2019. 8, 381. https://doi.org/10.3390/ijgi8090381 [20] C. C. Ho, M. -C. Ho and C. -Y. Chang. Markerless Indoor/Outdoor Augmented Reality Navigation Device Based on ORB-Visual-Odometry Positioning Estimation and Wall-Floor-Boundary Image Registration. 2019 Twelfth International Conference on Ubi-Media Computing (Ubi-Media). Bali, Indonesia. 2019. pp. 199-204. doi: 10.1109/Ubi-Media.2019.00046