Indoor spatial cognition of the hearing/visually impaired: pentest-inspired WiFi heatmap on Android platform Dmytro Zubov1,* , Andrey Kupin2 and Nurlan Shaidullaev1 1 University of Central Asia, 125/1 Toktogul St., Bishkek, 720001, Kyrgyz Republic 2 Kryvyi Rih National University, 11 Vitaly Matusevich St., Kryvyi Rih, 50027, Ukraine Abstract This study aims to improve the existing indoor spatial cognition systems for hearing and visually impaired people, especially in unfamiliar spaces where outdoor tools are not functional. The experiment conducted in the academic multistage building of the University of Central Asia (Naryn campus, Kyrgyz Republic) demonstrates a positioning accuracy of 100 % for indoor localization. For this purpose, two Java Android applications were developed on the Android 10 operating system. The first application is employed to design a WiFi heatmap of the building using the Java Android class WifiManager to gather information about BSSIDs (Basic Service Set Identifiers) of WiFi networks indoors (this task might be done by any project team member). The second application also implements the principle of the pentest passive reconnaissance and provides audio feedback to the visually impaired and textual/photo information to people with disabling hearing loss. Java Android applications are freeware and runnable on over 81 % of Android devices and 71 % of smartphones worldwide as of June 2024. Keywords Hearing impaired, visually impaired, indoor spatial cognition, WiFi heatmap, Android1 1. Introduction The World Health Organization reported that over 2.2 billion and 430 million people worldwide have vision impairment and disabling hearing loss, respectively [1, 2]. Nowadays, indoor spatial cognition remains challenging [3] for the hearing and visually impaired (HVI) because traditional solutions like guide dogs, white canes, and outdoor global navigation systems, such as BDS WeChat and Google Maps [4], are not functional inside unfamiliar spaces [5]. Existing assistive solutions, such as DeafSpace [6], [7], and ORB-SLAM2 [8], employ heterogeneous methods (e.g., computer vision and Bluetooth beacon fingerprinting) and soft-/hardware (e.g., Arduino, ESP8266/32, and Raspberry Pi microcontrollers) to support the spatial cognition and navigation of HVIs. They have distinctive drawbacks, such as requiring the installation of new equipment like Bluetooth low-energy beacons and complicated algorithms like image processing, that demand high- performance devices. This study addresses an important social problem of the indoor spatial cognition of HVIs. Passive reconnaissance with a WiFi heatmap as the main outcome is the distinctive feature and advantage of the proposed approach inspired by the pentest ethical hacking methods [9]. In this project, two Java Android applications [10] have been developed: one for the reconnaissance of the local area network (any project team member can do it) and another for the HVI localization indoors. The academic multistage building of the University of Central Asia (Naryn campus, Kyrgyz Republic) has been used as an experimental testbed. ICST-2024: Information Control Systems & Technologies, September 23-25, 2023, Odesa, Ukraine. Corresponding author. These authors contributed equally. dzubov@ieee.org (D. Zubov); kupin@knu.edu.ua (A. Kupin); nurlan.shaidullaev@ucentralasia.org (N. Shaidullaev) 0000-0002-5601-7827 (D. Zubov); 0000-0001-7569-1721 (A. Kupin); 0009-0003-5165-897X (N. Shaidullaev) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings The final product is an Android .apk file, which can be shared via Google Play Store, university website, and/or direct copying. It is designed to run on approximately 81 % of Android devices as of June 2024, as it was developed for the Android operating system (OS) 10. To support the indoor spatial cognition of HVIs, a new Java Android mobile application was developed to find the location of HVIs inside a multistage s contribution jointly includes the following: 1. Affordability: the software was developed in the Android Studio 4.0 free of charge, and it is planned to be shared without any cost. 2. User-friendly interface: textual and photo information is shown to the hearing impaired [11], and the audio is played to the visually impaired [12]. 3. No additional equipment is required in this project since up-to-date Android smartphones can scan (using active or passive reconnaissance techniques) and identify parameters of WiFi networks. The Android platform is widely used: market share is approximately 71 % worldwide as of June 2024. 4. Experimental results showed a positioning accuracy of 100 % for indoor localization at the academic building of the University of Central Asia. 5. The company (University of Central Asia in this study) assistance: improve positioning as an advanced and innovative organization. This paper is organized as follows: Section 2 presents the analysis of previous studies in the context of the active or passive reconnaissance pentest techniques. It also introduces the general architecture of the assistive system for the HVI indoor spatial cognition. Section 3 discusses three main phases of the project lifecycle: mapping the academic multistage building of the University of Central Asia; designing the WiFi heatmap and indoor spatial cognition of HVIs using two developed Java Android applications. Section 4 describes the successful experiment conducted at the University of Central Asia: indoor spatial cognition of HVIs using a Java Android Application and the WiFi heatmap. Results and discussion are presented in Section 5. Conclusions are summarized in Section 6. 2. Related Works Spatial awareness is a crucial and most difficult task in HVI navigation. Existing positioning methodologies might be divided into four main categories [13, 14]: • wireless networks; • microelectromechanical sensor systems; • magnetic field distribution-based methods; • computer vision. Global navigation system-based methods, such as the pseudolite GPS [14], are not considered since multistage building is discussed in this study. Nowadays, wireless networks are mainly represented by WiFi and Bluetooth technologies. WiFi access points are widely employed to access Internet/Intranet resources, making them suitable for HVI localization without additional equipment installation like Bluetooth beacons. Microelectromechanical sensor systems and magnetic field distribution-based methods require frequent recalibration, which can be quite complex after software release and updates. Computer vision needs high-performance soft-/hardware to provide accurate real-time feedback. Authors have a positive experience with QR codes [15] due to their ease of placement on walls. However, they may require additional space and permissions that organizations, as discussed in this study, may not always allow. This study focuses only on wireless networks, as the end user relies solely on a wireless connection to locate indoor areas in the academic multistage building. The analysis of previous studies showed that two opposite techniques, active and passive reconnaissance, are employed to scan wireless networks inside multistage buildings. In active reconnaissance, additional equipment like Bluetooth low-energy beacons and RFID (Radio Frequency Identification) tags, such as presented in [13, 16], is installed indoor, and then received signals are employed to identify the location. Typically, network administrators avoid using the same Basic Service Set Identifiers (BSSIDs) with Media Access Control (MAC) physical addresses when the network is reconfigured and/or access points or wireless routers are replaced. The key benefit of this technique is its independence from the host network topology and characteristics such as Bluetooth addresses and BSSIDs. A major drawback is that the installation of additional equipment is not free and is usually not allowed by the organization's policies. In this study, a passive reconnaissance technique [16, 17] is used to gather information about BSSIDs inside the multistage building at different locations, which is then used to design a WiFi heatmap (first mobile application). This WiFi heatmap is then utilized in the second mobile application for the actual indoor localization of HVIs. The main advantage of this technique is its affordability as no additional equipment is required. A major drawback is that the wireless networks are overlapped sometimes, which leads to incorrect localization or the need to consider several places at the same time. Regardless of the localization technique, the mobile application user interface should include the following features to ensure accessibility for HVIs [11, 12, 18]: 1. Visually impaired: the audio [19] should be played, and the button as a user interface element should be placed next to the smartphone screen border (the top in this project) to perform the scan of the WiFi host network. 2. Hearing impaired: textual and visual (photos in this project) information should be displayed on the smartphone screen. The general architecture of an assistive system for the HVI indoor spatial cognition based on the mobile application, the passive reconnaissance, and the above-stated analysis of previous studies is shown in Figure 1. The backend of the project, i.e., the database with BSSIDs and indoor locations, can be implemented as a part of the mobile application or using a Backend-as-a-Service app development platform like Google Firebase with a cloud-hosted NoSQL Realtime Database. In the current version, the first approach is employed since the network administrator can change the MAC address of the WiFi interface to the required one. The mobile application uses the database and the BSSIDs inside a building to identify the location and provide the textual/visual/audio information to HVI. 3. Methods The project lifecycle consists of three main phases: mapping [20] the academic multistage building of the University of Central Asia; designing the WiFi heatmap and indoor spatial cognition of HVIs using the first and second developed Java Android applications, respectively. 3.1. Mapping the academic multistage building of the University of Central Asia The University of Central Asia has several constructions on the Naryn campus, but only the academic multistage building (it is marked out with the red arrow in Figure 2) is challenging for the HVI spatial cognition, as other facilities are not typically visited by newcomers. In Figure 2, the photo was captured by the flagship triple-camera drone DJI Mavic 3 Pro Cine by author Nurlan Shaidullaev on June 14, 2024. Figure 1: General architecture of the assistive system for the HVI indoor spatial cognition. Academic building Figure 2: Academic multistage building of the University of Central Asia, Naryn campus, Kyrgyz Republic (photo was taken by the flagship triple-camera drone DJI Mavic 3 Pro Cine; author Nurlan Shaidullaev). The WiFi heatmap is designed at the following points in the academic multistage building: ground floor Figure 3, second floor Figure 4, third floor Figure 5 (the point number represents the order of data collected during the WiFi heatmap design). Figure 3 shows the following points on the ground floor: • 1, 2, and 3 apartments of teaching staff; • 4, 5, and 6 central corridor; • 7, 8, 9, and 40 main entrance; • 19 computer class; • 20, 21, and 22 canteen; • 23, 24, and 25 library. Figure 4 presents the following points on the second floor: • 10, 11, and 12 classes; • 13, 14, and 15 central corridor; • 16, 17, and 18 apartments of teaching staff; • 35 yellow classroom; • 36, 37, 38, and 39 corridor to dorms. Figure 5 presents the following points on the third floor: • 26, 27, and 28 offices of teaching staff; • 29, 30, and 31 central corridor; • 32, 33, and 34 apartments. •1 • 40 •9 •2 •8 •3 •7 •4 •5 •6 • 22 • 21 • 20 • 25 • 24 • 23 • 19 Figure 3: Points of the WiFi heatmap: academic building, ground floor. • 18 • 10 • 17 • 11 • 16 • 12 • 15 • 14 • 13 • 39 • 38 • 37 • 36 • 35 Figure 4: Points of the WiFi heatmap: academic building, second floor. The distance between points ranges from 15 m to 20 m around the academic building. In this study, the positioning accuracy takes binary values: if the positioning information is accurate within 10 m of the point, then the accuracy is 100 %; otherwise, it is 0 %. 3.2. Designing the WiFi heatmap of the academic multistage building of the University of Central Asia The WiFi heatmap is designed using the first developed Java Android application. A Java Android class WifiManager provides information about the SSID (Service Set Identifier), BSSID, and RSSI (Received Signal Strength Indication) of all available WiFi connections at specific points. Figure 6 shows two screenshots examples of SSIDs, BSSIDs, and RSSIs at points 6 and 23. • 34 • 26 • 33 • 27 • 32 • 28 • 31 • 30 • 29 Figure 5: Points of the WiFi heatmap: academic building, third floor. Analysis of BSSIDs in the academic building of the University of Central Asia indicates that most of points 1-40 (see Figures 4, 5, and 6) have unique BSSIDs. An example for the point 1 is as follows (SSID, BSSID, RSSI): • -50; • -53; • - -50; A) B) Figure 6: Designing WiFi heatmap: examples of SSIDs, BSSIDs, and RSSIs at points 6 (A) and 23 (B). • - -53; • - -53; • - -53. However, several points from different locations may be connected to an access point with the same BSSID: 1. Points 5, 21, and 22: t the HVI. 2. Points 6, 7, and 23: t : Campus administration & is delivered to the HVI. This case is presented in Figure 6: WiFi connections at these points have at least two common BSSIDs d8:84:66:0f:7d:93 and d8:84:66:0f:7d:90 with quite good RSSI  -63. 3. Points 24, 38, and 39: t 4. Experiment at the University of Central Asia: Indoor Spatial Cognition of HVIs using a Java Android Application and the WiFi Heatmap The experiment was conducted in the academic multistage building of the University of Central Asia. The WiFi heatmap was used to find the HVI location inside a building. For this purpose, the second Java Android Application was developed. It uses the Java Android class WifiManager to gather the information about SSID, BSSID, and RSSI of all available WiFi connections at specific points. Figure 7 presents four screenshots examples of the successful HVI localization inside a building at point 19 (smartphones are Doogee S96 Pro with Android 10 OS and Samsung M31 SM-M315F/DSN with Android 12 OS) and outdoors . The information (text and photo) is friendly displayed to the hearing impaired, and the audio .mp3 file, i.e., the speech navigation [21], is friendly played to the visually impaired. The audio can be easily switched off on the Android smartphone if necessary. Experimental results demonstrated a positioning accuracy of 100 % for indoor localization. Figure 7 shows the prospect for future development the constraint layout looks different on screens of non-identical smartphones. 5. Results and Discussion This study presents an assistive system to support the indoor spatial cognition of HVIs. Two Java Android mobile applications were developed to design a WiFi heatmap and to find the location of HVI inside a multistage building. The main result jointly includes the following: 1. An experiment conducted at the University of Central Asia shows a positioning accuracy of 100 % for indoor localization. 2. A user-friendly interface provides audio feedback to the visually impaired and textual/photo information to people with disabling hearing loss. 3. The software was developed in Android Studio 4.0 free of charge, and it is planned to be shared without any cost. Two questions were raised during the discussion of the presented project at the departments of Computer Science (University of Central Asia) and Computer Systems and Networks (Kryvyi Rih National University): 1. The constraint layout of the Java Android mobile application appears differently on screens of non-identical smartphones, which should be improved for consistency. A suggestion was to calculate layout parameters based on screen size to ensure uniformity across different smartphones. 2. The Android mobile applications were developed using the classical imperative approach with the Java programming language. However, it was suggested that the future development of the project should employ a declarative methodology, such as the Kotlin-based Jetpack Compose, which is currently prevalent in Meta Company. A) B) C) D) Figure 7: Examples of successful HVI localization inside a multistage building at point 19 (smartphones are Doogee S96 Pro with Android 10 OS (A) and Samsung M31 SM-M315F/DSN with Android 12 OS (C)) and outdoors (same smartphones, (B) and (D), respectively). 6. Conclusions In this study, two Java Android applications (design of a WiFi heatmap and localization of HVIs) were developed to assist the indoor spatial cognition of HVIs using the WiFi heatmap of the academic multistage building of the University of Central Asia (Naryn campus, Kyrgyz Republic). The main contribution jointly includes the following: a positioning accuracy of 100 % for indoor localization; a user-friendly interface that provides audio feedback to the visually impaired and textual/photo information to people with disabling hearing loss; affordable (free-software license) and widely accessible (the developed Java Android applications might be run on over 81 % of Android devices as of June 2024). The main advantage of the developed assistive system is that no additional equipment is required to be installed since up-to-date Android smartphones can scan WiFi networks and identify BSSIDs employing passive reconnaissance techniques (a Java Android class WifiManager in this project). The Android platform is widely used in the market nowadays, and hence the developed mobile applications might run on over 71 % of mobile devices worldwide as of June 2024. The most likely area for further development of this study is to enhance the Java Android application by adding a navigation component and ensuring that the layout design is responsive to the size of the smartphone screen. Acknowledgements This study and the research behind it have been inspired by the active social position of the Ukrainian Association of the Blind and the Kyrgyz Society of Blind and Deaf. The authors sincerely appreciate their suggestions and assistance in the project development. References [1] WHO, Blindness and vision impairment, 2023. URL: https://www.who.int/news-room/fact- sheets/detail/blindness-and-visual-impairment. [2] WHO, Deafness and hearing loss, 2024. URL: https://www.who.int/news-room/fact- sheets/detail/deafness-and-hearing-loss. [3] F. Shahini, V. Nasr, and M. Zahabi, Friendly Indoor Navigation App for People with Disabilities (FIND), Proceedings of the Human Factors and Ergonomics Society Annual Meeting 66.1 (2022) 1922-1926. doi:10.1177/1071181322661080. [4] A. Aljarbouh, D. Zubov, A. Kupin, N. Shaidullaev, Traffic-sign Recognition for Visually Impaired Pedestrians in Kyrgyzstan: Two-keypoint SIFT/BRISK Descriptor with CameraX, in: Proceedings of the 8th International Conference on Computational Linguistics and Intelligent Systems: Intelligent Systems Workshop, Lviv Ukraine, 2024, pp. 145-156. URL: https://ceur- ws.org/Vol-3688/paper11.pdf. [5] W. Jeamwatthanachai, M. Wald, and G. Wills, Indoor Navigation by Blind People: Behaviors and Challenges in Unfamiliar Spaces and Buildings, British Journal of Visual Impairment 37 2 (2019) 140-153. doi: doi/10.1177/0264619619833723. [6] S.E. Chidiac, M.A. Reda, and G.E. Marjaba, Accessibility of the Built Environment for People with Sensory Disabilities Review Quality and Representation of Evidence, Buildings 14 3 (2024) 707. doi:10.3390/buildings14030707. [7] V. Isazade, Advancement in Navigation Technologies and their Potential for the Visually Impaired: A Comprehensive Review, Spat. Inf. Res 31 (2023) 547-558. doi:10.1007/s41324-023- 00522-4. [8] W. Ou, J. Zhang, K. Peng, K. Yang, G. Jaworek, K. Muller, and R. Stiefelhagen, Indoor Navigation Assistance for Visually Impaired People via Dynamic SLAM and Panoptic Segmentation with an RGB-D Sensor. In: K. Miesenberger, G. Kouroupetroglou, K. Mavrou, R. Manduchi, M. Covarrubias Rodriguez, P. Penáz (Eds.) Computers Helping People with Special Needs. ICCHP-AAATE 2022, volume 13341 of Lecture Notes in Computer Science, Springer, Cham. doi:10.1007/978-3-031-08648-9_19. [9] N.Y. Conteh (Ed.), Ethical Hacking Techniques and Countermeasures for Cybercrime Prevention, IGI Global, Hershey, USA, 2021. doi:10.4018/978-1-7998-6504-9. [10] A.-R. Mawlood-Yunis, Android for Java Programmers, Springer Nature Publishing, Cham, Switzerland, 2022. doi:10.1007/978-3-030-87459-9. [11] Okeenea, What You Need to Do to Ensure Accessibility for Deaf People at Public Venues, 2024. URL: https://www.inclusivecitymaker.com/accessibility-deaf-people-public-venues/. [12] Okeenea, 8 Key Points to Ensure Accessibility for Customers with Vision Disabilities at Public Venues, 2024. URL: https://www.inclusivecitymaker.com/accessibility-customers-vision- disabilities-public-venues/. [13] D. Plikynas, A. Budrionis, and M Gudauskis, Indoor Navigation Systems for Visually Impaired Persons: Mapping the Features of Existing Technologies to User Needs, Sensors 20 3 (2020) 636. doi:10.3390/s20030636. [14] Y. Bai, A Wearable Indoor Navigation System for Blind and Visually Impaired Individuals, Ph.D. thesis, University of Pittsburgh, Pittsburgh, PA, 2014. UMI Order Number: AAT 23676. [15] B. Badawi, T.N.M. Aris, M. Zolkepli, N.A. Husin, and N. C. Pa, Indoor Navigation System for Blind People Using Color QR Code, Journal of Theoretical and Applied Information Technology 99 19 (2021) 4497-4510. [16] C. Jadhav, G. Rajput, A. Harshavardhan, Systems and Methods for Performing Inclusive Indoor Navigation, 2021. Patent No. US 2021/0231440 A1, Filled Jan. 8, 2021, Issued Jul. 29, 2021. [17] N.A. Giudice, B.A. Guenther, T.M. Kaplan, S.M. Anderson, R.J. Knuesel, and J.F. Cioffi, Use of an Indoor Navigation System by Sighted and Blind Travelers: Performance Similarities across Visual Status and Age, ACM Transactions on Accessible Computing 13 3 (2020) 11. doi:10.1145/3407191. [18] V. Nair, G. Olmschenk, W.H. Seiple, and Z. Zhu, ASSIST: Evaluating the Usability and Performance of an Indoor navigation assistant for blind and visually impaired people, Assistive Technology 34 3 (2020) 289-299. doi:10.1080/10400435.2020.1809553. [19] H. Jabnoun, M.A. Hashish, and F. Benzarti, Mobile Assistive Application for Blind People in Indoor Navigation. In: M. Jmaiel, M. Mokhtari, B. Abdulrazak, H. Aloulou, and S. Kallel (Eds.) The Impact of Digital Technologies on Public Health in Developed and Developing Countries, ICOST 2020, volume 12157 of Lecture Notes in Computer Science, Springer, Cham. doi:10.1007/978-3-030-51517-1_36. [20] T. Alhmiedat, A.A. Taleb, and G. Samara, A Prototype Navigation System for Guiding Blind People Indoors using NXT Mindstorms, International Journal of Online and Biomedical Engineering 9 5 (2013) 52-58. doi:10.3991/ijoe.v9i5.2848. [21] R. Ivanov, Indoor Navigation System for Visually Impaired, in: Proceedings of the 11th International Conference on Computer Systems and Technologies and Workshop for PhD Students in Computing on International Conference on Computer Systems and Technologies, Sofia Bulgaria, 2010, pp. 143-149. doi:10.1145/1839379.1839405.