Discoverable and Interoperable Augmented Reality Environments Through Solid Pods Maxim Van de Wynckel∗ , Beat Signer Web & Information Systems Engineering Lab, Vrije Universiteit Brussel, 1050 Brussels, Belgium Abstract Augmented Reality (AR) environments are physical environments with virtual objects superimposed through AR-enabled devices. These virtual objects can range from simple aesthetic objects like pictures to superimposed contextual information about physical items. In most modern AR applications, the augmented spaces exist only for the user who created the environment or for proprietary applications that enable multi-user collaboration in the same environment. However, there is a lack of solutions that enable interoperable collaboration in these personal AR spaces, allowing users to share and contribute to an AR space. We propose a solution that enables users to create their personal AR space that can then be discovered by other users who are in physical proximity to this space, enabling them to view or contribute to the augmented space. In addition, we discuss a solution that utilises the same technique to create AR spaces that are bound to a specific room and can be discovered by users who are in close vicinity to these rooms. Keywords Solid, Augmented Reality, Physical Pod Discovery, AR Collaboration 1. Introduction and Related Work In Augmented reality (AR), virtual objects are superimposed onto the physical world. Since its early days, AR has significantly advanced to portable devices such as smartphones and head-mounted displays [1], enabling the use in regular environments such as office buildings to superimpose virtual information to the physical world. Superimposed virtual objects or information can be anchored to physical objects and walls within these physical environments [2, 3]. These virtual objects might range from textual information such as timers or instructions to images, videos or interactive elements [4]. However, there is a lack of AR solutions that enable collaboration on the creation of these AR environments without the need for proprietary applications. A conceptual content-sharing solution for AR, offering a peer-to-peer and client-server collaboration solution using events that indicate changes to the virtual objects, was proposed by Ruth et al. [5]. While their solution supports collaboration in AR, it still requires a complex and proprietary peer-to-peer communication framework and does not tackle the problem of different devices using a different frame of reference [6]. The 1st Solid Symposium Poster Session, co-located with the 2nd Solid Symposium, May 02 – 03, 2024, Leuven, Belgium Envelope-Open mvdewync@vub.be (M. Van de Wynckel); bsigner@vub.be (B. Signer) GLOBE https://maximvdw.be/profile/card (M. Van de Wynckel); https://beatsigner.com (B. Signer) Orcid 0000-0003-0314-7107 (M. Van de Wynckel); 0000-0001-9916-0837 (B. Signer) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings In the i-VISION project [7], AR collaboration and interaction are enabled by creating a centralised publish and subscribe REST API. The project is built on top of the Data-Fu [8] framework for the downloading and querying of data from the REST gateway. While i-VISION offers a method for multiple users to interact within an environment, the centralised nature of the gateway limits the interoperability and scalability of AR applications. We propose an interoperable solution that enables users to create their personal AR en- vironments that can be discovered in the physical world by other users using AR-enabled devices. Unlike other existing work such as XSpace [9], we aim for a synthetic and semantic interoperable solution that decentralises the collaborative aspect of AR environments while also enabling the sharing of crucial environmental information such as common reference spaces. Our interoperable solution should enable the unambiguous description of the virtual environment in a way that other applications can access and understand. In this paper, we focus on the syntactic and processing interoperability [10, 11] of virtual environments. The semantic interoperability of virtual objects within environments through common vocabularies is limited to the OMG and FOG vocabularies to illustrate the basic concept of the relative positioning of virtual objects. However, various other ontologies exist to represent synthetic 3D content [12]. 2. Solution Our proposed solution aims to allow multiple AR devices to contribute to a single shared AR en- vironment or virtual space belonging to a user. We assume that the AR device used to contribute to augmented environments is a smart device with access to the Web and can broadcast an RF signal. In the general architecture of our proposed solution, we let AR devices broadcast a semantic Bluetooth Low Energy (BLE) beacon advertisement containing the URI of a specific resource. We utilise the advertising protocol of SemBeacon [13] that advertises an AltBeacon advertisement and Eddystone-URL compatible scan response to broadcast semantic data URIs. In addition, SemBeacon has a BLE v5.0 advertisement that prevents the need to shorten the URI to fit within the Packet Data Unit (PDU). The use of SemBeacon to advertise the environ- ment URI is illustrated in Figure 1. This environment resource contains information about the personal environment(s) owned by the user. Other devices can receive these advertisements when in proximity to the AR device and then access the URI to retrieve more information. For each environment, we have a link to a public inbox that other users can use to link their own modifications to the environment. Any modifications made to the superimposed space are stored in a Solid Pod owned by the user who made the modifications, which enables users to both, contribute to the same environment as well as control the access rights of modifications made to these environments. On the left-hand side of the architecture shown in Figure 1, we have a Solid Pod for user 1. The Pod contains all environments owned or modified by the user. An AR device connects to the user’s Solid Pod through a Solid application that authenticates the user, allowing it to modify the resources when editing a virtual environment. In order to enable the discovery of these virtual spaces, the AR device broadcasts the *.ttl file of the environment it is currently in via BLE advertisements. This resource contains all the information about the environment, such as its location, any identifiable features and all virtual objects placed relative to this environment. Pod User #1 Pod User #2 environments/ environments/ home/ Solid App desk/ ... ... office/ user1-office/ data.ttl.acl data.ttl.acl User #1 User #2 data.ttl data.ttl Read inbox/ inbox/ lanoitpO .acl Read, Append .acl Read, Append 1-create.ttl Advertising Advertising ... 2-create.ttl URI of URI of user1- 3-move.ttl office/data.ttl office/data.ttl Figure 1: A user’s discoverable AR environments with two example users (#1 and #2) having a Solid Pod While the broadcasted URI is public, the access to read this URI is controllable through access control lists implemented in Solid. SemBeacon offers a broadcasted flag to indicate whether a URI is publicly accessible to prevent applications that do not have access to the resource from attempting to access it. When another user (e.g. user 2) wants to modify the environment of user 1, they create a new resource including the modifications and additions to virtual objects or detectable features (e.g. markers). The application will then notify user 1 about these changes by referencing the user1-office/data.ttl file in the inbox [14] container of the environment that is being modified. Solid also enables the access rights of these modifications, more information about access control for environments and modifications is provided in Section 2.3. environments/ <> a seas:Room ; rdfs:label "Our Lab"@ en ; restaurant/ ldp:inbox <./inbox/> ; lab/ vcard:address [ ... ] . data.ttl.acl :printer_marker a fidmark:AruCo ; fidmark:markerIdentifier 12 . data.ttl :printer_info a sosa:FeatureOfInterest ; inbox/ poso:hasPosition [ .acl poso:isRelativeTo :printer_marker ; 1-create.ttl ... ] ; Advertising 2-modify.ttl URI of lab/data.ttl omg:hasGeometry [ ... ] . Figure 2: Single discoverable AR environment using the seas a , fidmark b , poso c and omg d vocabularies a Smart Energy Aware Systems Ontology: https://w3id.org/seas/ b Fiducial Marker Ontology: http://purl.org/fidmark/ c Positioning System Ontology: http://purl.org/poso/ d Ontology for Managing Geometry: https://w3id.org/omg# An alternative architecture is illustrated in Figure 2. In this scenario, a fixed Bluetooth beacon is placed in a room, broadcasting the URI of a single environment. This scenario can be used for public physical environments, such as a meeting room or laboratory, enabling collaboration in AR. Similar to personal environments shown in Figure 1, users store their changes to an environment in their Solid Pod and reference these changes in the inbox of the environment. 2.1. Usage Our solution is depicted in Figure 3 where we showcase the flow of our architecture previously illustrated in Figure 1. Two users with AR devices have their own Solid Pod. User 1 will create an environment (𝐴) on their Pod and subscribe to the inbox container of this environment. Once the environment is ready, the AR device will use the SemBeacon specification to advertise the URI of the environment and enable its discovery. User #1 Solid Pod User #2 Solid Pod Create Environment A Subscribe to inbox of Environment A Broadcast Environment A URI via SemBeacon Create rf Subscribe to inbox Environment A' of Environment A Modify Environment A' Add modification to inbox of Environment A Notify user of modification(s) Figure 3: Interaction flow of two users contributing to the same augmented reality environment When another user (e.g. user 2) discovers the resource URI, the AR application will access the environment to visualise the augmented objects. If user 2 makes a modification such as adding virtual objects, these modifications are stored in the Pod of user 2 as environment 𝐴′ , ensuring ownership of this contribution and enabling user 2 to choose the access rights to this modification. To keep up-to-date with changes in the original environment, the application will listen for changes in the inbox of environment 𝐴. All users who are subscribed to this inbox will receive linked data notifications [14] whenever a new modification is added. Unlike the i-VISION [7] project which uses RESTful polling to detect changes in the publish and subscribe gateway, linked data notifications can enable real-time updates without the additional data overhead introduced in polling. An inbox uses the LDP1 vocabulary to index all resources within this container. A user who wishes to accept contributors for their environment should configure this inbox container with public append access rights, giving other users the opportunity to append a new resource to the container. Each inbox item represents an action using the schema.org vocabulary [15]. An 1 https://www.w3.org/ns/ldp# action represents an event that occurred in the environment, such as creating a new virtual object, moving an object or the interaction with a virtual object. @prefix card: . @prefix office: . <> a schema:CreateAction ; schema:description "Created a new object 'painting' with label 'User 2'"@ en ; schema:agent ; # AR application that created the action schema:creator card:me ; # Owner of the modification schema:object office:painting ; schema:result office:painting . Listing 1: Inbox item to identify the creation of a virtual object Listing 1 demonstrates an individual appended inbox item to the Solid Pod of user 1. In the example we see a schema:CreateAction indicating that user 2 created a new painting object. Users can listen for notifications in the inbox to automatically apply changes to virtual objects in the shared AR environment as they are made [14]. The positioning of virtual objects is done using the POSO ontology [16], which allows the de- scription of (virtual) objects to be placed relative to other objects using the poso:isRelativeTo predicate. When a user wants to place an object in an environment using an absolute position that is not relative to a marker or detectable feature, the same predicate can be used to indicate that the absolute position is relative to the environment. The final AR environment, combining all modifications made by other users, is created by the AR application that has access to the Solid Pod. Our architecture allows users to easily ignore all contributions of another user or agent. Individual modifications or contributions can be rejected by ignoring the individual inbox items where these changes are referenced. Future work might address the moderation of individual contributions by applying quality assessment crowdsourcing techniques [17]. 2.2. Reference Frame Augmented reality uses one or more reference frames [6] to anchor virtual objects in the physical environment and to determine an absolute or relative position. This can be done through feature detection [2] that creates anchors based on visual patterns or artificial features such as fiducial markers [3]. In order to enable interoperable applications to contribute to the same environment, all applications need to operate within the same reference frame. Our solution assumes that con- tributions to an environment not only include virtual objects, but also additional anchor points such as markers and detectable features contributed by multiple users [18]. By semantically describing these reference frames as visual landmarks rather than a frame of reference, the POSO ontology can be used to position virtual objects relative to these anchors. 2.3. Access Control Solid enables users to control the access rights to their data and resources. In the scope of our solution, this entails the data shared in the AR environment (i.e. virtual objects and anchors) by both, the user who owns the environment and users who make their own modifications to the environment. Each modification to an environment is made in the Solid Pod of the user who made the modification, giving them full control of the access rights to these modifications, as well as control on the actions which are usually created in the original environment to notify the original owner of the environment (see Section 2.1). 3. Conclusion and Future Work We presented a solution enabling the collaboration and sharing of personal augmented reality environments by using Solid Pods. Our solution allows these personal AR spaces to be discov- erable within the physical space that they are augmenting by means of semantic beacons that advertise a URI. The URI directs to the environment that is currently being shared by a user. This user has full control over who can access the environment or who can contribute their changes. While our proposed solution allows users to choose which collaborators to accept for the environment through the inbox system, it is currently impossible to accept individual contributions from a single collaborator. Future work might address this issue by applying crowdsourcing techniques [17] to the data that is produced by other users. In this paper, we focused on the passive discovery of AR environments using BLE advertise- ments to enable seamless discovery of the linked data. Similar architectures can be designed using other discovery methods such as a QR code on the door of the environment or network discovery protocols such as UPnP [19]. In the future, we might compare the performance between keeping multiple versions of a virtual environment in different Pods (as proposed in our solution) and the use of a stream of events indicating changes to the environment, similar to the work proposed in [20]. References [1] G. Kipper, J. Rampolla, Augmented Reality: An Emerging Technologies Guide to AR, 1st ed., Syngress Publishing, 2012. [2] W. S. Lages, D. A. Bowman, Walking With Adaptive Augmented Reality Workspaces: Design and Usage Patterns, in: Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI 2019), Los Angeles, USA, 2019. doi:10.1145/3301275. 3302278 . [3] M. Kalaitzakis, B. Cain, S. Carroll, A. Ambrosi, C. Whitehead, N. Vitzilaios, Fiducial Markers for Pose Estimation: Overview, Applications and Experimental Comparison of the ARTag, AprilTag, ArUco and STag Markers, Journal of Intelligent & Robotic Systems 101 (2021). doi:10.1007/s10846- 020- 01307- 9 . [4] D. Rumiński, K. Walczak, Semantic Contextual Augmented Reality Environments, in: Proceedings of the International Symposium on Mixed and Augmented Reality (ISMAR 2014), Munich, Germany, 2014. doi:10.1109/ISMAR.2014.6948506 . [5] K. Ruth, T. Kohno, F. Roesner, Secure Multi-User Content Sharing for Augmented Reality Applications, in: Proceedings of the 28th USENIX Security Symposium (SEC 2019), Santa Clara, USA, 2019. URL: https://www.usenix.org/conference/usenixsecurity19/presentation/ ruth. [6] W. Mou, F. Biocca, C. B. Owen, A. Tang, F. Xiao, L. Lim, Frames of Reference in Mobile Augmented Reality Displays, Journal of Experimental Psychology Applied 10 (2004). doi:10.1037/1076- 898X.10.4.238 . [7] T. Käfer, A. Harth, S. Mamessier, Towards Declarative Programming and Querying in a Distributed Cyber-Physical System: The i-VISION Case, in: Proceedings of the 2nd International Workshop on Modelling, Analysis, and Control of Complex CPS (CPS Data 2016), Vienna, Austria, 2016. doi:10.1109/CPSData.2016.7496418 . [8] S. Stadtmüller, S. Speiser, A. Harth, R. Studer, Data-Fu: A Language and an Interpreter for Interaction With Read/Write Linked Data, in: Proceedings of the 22nd International Conference on World Wide Web (WWW 2013), Rio de Janeiro, Brazil, 2013. doi:10.1145/ 2488388.2488495 . [9] J. Herskovitz, Y. F. Cheng, A. Guo, A. P. Sample, M. Nebeling, XSpace: An Augmented Reality Toolkit for Enabling Spatially-Aware Distributed Collaboration, Proceedings of the ACM on Human-Computer Interaction 6 (2022). doi:10.1145/3567721 . [10] K. H. Veltman, Syntactic and Semantic Interoperability: New Approaches to Knowledge and the Semantic Web, New Review of Information Networking 7 (2001). doi:10.1080/ 13614570109516975 . [11] W. A. Khan, M. Hussain, K. Latif, M. Afzal, F. Ahmad, S. Lee, Process Interoperability in Healthcare Systems With Dynamic Semantic Web Services, Computing 95 (2013). doi:10.1007/s00607- 012- 0239- 3 . [12] J. Flotyński, K. Walczak, Ontology-Based Representation and Modelling of Synthetic 3D Content: A State-of-the-Art Review, Computer Graphics Forum 36 (2017). doi:https: //doi.org/10.1111/cgf.13083 . [13] M. Van de Wynckel, B. Signer, SemBeacon: A Semantic Proximity Beacon Solution for Discovering and Detecting the Position of Physical Things, in: Proceedings of the 13th International Conference on the Internet of Things (IoT 2023), Nagoya, Japan, 2024. doi:10.1145/3627050.3627060 . [14] S. Capadisli, A. Guy, C. Lange, S. Auer, A. Sambra, T. Berners-Lee, Linked Data Notifications: A Resource-centric Communication Protocol, in: Proceedings of the 14th International Conference on The Semantic Web (ESWC 2017), Portorož, Slovenia, 2017. doi:10.1007/ 978- 3- 319- 58068- 5_33 . [15] A. Iliadis, A. Acker, W. Stevens, S. B. Kavakli, One Schema to Rule them All: How Schema.org Models the World of Search, Journal of the Association for Information Science and Technology (2023). doi:https://doi.org/10.1002/asi.24744 . [16] M. Van de Wynckel, B. Signer, POSO: A Generic Positioning System Ontology, in: Proceedings of the 21st International Semantic Web Conference (ISWC 2022), Virtual conference, 2022. doi:10.1007/978- 3- 031- 19433- 7_14 . [17] M. Acosta, A. Zaveri, E. Simperl, D. Kontokostas, S. Auer, J. Lehmann, Crowdsourcing Linked Data Quality Assessment, in: Proceedings of the 12th International Semantic Web Conference (ISWC 2013), Sydney, Australia, 2013. doi:10.1007/978- 3- 642- 41338- 4_17 . [18] M. Van de Wynckel, I. Valadez, B. Signer, FidMark: A Fiducial Marker Ontology for Seman- tically Describing Visual Markers, in: Proceedings of the 21st International Conference on the Semantic Web (ESWC 2024), Hersonissos, Greece, 2024 (In Press). [19] J. M. S. Santana, M. Petrova, P. Mahonen, UPnP Service Discovery for Heterogeneous Networks, in: Proceedings of the 17th International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC 2006), Helsinki, Finland, 2006. doi:10.1109/PIMRC. 2006.254286 . [20] W. Slabbinck, R. Dedecker, S. Vasireddy, R. Verborgh, P. Colpaert, Linked Data Event Streams in Solid LDP Containers, in: Proceedings of the 8th Workshop on Managing the Evolution and Preservation of the Data Web (MEPDaW 2022), Online, 2023. URL: https://ceur-ws.org/Vol-3339/paper4.pdf.