One vision that has recently been formulated for enterprise modeling states, that within some years from now on, modeling shall be embedded in our daily work practices [8]. This means, that people engage in modeling without noticing it and it becomes a common practice, just like the use of office applications today. For achieving this vision, multiple challenges must be addressed in research including adequate model formats, the context of stakeholders or the scope of models.
In the following, first research results in augmented and virtual reality (AR/ VR)-based conceptual modeling towards realizing this vision are presented. We focus mainly on the presentation and representation of models, as well as on the scope of models. This includes the analysis of everyday work practices, the identification of adequate situations for model creation and use, as well as the selection of appropriate content in particular contexts. Thereby we build upon previous work where we derived constituents of AR-based applications [6].
As a sample scenario, let us imagine a domain expert working on a task using a machine in some business process. Suppose that the person would like to know about the possible next steps in the process. In a traditional setting, this person would have to revert to a classical modeling tool and be familiar with the used modeling notation. Consider now that the person wears a head-mounted display (HMD) that automatically displays the relevant information about the process and embeds the visualization into the real world at the specific location in the form of augmented reality. This would mean that the model is embedded into the current work practice.
When analyzing this scenario, there are many aspects that must be considered for combining modeling and AR. In particular, we can revert to a previously described conceptual framework for AR and denote the different steps of the process as content and the working environment of the domain expert as context [6]. Since the different tasks should be visualized automatically, this can be considered as the interaction. As existing metamodeling approaches in the area of enterprise modeling so far do not contain AR-specific concepts, we developed a novel meta-metamodeling framework for AR/VR for realizing such scenarios. This will serve for deriving more concrete requirements in in the following.
The remainder of the paper is structured as follows: In Section 2 we briefly discuss related work. In Section 3, we will introduce the framework we developed for integrating AR/VR concepts in metamodeling. In Section 4 we present the additionally derived requirements for such an approach. The paper will end with a conclusion and an outlook to future work.
The representation of models in three-dimensional space has been investigated by several authors. As summarized in [3], previous approaches focused for example on the 3D representation of business process models, their interactive generation or the layouting of three-dimensional models. Due to technological advancement, decreasing prices for high-end devices for augmented and virtual reality applications and the availability of high-level software libraries, more recent approaches explored how to use AR/VR technologies in this context.
Abdul et al. presented an approach for visualizing BPMN collaboration models extracted from a standard file format in VR [1]. The user can insert suitable three-dimensional representations for the different elements. Subsequently, the process can be simulated and validated in VR. However, this approach is specific for a given purpose and cannot be adapted to other use cases.
Ruiz-Rube et al. presented a tool that focuses on a metamodeling approach for the creation of AR editors for domain-specific languages (DSL) [7]. Their main contribution lies on creating AR model editors for mobile devices. The metamodel is based on ECORE and extended with AR concepts. However, it lacks several concepts typically used in enterprise modeling such as decomposition, ports, or attribute specifications for nodes, edges, and model instances.
Metzger et al. designed and implemented a system for interacting with virtual process models by using smart glasses [5]. Their approach permits to create and modify process models in virtual reality, other modeling languages are however not directly supported.
In summary, there are several previous approaches that target the use of augmented and virtual reality in conceptual modeling. However, to the best of our knowledge no publications so far address this topic on the meta-metalevel in a generic way.
For developing a meta-metamodeling framework for AR and VR, we followed an exploratory and experimental research approach. We first investigated existing meta-metamodels as described for example in [4]. This permitted us to identify the relevant concepts typically used in traditional 2D metamodeling. Subsequently, we derived the concepts necessary for AR and VR representations in 3D space. This was largely influenced by the technical requirements for realizing AR and VR applications using a state-of-the-art technology stack that would run on arbitrary AR and VR devices using a web-based environment. This resulted in the meta-metamodel shown in Figure 1. The innovative aspect of this meta-metamodel is, that it can be simultaneously used for 2D and 3D modeling. Unlike previous meta-metamodeling approaches, it is however natively based on 3D space. It must be noted that the 1 only contains an excerpt of the actual constructs due to limitations of space. It is composed of a meta layer and an instance layer. This is to show the relation between the definition of a modeling language and the instantiation of the specific objects when defining a model. The main classes in the meta-metamodel inherit the general properties from the superclass metaobject. The core classes inheriting from metaobject are class, role, scene type, attribute and attribute type.
The core part comprises classes and relationclasses that are contained in one or multiple scene types. A scene type represents the closed 3D space of a model. Classes, relationclasses and scene types have attributes that are further detailed with exactly one attribute type. Classes, relationclasses and scene types can be set in relation to each other by relationclasses. Each relationclass has exactly two roles assigned. A from role and a to role. Further, each role has at least one reference to a class, relationclass or scene type, that defines, to what this role can connect. Further classes and scene types can have ports. Also to these ports, roles can be assigned.
All constructs inheriting from metaobject have a visual representation. This representation is defined with a domain-specific language called VizRep, that defines the 3D representation and behavior of an object. This information is stored in the geometry attribute in metaobject. Further, each visual object has 2D coordinates for the positioning in a 2D modeling environment, as well as relative 3D coordinates (relativeCoordinates3D) for positioning objects in AR and VR environments relative to the user position. These positions may differ from the coordinates used for the 2D screen representation. Further, each metaobject may have absolute 3D coordinates (absoluteCoordinates3D) for the positioning of objects using real world coordinates like GPS coordinates or indoor positioning information.
In the lower half of Figure 1 the instance layer of the meta-metamodel is depicted. It shows the constructs for holding information of the instances of the metamodel when instantiating a model, for example instances for class, attribute and scene instances.
For evaluating the technical feasibility of the meta-metamodel, a prototypical implementation has been created using JavaScript and WebXR1 via the ThreeJS2 library. The resulting modeling tool works entirely in a 3D environment, and not like most other modeling tools on a 2D canvas. This has the advantage, that the models can be used in a traditional 2D environment by holding the depth coordinates constant, or without changes in a 3D mode for AR and VR. An example of the browser-based modeling tool is shown in Figure 2. Further, we conducted tests by specifying subsets of BPMN 3 and ERD [2] with the new tool. Examples of these tests by using an AR HMD (Head-Mounted-Display) in the form of an MS HoloLens2 can be seen in Figures 3, 4 and 5.
With the insights gained above, we can formulate the following requirements for enterprise modeling in AR/VR. First, the technology stack underlying such modeling tools must support AR and VR, including according hardware devices. Further, the graphical representation and positioning of objects must be accomplished in 3D space. For the representation of 3D geometries, the already mentioned VizRep language can be used. This is a new and generic JavaScript function to define the visual representation of the different components, the according labels, the used attributes for the labels etc. This enables the definition of 3D objects and the specification of according labels. This has direct consequences for the interaction with models where novel types of user-machine interaction need to be used, as for example discussed in detail in [7].
Concerning the positioning of objects, an AR/VR-enabled modeling environment permits to place objects in virtual 3D space as well as attach them to real-world coordinates, e.g. to attach a task in a process or an entity type in an ER diagram to a physical machine. Thus, this information needs to be provided in addition to the traditional 2D coordinates. This leads to new types of enterprise modeling scenarios, e.g. for using enterprise models as a guidance in the style of a map in the real-world.
Further, one of the strengths of AR devices is to analyze the environment and recognize the situation of the user through different sensors. For integrating this in enterprise modeling, the properties of the context need to be inferred so that the models can be adapted to a specific situation. Again, this information about the context has to be made available for the objects in an enterprise model. As existing modeling languages do not consider such aspects, they will have to be adapted for this purpose, e.g. by a context attribute for a BPMN task.
In this paper, a first design of an AR and VR enabled meta-metamodeling framework as well as a prototype were shown. With first tests we could verify that the use of the framework and the implementation of some basic modeling languages for AR and VR is feasible. In further work we will extend the framework and the implementation. In particular, we will address interaction techniques and the positioning of models and their elements using real world coordinates, as well as the integration of situational context in AR.