The objective of the MIoTA (Modeling IoT Applications for Air Conditioning Facilities) modeling method is to facilitate the configuration of Internet of Things (IoT) applications in the field of air conditioning facilities. The corresponding MIoTA tool incorporates a domain-specific modeling language (DSML) and its meta-model for air conditioning facilities and was developed in an industrial case study. Various functionalities support the modeling and development processes. Therefore, no specific IT skills are required at the application level. It is important that models created with MIoTA are readily accepted and correctly understood. This necessitates the use of an intuitively understandable notation. The current MIoTA tool and notation have been formally tested by domain experts who assisted in their development. In this paper, we conduct an experiment to assess the intuitiveness of working with the tool with users who have no prior experience with it and are no experts in the domain. The majority of the participants perceived MIoTA as useful and intuitive, and the results allow us to derive potential improvements.
The evaluation described in this paper aims to evaluate to what extent the MIoTA tool facilitates the creation of accurate models and the entry of configuration data by users who are no experts in the domain and have no prior experience with the tool. These tasks were conducted by diferent participants to test the intuitiveness of the MIoTA tool. Based on an analysis of the results, improvements to the current tool version are proposed.
The paper is structured as follows: Section 2 presents an introduction to the subject of quality evaluation in conceptual modeling and the MIoTA modeling method. Subsequently, section 3 defines the methodology employed for the evaluation and data analysis. In section 4, the results of the analysis are discussed. This led to proposals to improve the MIoTA tool in section 5. The paper concludes with a reflection and a summary of the key findings in section 6.
This section presents the subject of quality evaluation in conceptual modeling and introduces the MIoTA modeling method.
It is of great importance to have accurate representations in order to gain a deeper understanding of
the highly complex problem domains that exist within today’s organizations. Conceptual modeling is
a technique that creates representations and abstractions that remove much of the complexity found
in real-world problem domains [
A multitude of modeling languages have been developed for general applicability and wide adoption.
For example, the Unified Modeling Language (UML) [
The evaluation of quality in conceptual modeling can be conducted with an emphasis on diferent
perspectives of "quality". These include the quality of modeling methods [10], the quality of the modeling
result [11], the quality of the modeling process [12], and the quality levels of conceptual modeling [
From the perspective of our research, the model itself and its usage are of primary interest. This motivates the selection of a quality framework that integrates various quality aspects with an emphasis on the modeling result. The semiotic model quality framework (SEQUAL) [17] was selected for evaluation as it possesses three properties that are pertinent to the research in question. (1) SEQUAL ofers the potential to diferentiate between quality characteristics and the means of attaining these characteristics (goals). This enables us to examine the requirements of the model in the context of air conditioning facilities and how they were operationalized. (2) SEQUAL addresses quality on various semiotic levels, including syntax, semantics, and pragmatics. It is essential to incorporate the modeling language as such and also to examine its usage in the case study and its support by the tool. (3) Moreover, SEQUAL recognizes that models are frequently developed through collaboration between those engaged in modeling, whose understanding of the modeling domain evolves as modeling progresses. IoT development in the field of air conditioning facilities, to our experience, involves diferent stakeholders in such a collaboration process.
The MIoTA modeling method1 was initially introduced in [18] and subsequently developed in [19] in
the context of an industrial case study involving a small and medium-sized enterprise (SME) operating
within the air conditioning facilities sector. The objective was to enhance the energy eficiency of air
conditioning facilities by minimizing the number of sensors required. An examination of inspection
reports indicated that the implementation of low-cost technologies could result in energy savings of up
to 30% in the majority of facilities. Other research has confirmed these potential savings, which are
made possible by the straightforward identification of malfunctions in these facilities [ 20, 21]. In order
to facilitate the configuration for non-IT users, a method comprising the preparation, construction,
and application of IoT applications is proposed, and a tool has been developed with the ADOxx
metamodeling platform2 [22] to support this. MIoTA is a modeling method that facilitates the development
of IoT applications in the domain of air conditioning facilities. It adopts the definition of modeling
methods proposed by Karagiannis and Kühn [
The MIoTA tool comprises a DSML for air conditioning facilities developed in an industrial case 1https://www.omilab.org/MIoTA/ 2https://www.adoxx.org/ study. A visual notation (see Figure 1) was developed in accordance with the meta-model (see [19]) and is accessible within the MIoTA tool. It contains visual representations for the diferent Components, Air Types, Sensors, and MIoTA Relations to model air conditioning facilities. In addition, there are certain objects that are not directly related to the model; these are designated as Others. The object Configuration Data permits the representation and modification of configuration data during the modeling process. Notes allows the recording of remarks or tasks. The Room object allows for the visual grouping of components into a room.
A variety of functionalities are available to support the modeling and development processes, thereby obviating the need for specific IT skills at the application level. The DSML enables employees to model the facilities in accordance with their domain expertise. In order to address the diversity of facilities and facilitate energy savings when evaluating comparable configurations, it is essential to gather essential data from each facility. This configuration data encompasses information such as the operator, the components and their respective specifications, and the sensors. A familiar interface also allows users to enter the required configuration data and classify the facilities. The configuration data input into the tool automatically generates the requisite modeling objects. Subsequently, the user must supplement the model by incorporating additional components and establishing the requisite relations between them. The data can be exported (for example, sent directly to the AWS cloud as in our use case) to configure services and prepare visualizations and subsequent data analysis, thereby enabling energy optimization. In the context of the industrial case, we validated the modeling language and the way it can be used and supported by the tool. Previous work also applied quality criteria for evaluation.
This section describes our overall evaluation approach, summarizes which part of it has already been conducted, and describes what part of the evaluation is performed in this work.
As explained in section 2, we decided to use SEQUAL [17] as a general framework for the quality evaluation of MIoTA. SEQUAL proposes a distinction between seven diferent aspects of model quality: • Physical quality addresses the fact that the manner in which the model is presented (i.e., its externalization) is accessible to those who utilize the model. • Syntactic quality refers to the coherence between a model and the modeling language that is used for modeling. • Semantic quality refers to the correspondence between a model (or its meta-model) and the modeling domain (meaning of concepts in a domain has to be equivalent to the corresponding concepts in the model). • Empirical quality compares diferent models created by a modeler to express the same understanding but implemented diferently. • Pragmatic quality can be divided into social and technical pragmatic quality. Social pragmatic quality refers to how well the model is understood by human actors, comparing the modeler’s intended understanding of the model with the model user’s actual understanding. Technical pragmatic quality defines to what extent tools can’t interpret the model. • Social quality addresses the question of whether actors agree on the interpretation of the model. • Deontic quality investigates if all elements of a model contribute to fulfilling the goals of modeling and if all goals of modeling are addressed through the model.
In a previous work [23], we developed an approach to examining these aspects in our evaluation of MIoTA. Table 1 provides a summary of the evaluation approach selected for each quality aspect, indicating whether and how it has been implemented so far. The main part of the evaluation has been done in [23]. An experiment that included a modeling task and expert interviews was conducted with domain experts who were involved in developing the MIoTA modeling method. Further, we applied Moody’s "Physics" of Notations [24] to assess the physical quality of our DSML.
In [25], we ensured that models created with our DSML are able to represent the envisioned meanings of the domain correctly by interviewing domain experts and implementing the findings in the metamodel, and also ensuring the coherence between the DSML and the meta-model. While the coherence with physical, syntactic, semantic, perceived semantic, technical pragmatic, and deontic quality has been confirmed so far, the evaluation yielded only partial confirmation of both empirical and social (pragmatic) quality, which in turn gave rise to the proposal of a comparison of models created by diferent modelers and the involvement of additional actors. As the MIoTA tool is already used in practice in our case study company, we could see that further employees (domain experts) who were not involved in the development of the tool and DSML can use the tool in a straightforward manner and with minimal instructions. To investigate the empirical quality further, in this paper, we conduct an evaluation to test the intuitiveness of MIoTA with users who are no experts in the domain and have never worked with the tool before. We also assess how well the created models are understood, comparing the modeler’s intended understanding of the model with the model user’s actual understanding (social (pragmatic) quality).
Moody defines semantic transparency as the extent to which an inexperienced reader can infer the meaning of a symbol from its appearance alone [24]. In the literature, semantic transparency is often regarded as a synonym for intuitive understanding. A notation with high semantic transparency allows users to infer the meaning of a symbol or model from their working and/or long-term memory. It thus follows that semantic transparency is of great importance with regard to the acceptance of modeling languages. The aim of our investigation is to ascertain the extent to which this is the case in our particular case and to determine the extent to which the MIoTA tool supports this.
In light of the open-ended nature of the evaluation outlined in section 3.1, we elected to conduct an additional assessment from the vantage point of domain and modeling experts. This assessment was undertaken with two distinct objectives in mind: • We wanted to investigate the ability of users who were no experts in the domain and had no prior experience with the tool to create models. • We wanted to investigate how the tool could be improved based on the experiences of the users involved in the experiment.
The fundamental approach to examining both aspects was to engage a domain expert in the development of a sample solution (referred to as the "gold standard", illustrated in Figure 2 and Figure 3) for a specific modeling task. This sample solution was then evaluated for its accuracy, completeness, and correctness. The domain expert, with over a decade of experience in the field of air conditioning facilities, collaborated with the modeling experts (who developed the MIoTA modeling method) to create a model for a fictitious facility with realistic values.
The aforementioned evaluation strategy was implemented in four stages: (i) the initialization of evaluation tasks (section 3.3), (ii) the construction of the sample models (section 3.4), (iii) the performance of the evaluation tasks (section 3.5), and (iv) the discussion of the results (section 4). For the evaluation of the intuitiveness, the participants were invited via email containing the download link for the tool, exhaustive instructions on how to conduct the experiment, the necessary material for the experiment, and a link to the questionnaire. Based on an analysis of the results, improvements to the current version of the MIoTA tool are proposed. The experiment was conducted in accordance with the following procedure: 1. Preparation: The participants were provided with a brief introduction to the pertinent domain, namely IoT applications for air conditioning facilities, as well as an explanation of the rationale behind the necessity of the MIoTA tool. It should be noted that a visual aid was employed to allow the participants to connect the components and sensors as needed (not a comprehensive explanation of all elements). The participants were thus furnished with the requisite domain knowledge to enable them to conduct the experiment. 2. Study the MIoTA manual: The participants were provided with a manual for the tool to get the relevant information for installing the tool and how to start the modeling process with the input of configuration data. They were also asked beforehand to check how to use the functionalities relevant to the experiment. 3. Model creation and upload: The main part of the experiment was then to start the modeling process by entering the configuration data. Based on this, the participants needed to refine some data (as part of checking the diferent possibilities in the tool) and connect and arrange the objects in a given manner via relations. In the end, the export functionalities were tested, and the model ifles were uploaded. 4. Questionnaire: After conducting the experiment, the participants were asked to complete a questionnaire. It started with gathering demographic information (e.g., gender and age) and their level of education and profession/role. Then, the participants were asked to assess statements about their experiences while modeling and regarding the diferent functionalities of the tool. A 5-point Likert scale was used for most of the questions (1 = strongly disagree/very dissatisfied, 5 = strongly agree/very satisfied). Some questions were to be answered with yes or no. To ascertain the participants’ opinions regarding the most beneficial and least beneficial aspects of the tool, as well as improvement suggestions, a series of open questions were incorporated into the questionnaire.
In the context of the specified modeling task, the participants of the experiment are required to create a model for the example facility based on the provided material, enter the specified configuration data, and complete the model by adding objects and connecting them with relations. As the participants were only provided with the domain knowledge necessary for conducting the experiment and were not previously familiar with the specific notation, the arrangement of the objects and the relationships between them were predetermined. In light of these considerations, the following guiding questions (GQs) were posed to inform the evaluation of the modeling results: • GQ1: To what extent does the MIoTA tool allow the creation of accurate models by users who are no experts in the domain and have no prior experience with the tool? • GQ2: To what extent does MIoTA assist users who are no experts in the domain and have no prior experience with the tool in accurately entering configuration data?
In this evaluation episode, the two diferent types of facilities that are divided in practice were modeled so that the automatic creation of the diferent objects and sensors for each type could be evaluated. However, it should be noted that not all possible object types were tested. Nevertheless, this approach allowed for the testing of the two model types, which difer in their overall complexity. For the evaluation, it was deemed essential that the (i) required objects be correctly represented in the model, (ii) that these be correctly connected using the correct relations (Air Flow Direction and Relation), and (iii) that the configuration data have been entered accurately, completely, and correctly.
17 participants started the experiment, of which 14 (13 Windows users and one macOS user) completely finished the questionnaire and successfully uploaded the results. In the evaluation presented in section 3.5, the results of only the 14 participants are considered, seven of whom completed the tasks of groups A and B, respectively. One participant each stated high school and Bachelor’s degree, and twelve participants stated Master’s degree as the level of education. The participants stated roles/professions were academic/researcher (ten), IT specialist (two), student (one), and other (one).
In order to model the sample solution, it was decided that one model should be taken for each possible type of facility. This means we have to determine two diferent groups of participants. Group A comprises a facility that has dehumidification or humidification, while group B is the same facility with no dehumidification or humidification. This is reflected in the model by the fact that group A has more configuration data to be entered, more automatically created objects and sensors, and thus more relations (Relation and Air Flow Direction) to create. Two models were developed for this process. The number of elements contained in each model and the number of data to be entered are shown in Table 2.
Figure 2 and Figure 3 illustrate the models for the quality check (gold standard). They illustrate the various components (e.g., heat recovery or air filter), air types (e.g., exhaust air or supply air), and sensors. The sensors are represented by circles, with the corresponding type of sensor and the physical unit indicated (t = temperature (∘ C), rh = relative humidity (%), and CO2 = CO2 content (ppm = parts per million)) and the name underneath. The name is composed of the position (e.g., SUP = supply air, ODA = outdoor air, or ETA = exhaust Air), the type (ai = analog input), and the physical size. The components are connected with Air Flow Direction, which also indicates the air flow direction in the model. Sensors are connected to the component to which they are attached using Relation.
Group A had to create a model containing eleven components, eleven sensors, and 21 relations (eleven Relation and ten Air Flow Direction). 55 values needed to be entered for the configuration data.
Group B had to create a model with ten components, eight sensors, and 17 relations (eight Relation and nine Air Flow Direction). 50 values needed to be entered for the configuration data.
For GQ1 (create a correct model), we gave the necessary information for modeling each group A and B correctly to the participants. The identical models were employed for GQ2 (enter configuration data complete and correct). We provided a table containing the relevant values for the input of the configuration data. The idea behind dividing the two groups for the evaluation was to consider all scenarios that occur in praxis by using the MIoTA tool. We also wanted to compare if the complexity regarding the number of elements or inputs for the configuration data plays a role in the results.
For the evaluation of the accuracy, completeness, and correctness of the created models, the modeling experts checked the components, sensors, relations (Relation and Air Flow Direction), and configuration data. Table 3 summarizes the percentage of complete and correct elements and entered configuration data in the models created by the participants in total and separately for groups A and B.
In the models created by the participants, 94.5% of the components and 98.7% of the sensors were correctly created and placed in the model. With regard to the connection of sensors with components, 95.6% of the relations (Relation) were correctly used, while 95.2% of the connections between components (Air Flow Direction) were also correctly established. The accuracy of the configuration data input is 97.4%. Table 4 is a summary of answers to the questionnaire that are relevant to our GQs. It contains the responses to the most significant statements and questions, as well as the issues and positive aspects mentioned by the participants in the open questions. The issues highlighted primarily concern the design of the tool, which in some cases resulted in dificulty in finding relevant functionalities directly. Furthermore, some issues were reported with the experimental instructions, which may have introduced errors in the modeling results. It was repeatedly highlighted that the input of the configuration data takes over a large part of the modeling and avoids errors.
In light of the findings from the evaluation tasks, it may be beneficial to revisit and address the initial questions guiding the evaluation:
GQ1: To what extent does the MIoTA tool allow the creation of accurate models by users who are no experts in the domain and have no prior experience with the tool?
We chose to let users create models that are no experts in the domain and have no prior experience with the MIoTA tool to execute some tasks. The models created allow us to hypothesize about the potential causes of the failures that occurred. As shown in Table 3, the created models achieved 94,5% accuracy for components. This is a highly favorable outcome, particularly given that eleven of the fourteen models were free of errors regarding the components. The errors that were identified included the failure to add objects that were not created automatically, as well as the deletion of objects that were automatically created by one participant of group B. The accuracy regarding the relations (95.6% for Relation and 95.2% for Air Flow Direction) is also pretty good. The failures here were that sensors were placed and connected to the wrong components. Only one participant made an error while creating the correct sensors (forgot to confirm the creation of CO2 content sensors), so an accuracy of 98,7% was achieved here. The results of the questionnaire show that the overall handling of the tool is good, and it is helpful that most of the modeling objects are created automatically based on the configuration data. It is also stated that this way, errors are reduced. It was easy to navigate through the diferent sections of the MIoTA tool.
The icons and labels used in the MIoTA tool are clear and understandable.
I believe the created model and entered data are correct and complete.
It was easy to find the information needed in the MIoTA manual.
The process of creating a new model in the MIoTA tool was easy.
It was easy to enter initial configuration data when creating a new model.
It was easy to install the tool.
Question (1=very dissatisfied, 5=very satisfied) Overall, how satisfied are you with the MIoTA tool? Overall, how would you rate the usability of the MIoTA tool? How would you rate the overall satisfaction with the MIoTA manual? Issues The position of relations needs to be adjusted by hand every time Bar with modeling elements is confusing Some functionalities are hard to find/only after referring to the manual Menu icons are too small The design could be newer and fresher Positive aspects The functionality was perceived as pretty good Automatic creation of modeling object The procedure reduces errors made during data input and modeling Good possibilities to browse all information about the model The immediate impact/visibility after entering the configuration data Placing the modeling objects by drag-and-drop is straightforward Guided query of the data saves time and avoids entering redundant data 1 1 2 1
GQ2: To what extent does MIoTA assist users who are no experts in the domain and have no prior experience with the tool in accurately entering configuration data?
97.4% correct entered configuration data is also a highly satisfactory outcome. Based on the entered configuration data, as illustrated in the model, it is possible to ascertain the circumstances surrounding the occurrence of some of the failures. Five participants failed to enter a value for preheater power. This value was intended to be entered via the notebook after the regular configuration data had been input. The relevant information was provided during the experiment’s construction. It is possible that this information was not adequately described, resulting in the five participants forgetting to enter this value. The remaining failures were the result of typographical errors, the input of incorrect numbers, or the omission of values.
In conclusion, we put forth recommendations for enhancing the MIoTA tool. This proposal is founded upon the collective assessment outcomes that were previously outlined. In particular, we diferentiate between (i) alterations necessitated by the modeling outcomes and (ii) modifications proposed or resulting from the questionnaire. These changes may encompass the expansion of the tool’s functionalities, the provision of additional assistance throughout the process, or the redesign of the interface to enhance comprehension. It is important to note that not all of the potential improvements identified can or will be implemented. This may be due to the fact that some aspects are predetermined by the practical use case and, therefore, cannot be altered or that there are technical limitations inherent to the meta-modeling platform we are using.
Based on the modeling results, we got the idea that the orientation of the modeling objects could be automatically set depending on the direction of the Air Flow Direction. This would further ease the modeling process and avoid mistakes in manually setting up the orientation. The orientation is only a visual thing in the model and was thus not part of our evaluation. The questionnaire led to the idea that the automatic position of created relations could be improved to reduce time in manually adjusting them, also leading to a better-looking model. Some participants stated that the input of the configuration data takes a lot of time and should possibly be much faster. Based on our observations of this process in practice, we have found that it greatly reduces the complexity of the procedure. Therefore, we do not intend to modify the process itself but rather implement changes based on suggestions regarding the comprehensibility of the process. It is possible that our experimental instructions were not suficiently detailed to ensure that all participants fully understood the procedure. Some participants expressed dissatisfaction with the interface and clarity, and these are also points that we intend to address in future revisions. The structure can be readily adapted, although the design revision may be constrained by the use of ADOxx.
This paper describes the execution of an evaluation to test the intuitiveness of the MIoTA modeling method. The evaluation was conducted to validate the applicability of MIoTA and to enhance the comprehension and acceptance of the method among users who are no experts in the domain and have no prior experience with the tool. The evaluation tasks were performed by fourteen participants with diverse backgrounds and modeling experience. The analysis of these tasks and the questionnaire led to the proposal of several improvements to the MIoTA modeling tool. Moreover, our findings indicate that models with diferent expressions from diferent users can express the same understanding contingent on the accuracy of the model (empirical quality). The models generated are comprehensible to both model experts and domain experts, which implies that the social (pragmatic) quality is also confirmed. This is important because we realized that models are frequently developed through collaboration between those engaged in modeling, whose understanding of the modeling domain evolves as modeling progresses. IoT development in the field of air conditioning facilities, to our experience, involves diferent stakeholders in such a collaboration process. With regard to the complexity of the two models (groups A and B), no significant diferences were identified in the results. In both groups, models that were created completely error-free were observed. In group A (which had to create the more complex model), however, more errors were due to the fact that something was simply forgotten or the instructions were not read carefully. Regarding our GQs, we can say that users who are no experts in the domain and have no prior experience with the tool are able to create accurate models (GQ1) and enter the configuration data accurately (GQ2). The majority of participants perceived MIoTA as useful and intuitive. It can now be stated that we completed the evaluation approach introduced in section 3.1.
The research is not without potential threats to its validity [26]. To ensure construct validity, it is essential to guarantee that the assigned tasks are aligned with the objective of evaluating the intuitiveness of a tool. Consequently, we employed an evaluation approach that adheres to rigorous substantiation of the tasks’ origin. With respect to internal validity, external factors that influence the results must be avoided. To this end, participants were selected with the same foreknowledge (no prior experience with the tool and providing them with the same domain knowledge to conduct the experiment). Furthermore, all participants received an identical introduction to the MIoTA tool. The participation was entirely voluntary, and no compensation was provided. To mitigate the potential for allocation bias, two distinct, randomly assigned groups were utilized for the tasks. The selection of participants also impacts the external validity or generalizability of the findings. The participants are primarily researchers with a master’s degree, which raises questions about the generalizability of the results. This choice represents an inherent limitation, and further research is needed to replicate the evaluation with participants with diferent levels of education and employment status. Reliability is a measure of the extent to which the results of a study can be reproduced by other researchers in the ifeld. The procedure used to evaluate the reliability of this study is described in detail in section 3 to ensure its reliability. This section also provides a comprehensive analysis of the diferent evaluation tasks. While a real facility is not used for the experiment, the sample model (gold standard) was created with input from domain experts to ensure the most realistic values and comprehensive coverage of tool functions.
Further research is required to evaluate the proposed improvements and possible implementation. As some of these aspects have already been identified by experts during the investigative process in prior work, it seems plausible that they will be incorporated into the tool to enhance its functionality. This will entail an experiment in which the intuitiveness of the initial tool and the new tool will be compared. Such an experiment could be based on recall and comprehension questions, which would allow for a comparison of the efectiveness and eficiency of interpreting both versions of the MIoTA tool [ 27]. However, further research is required to establish a robust experimental design. In this regard, we are currently working on a new version of the tool that will implement the identified improvements and bug ifxes. Nevertheless, some of the proposed enhancements have not been taken into account. For instance, a few participants found the manual entry of configuration data somewhat lengthy. In reality, however, this process is frequently unavoidable, given that the relevant documentation for the systems is typically not accessible in digital format. This process is also considerably more expeditious and less susceptible to errors than the previous one. As SEQUAL [17] and Moody’s "Physics" of Notations [24] may be perceived as too generic approaches that lack certain technological specifics relevant to MIoTA, a few aspects remain open for consideration in future evaluations. For instance, while Moody’s approach is centered on static symbol encoding, ADOxx implementations allow for dynamic or interactive notation (termed "secondary notation" in [28] ), scripting of specific modeling tasks, recording of modeling actions, and the transfer of certain functional requirements to model annotations. While SEQUAL is concerned with coherence (e.g., meta-model coherence), ADOxx employs a notion of competency that can be operationalized through model queries and model-based report generation. Furthermore, consistency challenges across multiple model types, acting as "perspectives" (meta-model partitions) [29], may become relevant in the future when multiple interrelated diagram types are contained.