This paper presents an experiment analyzing the behavior of modelers in interpreting conceptual model fragments developed using an OntologyDriven Conceptual Modeling Language (OntoUML). Our goal is to evaluate the effect of two ontological constructs (relator and role) in modeling relationships. Our hypotheses are: (i) the use of these constructs for modeling relations increases the clarity of the models; (ii) as a consequence, this increases the performance of modelers in interpreting these models. The behavior of the modelers is evaluated considering the participants´ answers to predetermined questions asked about two model fragments (compared with a template response). We also consider the rationale underlying the answers given by the participants. We have collected indications that both the role and the relator constructs influence positively in the quality of the resulting interpretations.
Despite its fundamental importance in conceptual modeling, there is evidence that the
relationship construct and its modeling implications are still not fully understood by
modelers. Wand, Storey and Weber [
Most modelers have an intuitive understanding of what a relationship is. However,
recognizing relationships in a real-world context, or correctly representing them in a
model has proven to be challenging. There are various ways to represent a
relationship [
In [
Concerning relationships, UFO's theory of relations makes a fundamental
distinction between two main types of relationships, namely: formal and material relations.
Whilst the former holds directly between two entities without any further intervening
individual, the latter is induced by the presence of mediating entities called relators.
Relators are individuals with the power of connecting entities. For example, an
enrollment connects a student with an educational institution, an employment connects
an employee with an employer [
Role is another object type strongly related to the modeling of relationships, since it represents relationally dependent entities. For instance, the role student is played by a person, when she is enrolled in an educational institution. Every role class must be connected to an association end of a mediation relation. Thus, roles and relators are important and heavily used constructs for modeling relationships in OntoUML.
In this paper, in order to evaluate the behavior of modelers in interpreting relationships in a conceptual model, we partially describe an experiment whose goal is to collect early indications about the interpretation of the constructs relator and role in OntoUML conceptual model fragments. Our research hypothesis is that the presence of these constructs increases the clarity of the models and, hence, improves the performance of modelers in interpreting model fragments using both constructs (considering correctness of answers according to a template, as well as the rationale given by the participants) when compared to the behavior of participants interpreting model fragments without them. The subjects are Computer Science students that have some experience in conceptual modeling. The results contain indications that the hypothesis is true. The findings indicate that role is a valuable assistance to avoid the problematic use of optional associations, while relator influences positively the participants´ performance. It is important to highlight that, although initially we assumed that the modelers’ level of experience could be easily balanced, the results show that this aspect interfered in the interpretations made by the participants.
The remainder of this paper is organized as follows. Section 2 contains the background about the concepts involved in the experiment. Section 3 discusses the empirical study itself. Section 4 describes some related works. Section 5 presents our final considerations. 2
Conceptual Modeling refers to formally describing in diagrammatic notation aspects
of the physical and social world for the purposes of understanding, communication
and problem-solving [
There are mandatory relationships (always exist) and optional relationships (may
or may not exist). A mandatory relationship is identified by assigning the minimum
cardinality of 1 to the opposite association end of a given related entity, while an
optional relationship is identified by a minimum cardinality of 0 to that position. The use
of optional relationships is controversial. Some researchers argue that it should be
avoided. Weber et al. [
Researchers have proposed languages and methods that aim to support creating
quality conceptual models. In this context, foundational ontologies have been
explored as a means to improve conceptual modeling languages. For instance, in [
A <<role>> represents an anti-rigid and relationally dependent universal.
Every <<role>> must be connected to an association end of a <<mediation>> relation. E.g.: Student.
A <<relator>> universal is a relational moment universal. Every <<relator>> must be (directly or indirectly) connected to an association end on at least one <<mediation>> relation. E.g.: Marriage.
A <<mediation>> is a formal relation that takes place between a relator universal and the endurant universal(s) it mediates. E.g.: the relator universal Marriage mediates the role universals Husband and Wife.
A derivation relation represents the formal relation of derivation that exists between a material relation and the relator universal this material relation is derived from. E.g.: the material relation married to, derived from the relator universal Marriage.
A <<material>> relation is a relational universal which is induced by a relator universal. E.g.: a person is married to another person.
In this section, we describe the empirical study performed. This experiment was
conducted following the guidelines presented in [
The experiment goal is to collect indications of the impact of different representation strategies for modeling relationships in the interpretation of model fragments. The objects of study are conceptual model fragments developed using OntoUML in different domains. The research hypothesis is that the presence of relators and roles increases the models clarity and, hence, improves the performance of modelers in interpreting model fragments using both constructs (considering correctness of answers according to a template, as well as the rationale given by the participants) when compared to the behavior of participants interpreting model fragments without them.
The experiment has qualitative and quantitative strategies. The experimentation level was in-vitro (it was conducted in a controlled environment). The research approach is primarily analytical, to collect early indications for further experiments. The first model fragment focus on representing a reflexive relationship, while the second focus on representing a binary relationship between two different concepts.
The subjects are Computer Science students, which attend classes of a Conceptual Modeling course. The minimum requirement expected for participating in the experiment is having basic knowledge of UML and OntoUML. A questionnaire was applied to capture the participants’ profile. Regarding the sample size, there were 22 participants. They were divided into four groups (GA, GB, GC and GD) randomly. GA and GB have 6 participants each, and GC and GD have 5 participants each.
The participants’ profile is of students acting as model users (readers) with some level of knowledge in conceptual modeling, specifically on using UML and OntoUML relations. The educational level (undergraduate, master and doctoral students) of the four groups was balanced (most participants were postgraduate students, but all groups have at least one undergraduate student). Regarding experience time in conceptual modeling using UML, all members of GA and GC have more than one year of experience, while GB has around 80% of its members in this range, and GD has around 60% of its members in this range. Concerning knowledge in OntoUML prior to the Conceptual Modeling course, around 80% of the participants in GA and GC already had prior knowledge in the language. That number drops to around 30% in GB and 40% in GD. However, over the course, OntoUML was studied and all the participants had access to the minimum knowledge necessary for performing this activity. At first, we considered that this would cause equilibrium between the distinct groups. The results, however, showed us that this was not the case, and the level of experience of the modelers influenced their interpretation of the diagrams.
The factor of the experiment is the representation of relationships using relator and role constructs in conceptual models. The alternatives are: i) representing only the material relation, without relator and role (see Figure 1, Group A); ii) representing roles and material relation, without relator (see Figure 1, Group B and Figure 2, Group C); iii) representing relator, roles and the corresponding mediation relations (see Figure 1, Group D and Figure 2, Group B); iv) representing relator and the corresponding mediation relations, without roles (see Figure 2, Group D); iv) representing role, relator, the corresponding mediation relations, and the material relation derived from the relator (henceforth termed “complete representation”) (see Figure 1, Group C and Figure 2, Group A). The task is to interpret two conceptual model fragments, each one regarding a different domain, using different representations. Figures 1 and 2 depict the representations of the model fragments for each domain and group. The interpretation is done by means of answering four questions relative to each model fragment. Each response should include an explanation of how the participant arrived at the answer.
The selected domains were artificially designed. In the first domain, the model fragments intend to capture a relation between Witches involved in a Bewitchment. In this fictional world, witches can put spells (bewitchments) on each other. In the second domain, the fragments are designed to represent a relation between a Knight and a Tamed Dragon. In this (again) fictitious domain, knights and their tamed dragons are connected by a loyalty bond. The intention behind the choice of model fragments in fictional domains has the goal of minimizing the use of previous domain knowledge in answering the questions.
The questions, which are the same for all groups, are: • Domain 1 (Bewitching Witches bewitch Bewitched Witches): Q1) How many Witches can be bewitched by a Bewitchment? Q2) How many Witches can be the Bewitching Witch of a Bewitchment? Q3) Can a Witch be bewitched by her own Bewitchment? Q4) Can a Bewitchment exist without affecting any Bewitched Witch? • Domain 2 (Bonds between Knights and Tamed Dragons): Q1) Can a Tamed Dragon exist without a Knight? Q2) How many Bonds might exist between a Dragon and a Knight? Q3) Can a person be a Knight without a Bond with a Dragon? Q4) How many Bonds a Knight may participate at the same time?
Q2 in Domain 2 includes an implicit condition being tested. Instead of asking for "Tamed Dragon" (a role) it asks for "Dragon" (a kind). There is nothing in the model specifying that all dragons must be tamed. We wanted to verify if the participants are aware of this difference.
The dependent variable is correctness of answers, considering also the rationale followed by the participant. This variable is measured by comparing the fragments of the participants’ answers to the corresponding fragments of a template. If they are the same, then the fragment is correct; otherwise, the fragment is considered incorrect.
Table 2 presents the tabulated results in terms of percentage of correct answers for each question per group. As we can notice, GA and GC (groups containing the most experienced modelers) presented better performance (over 50% of correct answers), than GB and GD (less experienced groups). These latter groups answered less than 50% of the questions right. In both domains, the highest success rate (100% correct) was obtained by groups GC and GA. We had also cases in which no correct answers were given (0% correct). The latter case happened in groups GA, GC and GD. It is worthwhile to remember that Q2 of Domain 2 had an implicit condition being tested. This could justify the poor performance of the groups, including GA and GC.
Another aspect useful for evaluating the groups´ performance are the cases of participants who indicated difficulties in answering a question due to the absence of a relator (see table 2). For instance, consider Q1 in Domain 1. In our template, we established as an ideal response: Minimum = “1”. Maximum = “it cannot be determined with certainty”. An instance of the relationship “bewitches” must exist for the roles “Bewitching Witch” and “Bewitched Witch” exist too. If such relationship exists, then there is at least one “Bewitching Witch” and one “Bewitched Witch” instances. The relationship´s cardinality also allows to infer that the same “Bewitching Witch” can “bewitches” several “Witch” (“Bewitched Witch”), but this may occur through one “Bewitchment” or even several “Bewitchment” - which cannot be deduced from the model where such concept is not present. We notice that, in comparison with group GB, a significant percentage of GA’s members indicated that the absence of an explicitly represented relator prevented them in answering these questions. An unexpected outcome was noticed in the behavior of GC (more experienced group), which has less than 30% of participants indicating difficulties to answer a question due to relator absence. It is worthwhile to point out that this aspect did not apply to GD, since all model fragments read by them contained relators in both domains.
Table 3 also presents the results regarding correctness, but focusing on the different representation strategies. We noticed that the best performance was achieved by groups using the complete representation. The second best result is when both relator and roles are presented, but the derived material relation is not. However, we should reinforce that, in general, we perceived an influence of the experience level in the results. Thus, it is more adequate to compare the performances of a same group using different representations. This comparison provides an indication about the influence of the presence/absence of relator and/or role constructs. For instance, consider GC. In the complete representation, GC obtained 80% of correct answers. However, using a representation without relator, the same group answered only 40% of the answers correctly.
Table 4 shows an interesting aspect regarding the use of optional associations versus the use of roles and mandatory associations. This observation involves questions of type "how many" (Q1 and Q2 in D1; Q2 and Q4 in D2) concerning the minimum value indicated in the answers. To simplify we compare only two groups per domain. In D1, we compare GA (without role, with optional relationship) and GC (with role, with mandatory relationship). In D2, we compare GD (without role, with optional relationship) and GB (with role, with mandatory relationship). For these questions, the minimum cardinality should be 1, except for the case of Q2 in D2, which, as previously mentioned, has an implicit condition being tested, and because of this, the minimum 0 is acceptable. In both domains we noticed better performances by the groups that worked with representations that explicitly show the role construct.
In general, the results of the experiment point out that: 1) The experience level interferes in the modelers' performance in interpreting models. Most experienced groups presented better performance; 2) The representation strategy affects the modelers' performance in interpreting models. Participants reading model fragments that explicitly show the constructs relator and role presented better performance.
For answering the experiment research questions, we used the collected data presented above, as well as analyses of participants’ responses. To illustrate these analyses, next we present the analysis done for question Q4 of domain D1. Similar analyses were performed on the other questions of the experiment, but only the main findings are identified in this paper.
D1. Q4: Can a bewitchment exist without affecting any Bewitched Witch? Table 5 presents the performance of the participants in this question.
In GA, 3 out of 6 answered the question incorrectly (“yes”, due to the zero minimum cardinality constraint). The correct answer is: “no”, given that a Bewitchment is an existentially dependent entity and it depends on at least one individual playing the role of a Bewitched Witch as well as at least one (distinct) individual playing the role of Bewitching Witch. A 4th participant answered that it was impossible to answer the question precisely without the explicit representation of the Bewitchment. Despite sensible, we consider this answer as incorrect since: (i) in order to be a Bewitched Witch one has to participate in at least one relationship; (ii) since this is a material relation, participating in this relationship means being mediated by a relator (a Bewitchment); (iii) given that a relator is an existentially dependent entity, it must connect at least one instance of Bewitched Witch (and at least one distinct instance of Bewitching Witch). A 5th participant answered the question stating that if a Bewitchment is interpreted as a pair, then it should contain an instance of Bewitched Witch in the pair. Notice that although showing a sensible reasoning, this answer highlights a conceptual mistake, namely, that a Bewitchment is an instance of the relationship. In contrast, given the cardinalities of the model, it is clearly possible for the same Bewitchment instance involves a number of Bewitching Witches and Bewitched Witches. Anyway, we consider in this context the answer as a correct one. A 6th participant answered: “the role indicates that an individual is a Bewitching Witch when participating in a relationship”. We envisage that the participant followed the same reasoning of the previous one.
In GB, 3 out of 6 participants stated that it was impossible to answer the question. However, as explained about for the 4th participant of GA, we consider this answer incorrect. Three of the participants answered the question correctly. Moreover, one of them made an explicit reference to the relator, while the other two based their answer solely on the cardinality constraint (minimum of 1).
In GC, all participants answered the question uniformly and correctly. A similar tendency can be observed for GD in which 4 out of 5 participants gave the same correct answer. A 5th participant of GD mentioned that the model did not contain enough information for the question to be answered (which is an incorrect answer).
In summary, the groups with an explicit representation of the relator performed better in answering the question – observe GC and GD in Table 5. Moreover, the group with the complete representation (GC) was the only group with unanimously correct and justified answers. GA had the worst performance, with the majority of the participants showing a shallow reasoning process by reasoning only on the directly represented cardinality of the represented relation and, hence, answering the question incorrectly (50% of incorrect answers for this reason). The participants of GA and GB that answer the question correctly did so by reasoning on the instance of the relation and on the role at hand. It is no surprise that this performance was then better in GB where the roles were explicit represented (50% correct answers) than in GA (33.33%). Finally, half of the participants in GB and one participant in GA answered the question incorrectly by being unsuccessful in reconstructing the relation between roles, material relations, the relator. Furthermore, the only participant in GA that clearly answered the question correctly also made a conceptual mistake in equating the instance of the relation with the relator. This is a common mistake. However, we interpret this mistake as also being influenced by the non-explicit representation of the relator and its connection to the material relation at hand.
RQ1) What are the effects of optional relationship in interpreting model fragments?
Regarding the use of the role construct to prevent the occurrence of optional associations, we conclude that it leads to a better performance of the participants. This evidence can be perceived by contrasting model fragments with optional association (without role) and with mandatory association (with role), as Table 4 shows.
Contrasting groups with equivalent experience, in Domain 1, we see that GC had 100% of correct answers (minimum 1) in both questions (Q1 and Q2), while GA indicated more than 65% of incorrect answers (minimum 0) in both questions. The remainder of the responses were also incorrect, but due to some other reasons that we refrain from discussing here (due to space limitations). This is a clear indication of the influence of the role construct in guiding the participants to answer questions correctly. In Domain 2, question Q4, we realize that GB's participants indicated the correct answer (minimum 1) more than 65% of cases, while GD's participants indicated 20% of incorrect answers (minimum 0) and no correct answers. The other occurrences of errors were due to other reasons. Here, we once more notice the influence of role to guide the correct answer, although not as significantly as in Domain 1. Regarding question Q2, the minimum 0 is the correct answer, for those who noticed the implicit condition being tested in this question. However, only 3 participants explicitly mention that they have perceived this implicit condition being tested, and thus the results of this question mask the effect of using the role construct.
In summary, the evidence points out that in questions involving minimum cardinalities interpretation, optional relations confuse the reader. Using roles, thus representing only mandatory relations, clearly leads to a better performance of the participants. RQ2) What are the effects of the presence / absence of relators in interpreting model fragments?
The findings of the experiment show that in model fragments where relators are present, the tendency of the participants is to reason on them to interpret the question. Moreover, in general, relators contribute positively to a quality interpretation.
Table 3 gives us some interesting indications: (i) There was better performance of groups with the complete representation (total above 70%, while other representation forms did not reach 50% of correctness); (ii) When contrasting the results of GA and GC (the most experienced groups, which used the complete representation and a representation without relator), we also realize a better performance of each in cases when they interpreted model fragments that had the presence of relators. The result of GB also provides evidence of the positive influence of relator, even though less significantly; (iii) The representations with relator (first two columns of table) had better performance than representations without relator (last two columns of table). One exception occurred in the representation with relator, without role (the middle column), which seems to have been affected by the experience level of GD (there was a significant number of partially correct answers, which we count as error); (iv) We notice that a small change in the representation strategy (between columns 1 and 2, by showing the material relation derived from the relator) apparently causes a large variation in the result (correctness percentage). In this case, however, it seems to be an influence of the experience level more than of the representation strategy (GA and GC have more experience than GB and GD).
Another aspect should be highlighted: Were the participants able to notice when a question could not be satisfactorily answered by a representation? Without relator, there were attempts to follow other paths of reasoning, and usually only the most experienced modelers realized the implications of the absence of the relator construct in questions where it was required. This situation should have been noted by the participants of GA and GB in the four questions of Domain 1, as well as by the participants of GC in three questions (Q2, Q3, Q4) of Domain 2. The last line of Table 2 contains some values related to it. The values are lower than expected, especially in the more experienced groups. This may be an indication that the participants tried to answer the questions with the information provided, without thinking enough about what might be missing.
During the experiment, we identified some limitations and validity threats. Firstly, we invited to participate in the experiment students of a course that involved different student levels (undergraduate, master and doctoral), with different experience levels in conceptual modeling. We tried to balance this diversity along the course. Thus, the participants were aware of the concepts necessary to execute the proposed activity. However, the results showed us that the different experience levels affected the experiment outcome. Secondly, we selected some unusual problem domains to be interpreted through the models, in order to reduce the influence of background knowledge. So, the modeler could concentrate on the model to answer the questions. However, the use of such domains may have had an inverse effect, distracting some participants instead of allowing them to focus on the model. Thirdly, we considered the interpretation of small fragments of conceptual models. The understandability in larger models can lead to different results from those obtained in this experiment. This is not clear yet, as the use of relator and role constructs in these models could contribute for enlarging the models and increasing their complexity. Fourthly, the number of participants was small, and thus we had not a representative sample. Because of that, we could not to apply statistical hypotheses tests. Fifthly, each group analyzed different relationship representations in different problem domains, thus analyzing different relationship types (for example, reflexive and binary relationships). The results may have been influenced by the familiarity of the participants with such aspects. Sixthly, as the number of participants was small and we were ambitious in the quantity of representation strategies that we intended to evaluate, not all strategies identified were analyzed by all groups. This fact complicated the analysis of the collected data, because not all possible variations (groups versus representation strategies) were available for analysis. Finally, the participants knew that the experiment aims to seek for evidence of the usefulness of relator and role constructs. The awareness of the importance of such concepts may have induced them to reason in a certain way. Indeed, as aforementioned, some participants detached the relator absence and pointed out such situation as a factor that hindered the model interpretation (which was expected). 4
Several works have been undertaken aiming at studying the use of one of the most
basic constructs in conceptual modeling: relationship. Regarding the difficulty of
representing relationships, Wand, Storey and Weber [
Regarding empirical studies related to the relationship construct, Poels et al. [
Weber et al. [
The experiment partially presented in this paper collected evidence on the use of the relator and role constructs in conceptual models fragments developed in OntoUML and interpreted by modelers. The findings are in favor of our hypothesis that the use of such constructs increases the models quality, allowing better interpretation of them. Some evidences identified are: (i) The use of relator and role constructs influences the quality of the resulting models, making them clearer; (ii) The role construct contributes positively in avoiding the use of optional associations; (iii) The relator construct assists both novice and experienced modelers. However, such assistance is manifested differently; (iv) The experience level interferes in the performance of the participants. We were not able to isolate this variable in the experiment, as we thought at the beginning. It is necessary to contrast the influence of experience level versus representation strategy in another experiment, to better understand the ways each aspect interferes in the performance of the participants.
Concerning the use of roles and mandatory relations in place of optional relations,
our findings are in line with the ones achieved by Weber et al. [
Regarding relators, although the results also confirm our hypothesis, the evidences
are more subtle. In fact, we perceived an influence of the experience level in these
results. Although we have tried to balance the experience level of the modelers in
each group, we did not really achieve such balance. Thus, further studies should be
undertaken for improving our understanding regarding modelers' profile. In this sense,
we intend to perform other experiments, taking as basis the design used by Hadar in
[
This experiment collected evidence that we intend to use to guide future studies, more rigorous and focused. We are aware of some limitations and threats to validity of this study, as discussed in Section 3. So, we intend to use these results to orientate future efforts, improving the design of the experiments and deepening in the evidence identified. Some future works include: (i) Evidence points positively to the proposed hypothesis, but further studies should be developed to check them. For instance: Should the results be the same with larger and varied groups? Should the results be similar in the interpretation of larger models? (ii) We need to evaluate some limitations and threats to validity that we identified and did not explore yet. For instance, further studies should be undertaken for improving our understanding regarding modelers' profile.
Due to the lack of space we described the experiment partially in this paper. We intend to publish the complete result soon, also making possible the experiment replication by other researchers interested in the field. This work was partially supported by the Lucretius ERC Advanced Grant #267856 and FAPES/Brazil.