This article reports the findings from a controlled experiment where both the comprehensibility and the quality of UML interaction diagrams were investigated in two application domains: management information system (MIS) and real-time (RT) system. The results indicate that collaboration diagrams are easier to comprehend than sequence diagrams in RT systems, while there is no difference in their comprehension in MIS. With respect to quality of diagrams constructed by analysts, in MIS collaboration diagrams are of better quality than sequence diagrams, while in RT there is no significant difference in their quality.
UML defines twelve types of artifacts in the form of diagrams which are divided into
the following three categories: class, object, component, and deployment diagrams
represent the application's static structure; use case, interaction (sequence and
collaboration), activity and state charts represent the application's dynamic behavior; and
packages, subsystems, and models represent the application modules' organization
[
The focus of this study is on UML's interaction diagrams, which depict a pattern of interaction among objects. Interaction diagrams come in two forms emphasizing different aspects of an interaction: sequence diagrams and collaboration diagrams. Our goal is to evaluate and compare these two types of diagrams. Sequence Diagram depicts an explicit sequence of stimuli messages exchanged among object instances participating in the interaction. Sequence diagrams include lifelines for the instances used to portray the temporal dimension of the modeled pattern. Collaboration Diagram is a directed graph where nodes represent communicating entities and edges represent communications. The edges are numbered to represent the order of communications.1
Sequence diagrams and collaboration diagrams express similar information but
depict it in a different way. Both diagrams are considered symmetric and it is therefore
1 Due to space limitation we do not show examples of sequence and collaboration diagrams.
possible to convert a sequence diagram to a collaboration diagram and vice versa [
A collaboration diagram addresses the loopholes of a sequence diagram by
depicting relationships among involved objects. A collaboration diagram is therefore
recommended for supplementing both class and use-case (static-view) diagrams because
it depicts the interactions among objects (dynamic-view) [
Most of the extant work on UML interaction diagrams focused on conceptual analysis
and comparison of the features of collaboration and sequence diagrams. The only
empirical research that we know of comparing interaction diagrams was performed
by Otero and Dolado [
In their first study [
The experiments described above suffer from several limitations. First, they compare non-equivalent types of diagrams because state diagrams are analyzed as equivalent to interaction diagrams (sequence and collaboration). Second, the 2004 study evaluated pairs of models without providing sufficient rational and proving that the pairs are in fact interchangeable. Third, they did not address the issue of building diagrams and their quality with regards to different application domains.
The purpose of our study is to fill the research gaps on UML interaction diagrams. As we have seen, prior research has mainly focused on comprehension of diagrams, but the issue of the quality of the diagram types has not been addressed yet. By investigating performance in terms of comprehension, quality, time and user/analyst preference in different application domains, the findings of this study are expected to provide a wide-angle view on the UML interaction modeling aspect, and hopefully contribute to the productivity of analysts, designers and programmers in complex information technology environments. Such environments are more likely to comprise of both heterogeneous types of applications (i.e., real time reactive/non reactive, managerial) and tight interaction between system designers and analysts.
The goal of this study is to evaluate and compare sequence diagrams and collaboration diagrams from the two main perspectives (analysts and users) by conducting two controlled experiments. From the users' perspective, we are interested in the comprehensibility, i.e. understandability of the diagram, while from analysts' perspective, we are interested in the quality, i.e. correctness and completeness of the diagrams. The two types of diagrams are evaluated in two types of applications: a management information system (MIS) and a real-time reactive system (RT). Figure 1 depicts the experimental design of the study (it actually describes both experiments that were carried out as a part of a student test.)
In the "comprehensibility" experiment we compare users' comprehensibility of diagrams, the time it takes them to comprehend the diagrams and the perceived (subjective) ease of comprehension. We measured the following three dependent variables:
C – total score for questions on diagrams comprehension Tc – total time spent to answer questions on diagrams comprehension Pc –ranking of subject's perceived comprehensibility of a certain diagram In the "quality" experiment we compare the quality of diagrams as created by analysts, the time it takes to create the diagrams and the perceived ease of constructing them. We measured the following three dependent variables:
Q – total score for quality of an interaction diagram constructed Tq – total time spent on diagram construction
Pq – ranking of subject's perceived ease of construction of a certain diagram
In both experiments the independent variables are the two types of diagrams:
sequence (Seq) and collaboration (Col), and two types of systems: a security system,
representing a real-time reactive system involving time dimension and concurrency
(RT); and a library system, representing a management information system (MIS).
These type of systems represent a substantial portion of prominent industry
applications and were also used extensively in previous research [
The controlled variables are the tasks and the subjects: in the "comprehension" experiment the task was to answer a questionnaire which measures comprehension of a Independent Variables
Interaction Diagram Sequence
Collaboration
Case Study MIS – Library
Real Time –
Security Monitoring System Controlled Variables
Tasks
Subjects Construction
Comprehension G1
G2
G3
G4
t teo trcu iTm sonC -enh erco e S given diagram; in the "quality" experiment the task was to construct a diagram. The subjects in the two tasks were randomly divided into four groups, as explained below.
The experiment was divided into two sessions. In Session 1 we conducted the "comprehension" experiment: subjects were asked to express comprehensibility of a given interaction diagram by answering five multiple-choice questions. In Session 2 we conducted the "quality" experiment: subjects were asked to construct an interaction diagram based on a brief narrative specification of the system, a class diagram and a use case diagram. Quality was measured by the correctness of the created diagrams with respect to the correct solutions. In addition to performing the tasks, the subjects were required to record the overall time spent to complete each session. Finally, they were also asked to express their subjective opinions on diagrams comprehension and construction.
Our objective is to address the following questions:
(1) is there a difference between a sequence and a collaboration diagram (in terms of
quality and/or comprehensibility) for dynamic modeling of a RT system?
(2) is there a difference between a sequence and a collaboration diagram (again, in
terms of quality and/or comprehensibility) for dynamic modeling of a MIS?
As we have seen, previous research indicated that sequence diagram has an
advantage over collaboration diagram in the representation of the temporal order, which is
particularly important in real-time systems where the processes take pre-defined time
slots and may be processed concurrently [
Since we are comparing between the two types of interaction diagrams using six dependent variables (as listed above) and the comparison is performed separately for two types of applications (RT vs. MIS), we have a total of 6x2 = 12 statistical tests. For each test, the null and the alternative hypotheses are defined in Table 1 below.
The experiment was carried out as a midterm test in a course on OO Analysis and
Design. Each subject was randomly assigned to one of four groups, differing by
diagram-type (collaboration vs. sequence) and application-type (MIS vs. RT system).
The theoretical material and the practical examples have been taken from the course
textbook [
Seventy-six subjects participated in the experiment. This sample size enabled us to reach conclusions at the significance level of 0.05 and higher. In the first session of the experiment, which dealt with comprehension of diagrams, each subject has received a class diagram, a use case diagram and an interaction diagram (collaboration or sequence) of one of the systems. In the second session of the experiment, which dealt with quality of constructed diagrams, each subject has received the system requirements in the form of a brief narrative specification of the system, a class diagram and a use case diagram of one of the systems. This session assignment was to construct a sequence diagram or a collaboration diagram for the corresponding system.
MIS
Dependent Variable C: score for diagram comprehension Q: score for quality of diagram constructed Tc: time (min.) spent on diagram comprehension Tq: time (min.)spent on diagram construction Pc: subjective comprehensibility of each diagram type
In this experiment we have manipulated two factors (diagram and system), each having two possible levels (types). Consequently, we have used a “2x2” factorial design. Each subject was assigned randomly to one of the resulting four groups (“cells”), denoted as G1, G2, G3, and G4 respectively, and then measured two times, first in the comprehension session and then in the construction session. Both the type of diagram and the system were different from one session to the next. In other words, each subject performed a comprehension task using a certain type of diagram and a certain system; and later on he/she performed a task of constructing another type of diagram for another system. The treatment conditions of each group are shown in Table 2. The total score for each session was calculated as follows: • Each correct answer to a multiple-choice question has received one point. No points were deducted for a wrong answer. Hence, the maximum total score for the first session was five points. • The maximum total score for the second session was 60 points and it was calculated over all diagram components. The number of points deducted for each mistake in a diagram component was based on the severity of mistake, depending on the different components of the diagram. We identified the following components of interaction diagrams: objects, messages, message sequences, links, and time line. The number of points deducted for each mistake was 2, 4, 6, 8, or 10, depending on the severity of a particular mistake. Totally omitting an object was considered the most severe mistake resulting in a deduction of 10 points. Some mistakes were relevant only to one type of diagram. For example, omitting message number was relevant only to collaboration diagrams, while not showing activity duration was relevant only to sequence diagrams.
In order to evaluate the hypotheses stated in Table 1 with respect to the six dependent variables, we used two-sided t-test, which is based on the following assumptions regarding the two samples: • Independence. Based on our experimental settings, there is no reason to believe that the results of the subjects in the first group (e.g., those doing sequence diagrams) are in any way related to the results of the subjects in the other group (e.g., those doing collaboration diagrams). • Normal distribution. The scores of every group in each session were tested for normality using the chi-square test. The results varied between p = 0.054 (Group 4, Session 1) and p = 0.976 (Group 1, Session 2). This means that no statistically significant departure from the normality assumption was found. • Equal variance. We have compared the variances of every group in each session using F-test (see Table 3). Most groups were not found significantly different in terms of their variance. Based on the results of the F-test, we have applied equal variance (homoscedastic) or unequal variance (heteroscedastic) t-test as appropriate. In general, violation of the equal variance assumption is not problematic unless the two samples are quite different and one of the samples is small. In our study, we use nearly equal-size and relatively large groups (see Table 2 above).
The results of two-sided t-tests performed in the experiments are summarized in Tables 4-15 below. The two t-tests corresponding to each dependent variable were analyzed independently of each other (and not as a series of t-tests), since we are not interested in the overall differences between diagrams across the diverse application types. The minimum significance level for rejecting a null hypothesis was α = 0.05. Detailed explanation and interpretation are provided in the sub-sections below.
The results of two-sided t-tests evaluating the difference in diagrams comprehension for the MIS and the RT systems are shown in Tables 4 and 5, respectively. According to Table 4, the difference in comprehension of the two diagram types is not statistically significant in the case of the MIS. On the other hand, Table 5 shows that in the case of the RT system, the comprehension score of the collaboration diagram is significantly higher than the score of the sequence diagram. Cell means for diagram comprehension in each system are shown graphically in Figure 2.
The results of two-sided t-tests evaluating the difference in diagram quality for the MIS and RT systems are shown in Tables 6 and 7, respectively. Table 6 depicts that in the case of the MIS the scores of the collaboration diagram are significantly higher than the scores of the sequence diagram. On the other hand, Table 7 indicates that in a RT system, the difference in the quality of the two diagram types is not statistically significant. Cell means for diagram quality in each system are shown graphically in Figure 3.
Diagram Type
According to the results of two-sided t-tests evaluating the difference in the time (number of minutes) spent on diagrams comprehension for MIS and RT systems (shown in Tables 8 and 9, respectively) there is no statistically significant difference between the average times spent on each diagram type in either application.
According to the results of two-sided t-tests evaluating the difference in the time (number of minutes) spent on diagrams construction for MIS and RT systems (shown in Tables 10 and 11, respectively); there is no statistically significant difference between the average times spent on each diagram type in either application.
In the post-test questionnaire, each subject has expressed the perceived comprehensibility of the diagram he/she analyzed. Comprehensibility was expressed using a 1-5 Likert ordinal scale, where the score of 1 indicated that the diagram was very comprehensible, while the score of 5 indicated that the diagram was absolutely incomprehensible. The questionnaires were summarized with respect to the test versions done by the subjects, i.e. by diagram types and application types. According to the results of two-sided t-tests evaluating the difference in comprehensibility scores for MIS and RT systems (shown in Tables 12 and 13, respectively), there is no statistically significant difference between the scores of each diagram type in either application.
The correlation coefficient between the comprehension score and the perceived comprehensibility score of the same subject is 0.207. This correlation coefficient is not significantly different from zero (p-value = 0.086). It means that no relationship was found between subjects’ perception and their actual performance in the test.
In the above post-test questionnaire, each subject has also estimated the perceived ease of constructing the diagram he/she has built in the test. The easiness score was given on the 1-5 scale, where the score of 1 indicated that the diagram was very easy to build, while the score of 5 meant that it was extremely difficult to build the diagram. The questionnaires were summarized with respect to the test versions done by the subjects, i.e. by diagram types and application types. According to the results of two-sided t-tests evaluating the difference in easiness scores for MIS and RT systems (shown in Tables 14 and 15, respectively), there is no statistically significant difference between the scores of each diagram type in either application. The correlation coefficient between the construction score and the perceived ease of construction of the same subject is 0.054. This correlation coefficient is not significantly different from zero (p-value = 0.657). It means that, like in the case of comprehension, there is no relationship between subjects’ perception and their actual performance in the test.
With respect to comprehension of diagrams, the results of this study indicate that
collaboration diagrams are easier to comprehend in the case of RT applications, but
there is no difference in comprehension of the two diagram types in the case of MIS.
Our findings contradict the findings of Otero and Dolado [
With respect to quality of diagrams constructed by analysts, we found that in MIS, collaboration diagrams are significantly better than sequence diagrams, but there is no significant difference in the quality of diagrams produced for RT systems.
Our findings were not able to confirm a significant difference between the diagrams with regards to the time needed to construct them. But as we have seen, subjects spent more time on constructing interaction diagrams of an MIS than of a RT system. This result is consistent with the equivalent result on time spent on comprehending diagrams. As explained for the equivalent results, a possible reason for this may be that participants prefer spending time on a type of system which they are more familiar with (because they see a better chance to obtain a higher test grade on quality, as the amount of time spent on the task did not affect their test grade).
With respect to the perceived comprehensibility, the results we obtained are
congruent with the expectations based on related work [
The implications of our study are in further investigating the contingency of UML
interaction diagrams in terms of quality and comprehension for various application
settings. The results from our controlled experiments suggests a rule of thumb for
employing collaboration and sequence diagrams in modeling RT and MIS
applications for both analysis and design stages. In some cases, however, our results
contradict earlier studies [
Our study suffers, however, from several limitations. First, the applications tested in the study (a library system and a security monitoring system) cannot be considered full-scale commercial applications. Second, the fact that only two applications were evaluated limits the external validity of this study. Third, the problems had a very slight difference in complexity and the results are assumed to depend, to a certain extent, on the problem description which might have caused some bias in the results; however for a given problem (MIS, RT) this should not have an effect on the findings with regards to the recommended interaction diagram. A common limitation of experimental research on model/method evaluation refers to the students participating, who in our case played both the role of users (in the "comprehension of diagrams" part of the experiment) and analysts (in the "quality of diagram construction" part).
Future research on this matter should first attempt to address some of the
limitations mentioned above. Thus, a repeat experiment should use more experienced
participants, possibly with professionals from the IT industry working on RT and MIS
projects. The bias in favor of the MIS applications could be remedied by using a
heterogeneous population of participants, including students majoring in Software
Engineering, who are more acquainted to RT systems than Information Systems
Engineering students. It is also recommended to diversify and control the complexity of
the applications in the experiment, in a similar method used by Otero and Dolado [