In this article, we describe the rst results of two series of experiments. It is the aim of our experiments to compare the performance of novice users working with {_ models. We compared the results achieved with {_ diagrams in the traditional notation with the results using the alternative notation which has been suggested by using Moody's \Physics of Notations" framework. We created two models in both variants of the notation. In the rst experiment, we asked comprehension questions about models printed on paper. In a second experiment, the participants had to answer questions using models on a computer screen. The results give rst support to the hypothesis that the improvements suggested by Moody could be helpful for understanding {_ models. We also identi ed a detail for which the newly suggested notation variant can lead to understandability problems and should be improved.
The language {_ [
In particular, Moody and his co-authors suggest to replace the shapes traditionally used in {_ (see Fig. 1) by another set of graphical symbols (see Fig. 2). We will refer to this notation as PoN notation. When comparing Fig. 1 and Fig. 2, we see that it is easier to di erentiate between the symbols in the PoN notation. Furthermore, it should be easier to understand the meaning of these symbols by intuition. For example the concept of a \Task" is symbolized by a stick-notice which is more intuitive than the traditional hexagon symbol.
Several arguments of the PoN framework are backed by empirical
validation on understandability of symbols used in graphical user interfaces: Byrne [
However, there are no empirical results for evaluating the improvements
suggested in [
Experiment 1: Working with a Model on Paper
For two scenarios, we constructed two {_ models: One using the traditional {_
notation and another one using the notation suggested by Moody et al. The rst
scenario was the classic meeting scheduler problem [
Students of public management and students of information science were asked to answer comprehension questions about those models. For this purpose, all students got an introduction lecture on the concepts of {_ , but without referring to a speci c visual notation. Before the comprehension experiment, they have been assigned to two homogeneous groups - one for each notation variant. An introduction of the {_ symbols (the traditional ones in one group, the PoN symbols in the other one) was given to the students before they had to work with the models (printed in colour on paper) in order to answer 6 paper-based comprehension questions. While working with the models, each student had a legend explaining all the symbols in form of a one-page handout. No time limit was given for answering the questions.
The purpose of three questions was to reason about the contribution that model elements have to softgoals. An example question would be \Which softgoal of the youth counseling organization will be a ected negatively when solution A is used?" One question was aimed to measure understanding of task decomposition. Two questions were \counting questions" (such as: How many actors depend on the appointment participant?)
Altogether, the questions have been answered by 121 persons so far (32 bachelor students of public management, 53 master students of public management, 36 bachelor students of information science). 2.2
Tab. 1 shows how well the participants answered the questions. Our assumption that the users who worked with the PoN models would perform better could not be con rmed. In one group (bachelor students of public management) the participants using the PoN notation answered more questions wrongly than the participants working with the traditional notation. After this unexpected result, we had a closer look at the results of each question for this group of participants. We identi ed one \counting" question for which in this group of students, Public Management, BA 56.8 53.3 Public Management, MA 47.5 55.7
Information Science, BA 54.2 60.4
Table 1. Percentage of correctly answered questions (all questions)
the participants using the PoN notation had less than half as much correct
answers as the participants using the traditional notation. This question required
to answer how many actors depend on a given actor. For solving it correctly,
it was important to understand the direction of the dependency (Who provides
something and who needs something?) As shown in Fig. 4, it is easy to
misunderstand the PoN notation to depict these facts: An intuitive interpretation of
the arc symbol could be that the resource is transported from the origin of the
arc in the direction of the arrowhead (comparable to the intuitive interpretation
of causal relationships reported in [
If we exclude the question that requires to count dependencies, the results are as shown in Tab. 2. While the results shown in this table show a trend in the expected direction, they are not statistically signi cant, and experiment 1 does not con rm the assumption that working with models in the PoN notation leads to less errors. In the future, we plan additional experiments in order to get results which are founded on the performance of a larger number of participants. Public Management, BA 49.0 50.0 Public Management, MA 43.1 48.0
Information Science, BA 48.5 55.5 Table 2. Percentage of correctly answered questions (dependency counting question removed) 3
Experiment 2: Working with a Model on a Computer Screen 3.1
In a second experiment, we used the eye tracking device Tobii T60 XL Eye Tracker with a 24 in. screen and the software Tobii Studio 3.2.1. Sixteen participants (eight for each notation variant) had to solve a set of six tasks while their eye movement was recorded. There were 3 students of business information science in the group using the traditional notation and 4 students of business information science in the group using the PoN notation. All other participants were recruited from the academic sta at the department of business information science at the University of Hamburg.
Before the experiment, each participant got an introduction to {_ by means of a self-learning course (they were not familiar with {_ before). Two di erent variants of the course were developed. One used the traditional {_ notation, the second one the PoN notation. In all other aspects, the self-learning courses were identical. The participants had as much time as they wished to get familiar with the {_ concepts and the notation, and they had the possibility to ask additional questions if something remained unclear.
Then the participants had to solve six questions on the same {_ diagrams that have been used in experiment 1. We could not always use the same questions as in experiment 1, because in experiment 2 the participants had to remember the questions. Therefore, some questions had to be simpli ed or changed. The participants were instructed to press a key as quickly as possible if they have found the answer. Afterwards, they showed their answer on the screen. This way, we recorded both the time needed to nd the answer as the eye movement of the participants while looking at the diagrams in order to answer a question. 3.2
As already observed in experiment 1, the correctness of the answers to a question where the number of dependencies had to be given was better in the group using the traditional notation (4 errors, compared to 5 errors in the PoN group). Tab. 3 shows the number of wrongly answered questions for all participants.
If we include all six questions, the mean number of errors in the group using the traditional notation is 2.625. For the PoN group, it is 1.125. We performed a one-tailed Mann-Whitney U-test on the data groups. Both sample sizes were 8, the U value for the one-tailed Mann-Whitney test is 11 and the P-value is 0.014, i.e. the result is signi cant at the standard level of 95%. If we exclude the \dependency counting" question, we even get a highly signi cant result (P-value for the one-tailed test 0.00148).
A large amount of data has been collected by the eye-tracking system. We expect additional insights on how people work with {_ models by analyzing this data, but have not yet started the analysis. This paper describes the rst preliminary results of two experiments. Experiment 1 could not con rm in a statistically signi cant way that working with the PoN notation has a positive e ect on understanding the diagrams. We plan to repeat the experiments with more participants in order to get more reliable results. However, one side-result of the experiment (which has been backed by experiment 2 as well) was the suggestion to replace the depiction of the dependency link in the PoN notation. Experiment 2 showed that participants using the PoN notation performed signi cantly better (working on a computer screen). The statistical signi cance of this result is very high despite the fact that the number of participants was rather low (which is not uncommon for eye-tracking experiments).
A possible reason for the high signi cance achieved in experiment 2 is that in the eye-tracking environment, the participants had to rely on their memory for deciding about the meaning of a graphical symbol - possibly a situation where easy-to-remember symbols are particularly helpful.
For the future, we plan to repeat our experiment with more participants. Furthermore, we expect additional insights by analysing the logs of the eyetracking sessions.
A limitation of our research is that all experiments have been conducted with novice users, therefore the results cannot be transferred to experienced {_ users.