This paper presents a general framework how controlled natural languages can be evaluated and compared on the basis of user experiments. The subjects are asked to classify given statements (in the language to be tested) as either true or false with respect to a certain situation that is shown in a graphical notation called \ontographs". A rst experiment has been conducted that applies this framework to the language Attempto Controlled English (ACE).
The approach that we propose here solves the discussed problems and relies on a graphical notation which we call \ontographs". Each ontograph consists of a legend that introduces types and relations and of a mini world that introduces individuals, their types, and their relations. Individuals are represented by different symbols depending on the types they belong to. Relations are represented by arrows that point from one individual to another. Figure 1 shows four such ontographs. Ontographs require that we have complete information about the mini world and, as a consequence, do not need an explicit notation for negation. Everything that is not shown in the ontograph is not true.
We assumed that ontographs are very easy to understand and our results con rm this assumption. The use of intuitive graphical icons is one of the reasons of the good understandability. More important, however, is the fact that the ontograph notation has no generalization capabilities and does not support partial knowledge. This excludes many potential misunderstandings. An ontograph explicitly shows every existing individual and depicts every single relation instance between them. There is not much to misunderstand with these basic elements. While we can make general statements like \every man loves a woman" about the mini world de ned by an ontograph, there is no way to express such statements in this general way in the ontograph notation itself.
Ontographs are designed to be used in experiments to test the understandability of formal languages. Ontographs could also be used to test the writability by asking the subjects to describe the given situation. However, only the rst approach has been investigated so far.
In order to test the understandability of a language in an experiment, an ontograph and several statements (written in the language to be tested) are shown to the subjects who have to decide which of the statements are true and which are false with respect to the mini world depicted by the ontograph.
An important property of ontographs is that they use a graphical notation that is syntactically very di erent from textual languages like CNLs. This makes it virtually impossible to accomplish a task like the one described above just by looking at the syntax. If subjects manage to systematically classify given statements correctly as true or false with respect to a certain ontograph then we can conclude that the subjects understood the statements and the ontograph.
Using the presented testing framework, we conducted a small experiment that
tests the language Attempto Controlled English (ACE) [
Four ontographs (see Figure 1) | each accompanied by 10 ACE statements | were shown to the subjects who had to classify each of the statements as true, false, or \don't know". There was a time limit of ve minutes for each ontograph. There was no explanation how the ACE statements have to be interpreted with the one exception that it was explained that \something" can stand for persons and objects.
The ACE sentences are chosen in a way that they cover a broad variety of semantic structures. All sentences would be expressible in OWL and would cover most of the axiom types provided by the OWL standard. The rst ontograph Sue
Tom
Mary
Bill $ $ ? ?
Tom
Lara Tom Sue
$ contains only individuals and types but no relations. The second ontograph introduces relations. The statements of the third ontograph use more complicated structures like domain, range, and cardinality restrictions. The statements of the fourth ontograph, nally, use no individuals and no types but talk only about relations. In this way, we cover a subset of ACE that corresponds to a subset of rst-order logic that is similar to the one used by OWL. 3
Figure 2 shows the result of the experiment. Overall, the decision rate (i.e. the percentage of subjects who classi ed a particular sentence as either true or false) was 93%, and on average each decision took 21 seconds. The decisions were correct in 85% of the cases (compared to 50% which can be achieved by mere guessing). The chart shows that the incorrect decisions are not equally distributed. In the case of ve statements the incorrect behavior is predominant. The statements 1/3, 1/10, 2/7, and 2/9 represent special cases and for this reason it is not unexpected that they are often classi ed wrongly.
The statement 1/3 is a very simple statement using \or" that is true but was often classi ed as false. One could think that many subjects misinterpreted the \or" as being exclusive (instead of inclusive as ACE de nes it). However, if that is the case then the subjects should also interpret the \or" of statement 1/9 as exclusive, but they did not. A more plausible explanation is that the subjects recognized that the statement 1/3 is imprecise in the sense that using \and" instead of \or" would be more accurate. This prevented some of the subjects from realizing that the statement is nevertheless true in a logical sense. Things are di erent with statement 1/9 where the replacement of \or" by \and" would not be more accurate but would make the true statement false. As a result, 1/9 was classi ed correctly by almost all subjects. It seems that people in such cases often fail to distinguish accuracy from logical truth.
The statement 1/10 is another special case. It is a conditional statement with a false precondition. It is not surprising that people with no background in logics fail to classify this statement in a correct way. We presented a framework to test controlled natural languages relying on the graphical notation of ontographs. We applied this framework to test ACE and come to the conclusion that most ACE sentences are understood very well and very quickly. These results and individual discussions with the subjects let us conclude that ontographs are a suitable and powerful concept for understandability evaluations of CNLs. A larger and more thorough experiment that compares ACE to another formal language is currently being set up.