Methodology for Reliable Schema Development and
Evaluation of Manual Annotations
Petra S. Bayerl Harald Lüngen
Applied and Computational Linguistics Applied and Computational Linguistics
Otto-Behaghel-Strasse 10D, 35394 Giessen Otto-Behaghel-Strasse 10D, 35394 Giessen
Justus-Liebig-University, Germany Justus-Liebig-University, Germany
Petra.S.Bayerl@psychol.uni-giessen.de Harald.Luengen@germanistik.uni-giessen.de
Ulrike Gut Karsten I. Paul
Department of English Organizational and Social Psychology
Fahnenbergplatz, 79085 Freiburg Lange Gasse 20, 90403 Nürnberg
Albert-Ludwigs-University, Germany Friedrich-Alexander-University, Germany
ulrike.gut@anglistik.uni-freiburg.de Paul.Karsten@wiso.uni-erlangen.de
ABSTRACT different coders. Both types of inconsistencies indicate a re-
The quality of manual annotations of linguistic data depends duced usability of annotated data.
on the use of reliable coding schemas as well as on the abil-
ity of human annotators to handle them appropriately. As As many annotation projects have shown, especially highly
is well known from a wide range of previous experiences an- complex coding schemas are difficult to use and will thus
notations using highly complex coding schemas often lead to often lead to an unacceptable low quality in manual anno-
unacceptable annotation quality. Reducing complexity might tations [29, 4]. Most of the time, researchers will choose
make schemas easier to handle, but in this way valuable in- to reduce the complexity of schemas to make them more
formation needed for more sophisticated applications is ex- manageable for human annotators. [29], for instance, re-
cluded as well. In order to deal with this problem, we devel- duced her schema from originally 31 to seven basic cate-
oped a systematic approach to schema development, which gories for these reasons. But even when the evaluation of
allows for developing coding schemas for fine-grained se- an annotation schema leads to good results in terms of re-
mantic annotations while systematically securing the quality liability, the number of categories might still be reduced to
of such annotations. For illustration, we present examples yield even better results [20]. This procedure, however, has
from two projects where text and speech data are annotated. the severe drawback of also reducing the amount of informa-
tion which can be represented in linguistic data and which
Keywords may be valuable for more complex research questions or ap-
schema development, reliability, kappa, semantic annotation, plications such as information extraction, word-sense disam-
speech data biguation, document layout, or in the context of the semantic
web [24, 32, 27] Hence, it seems vital to develop a system-
INTRODUCTION
atic approach to schema development with the aim to create
Despite efforts to automatize annotations of linguistic data highly complex reliable coding schemas which are nonethe-
[33, 3] manual annotations still play an important role in the less manageable for human annotators. The approach pre-
compilation of corpora and linguistic research material. The sented here consists of measuring the reliability of the newly
quality of such manual annotations depends on the use of developed schema, systematically considering and identify-
adequate and reliable coding schemas, which define the cat- ing sources of unreliability by statistical means, and thus iter-
egories underlying annotations of linguistics data. Their de- atively evaluating and improving the schema. The resulting
velopment and evaluation must therefore be seen as one of methodological framework is believed to be fruitful, espe-
the major tasks in annotation projects. Schema development cially as most previous approaches seem to lack a systematic
is crucial because unreliable schemas may lead to inconsis- methodology of schema development and evaluation, which
tencies in the labeling of objects not only by a single coder often makes complex schema usage so dissatisfying. Ap-
over time, but also to inconsistencies in the labeling among proaches like the one presented by [8] for a dialogue coding
Permission to make digital or hard copies of all or part of this work for scheme still seem to be an exception.
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that In the remainder of the article, we first want to describe the
copies bear this notice and the full citation on the first page. To copy oth-
erwise, or republish, to post on servers or to redistribute to lists, requires
basic principles of the methodological framework and there-
prior specific permission and/or a fee. after demonstrate its applicability by presenting data from
K-CAP’03, October 23–26, 2003, Sanibel Island, Florida, USA. two separate annotation projects.
Copyright 2003 ACM 1-58113-583-1/03/0010. . . $5.00
Critical Issues in Schema Development for systematic variance in manual annotations. Problematic
The task of annotating linguistic data can be seen as a cat- features in the schema or its application must thus be ana-
egorization task in which objects (e.g. morphemes, words, lyzed thoroughly in order to ensure or improve the reliability
phrases, sentences) have to be assigned to a single category. of the coding instrument.
The assignment of one object (usually) has to be exclusive
and independent from the categorization of other objects. OUTLINE OF METHODOLOGY
To be valuable, the prerequisite of such an assignment is The above considerations led to the design of a methodologi-
that identical objects will be assembled in the same category, cal framework for systematic schema development and eval-
which leads to the following conclusions: uation. It comprises five successive steps, which are repeated
as long as the results are not considered to be sufficient. These
steps are
1. The categories of the coding schema must be defined in a
way that enables humans to adequately differentiate among
them. Step 1: let two or more coders annotate a sufficient amount
2. The schema must be usable in a consistent way by several of data with a preliminary coding schema
persons as well as by one person over time. Step 2: repeat the annotation with the same coders and ma-
terial after a certain period of time
Step 3: check both annotations for inter- and intra-coder agree-
Although the first point seems rather trivial, it constitutes ex- ment as a measure of reliability
actly the way in which most complex coding schemas fail. Step 4: identify sources of inconsistency
Especially in the case of semantically close concepts the in- Step 5: take appropriate steps to improve the coding schema
terpretations of single coders often vary considerably [2, 35].
This leads to the second point, which refers to the question of
consistency in manual annotations and is thus directly related Methodological considerations concerning single steps will
to the issue of reliability. Reliability can be defined as ”the be considered further in the following sections.
complex property of a series of observations or of the mea-
Preliminary Considerations (Step 1)
suring process that makes it possible to obtain similar results
Before starting the evaluation process some basic facts have
if the measurement is repeated” [15, p. 51].
to be considered such as the number of coders and the amount
In contrast to the use of existing schemas where inconsis- of material to be annotated in order to give a sufficient data
tency in annotations are usually attributed to differences in basis for statistical analyses. Concerning the amount of data
the application of the schema by annotators or even to char- needed, to our knowledge, there is no evidence of how many
acteristics of the human annotators, the sources of inconsis- observations dependent on the overall number of categories
tency in schema development must be attributed to a lack of should be obtained to get reliable statistics.1 Literature on
reliability of the schema itself. Accordingly, the steps to be kappa (see below) usually mentions at least 100 observa-
taken when reliability is not high enough are supposed to be tions to get sufficient data for calculating significance [13,
different. Whereas in the case of text annotation intensive 12]. [12] propose a much larger amount of data if consid-
training and the application of supporting tools are appro- erable differences in the number of cases per categories are
priate measures to secure annotation quality [21], in schema expected.
development improvement of the coding schema must be the
Second Annotation (Step 2)
primary goal.
The second annotation is done a certain time after the first us-
When asking why schemas might be used inconsistently by ing the same material. The amount of time elapsing between
different coders or even one coder over time, several reasons the two annotations is not easy to chose, however.
may play a role. On a theoretical basis, two major problem-
As known from social sciences the time between the first and
atic aspects in schema development can be differentiated that
the second testing, i.e. annotation may influence the result
lead to systematic variance in annotation behavior [16, 31].
or the degree of agreement between the two measurements
As stated above the interpretation of categories might be am-
[9, 14]. Since linguistic annotations do not deal with per-
biguous. Secondly, the probability of assigning an object to
sonal traits or attitudes that may change over time, the impact
a category may differ among coders. In addition, annotators
might not be as severe as in the context of the social sciences.
might develop new or slightly aberrant coding habits over
Still, certain effects of time should be taken into account.
time. The latter point refers mainly to the reliability of the an-
For instance, if the two annotations lie too close together, the
notation process, but it can be hypothesized that such aberra-
coders may remember large parts of their former annotation
tions occur mostly when category definitions and boundaries
which leads to overestimations of retest-reliability. Too long
are diffuse.
1 Some considerations about suitable sample size could be found in case
As a consequence, the actions taken to improve the quality of of equal marginal frequencies or with restriction to only two categories [6].
a schema must be based on the specific reasons responsible Both cases are not applicable to our problem, however.
a period, on the other hand, might not only cause undesirable Categories – Coder 2
delays in annotations. If the annotation process is stopped A B C D E F G Σ
throughout this period, unrealistically high negative effects A 0 0 0 0 0 0 0 0
B 0 1 0 0 0 1 0 2
could appear as definitions or whole categories of a coding
Categories C 0 4 26 0 1 0 3 34
schema tend to be forgotten (especially in the case of highly Coder 1 D 0 0 0 0 0 0 0 0
complex schemas). Obviously, the calculation of the appro- E 1 1 1 1 46 0 3 53
priate period of time that should elapse before starting the F 0 0 0 1 0 2 1 4
second annotation is not as trivial as it may first seem. We, G 0 5 2 0 3 2 13 25
unfortunately, do not have a definite answer to what the ’best’ Σ 1 11 29 2 50 5 20 118
span of time is. This part is still open to research.
Table 1: Assignment decisions of two coders (imaginary data)
Measuring Inter- and Intra-Coder Reliability (Step 3)
Types of Reliability Two ways to measure the reliability of just use single categories with different frequencies [31]. The
a coding schema seem feasible, known also from social sci- comparison is based on differences in coders’ assignments to
ences for the development of rating instruments. The first single categories, e.g. the number of times coder 1 assigned
possibility is the application of a schema by different coders an object to category A (0 times) with the number of times
annotating the same material which leads to the measurement coder 2 used category A (1 times) (cp. table 1). Homogene-
of inter-coder agreement (ICA). The second is its applica- ity is assumed when the distributions, i.e. marginals do not
tion to two or more times by one coder for the same material differ significantly.
which indicates intra-coder agreement or test-retest reliabil-
ity (TRR) in terms of measurement theory [14]. These two For two coders this check can be done with the non-parame-
types are comparable to what [18] named stability and re- tric test by Stuart and Maxwell [28, 22], which calculates
producibility. The first approach thus measures consistency the overall homogeneity over all categories. Significance of
among different persons, whereas the second approach mea- the test indicates that marginal homogeneity is not given, and
sures consistency of one person over time. These measures thus a different interpretation of categories must be assumed.
can be affected by variations in the coding behavior, leading
Furthermore it is important to identify the problematic cat-
to inconsistencies, which may be both observable in man-
egories, i.e. those which are interpreted differently by the
ual annotations, without being necessarily attributable to the
coders. This can be done with the aid of the McNemar-
same sources. Since features of highly complex schemas
test [23]. This test considers the marginal distributions of
may induce both kinds of inconsistencies independently, they
schemata with only two categories, i.e. the category under
should hence be analyzed separately.
consideration and a compound in which the remaining cate-
gories are joined together. Significance of the test indicates
Calculating Reliability Inter-coder agreement and intra-coder
different interpretations of the category under consideration.
agreement (test-retest reliability) can both be calculated by
For the calculation of both statistics we resorted to the MH-
the same means. As annotations of linguistic data primar-
program developed by Uebersax. The tool can be obtained as
ily consist of annotations with mutually exclusive categories
freeware from http://ourworld.compuserve.com/
without any ordering (i.e. nominal data) calculations can be
/homepages/jsuebersax/mh.htm.
done with the κ-(kappa)-coefficient developed by [10]. κ
measures the agreement between two coders while correcting CASE 1: CODING SCHEMA FOR SEMANTIC TEXT
for chance agreement, which is the reason why it should be ANNOTATIONS
preferred to the mere calculation of the percentage of agree- Setting
ment [7]. The value of the resulting kappa-coefficient indi- The methodological framework for schema development was
cates the degree of agreement. For the interpretation of the developed within an annotation project at the University of
resulting kappa one may refer to rules of thumb like those Giessen. The aim of this project is the analysis of the seman-
given by [19], where 0 ≤ κ < 0.2 means light agreement, tics of document structures.2 For this purpose, English and
0.2 ≤ κ < 0.4 fair agreement, 0.4 ≤ κ < 0.6 moder- German scientific articles are manually annotated on mul-
ate agreement, 0.6 ≤ κ < 0.8 substantial agreement, and tiple levels, namely the structural and two semantic levels,
0.8 ≤ κ < 1.0 almost perfect agreement. In spite of several called rhetorical and thematic. The thematic structure of the
problems with this measure of agreement [1], kappa has the article describes the ’text world’ that is referred to by the ar-
advantage of being widely accepted and easy to calculate. ticle, the article’s rhetorical structure describes the rhetorical
relations that hold between the discourse units of the article.
Identifying Sources of Unreliability (Step 4)
To detect possible reasons for lack of agreement, we decided
to test the homogeneity of marginal distributions, which can 2 Project C1/SemDoc, DFG-Forschergruppe 437/Texttechnologische In-
be seen as an indicator of whether coders have different in- formationsmodellierung. For more detailed information about the project
terpretations of the meaning of categories or whether coders see http://www.text-technology.de/
Category Definition TRR
assumption theoretical assumption or supposition by ICA coder 1 coder 2
the author text 1 .18 .45 .64
theoreticalBasis well established theoretical knowledge in text 2 .26 .55 .62
the research area mean .22 .50 .63
hypothesis concrete formulation of a statistically ICA: inter-coder agreement; TRR: test-retest reliability
testable assumption, which is to be either
Table 4: Degrees of agreement at the first evaluation cycle
corroborated or refuted by the results of the
[kappa-values]
study
Table 3: Examples of category definitions Number Significance
Category Coder 1 Coder 2 Level
2 4 10 0.031
While we could resort to existing coding schemas for the 3 9 0 0.004
structural and rhetorical level, which only needed to be ad- 5 45 79 0.000
justed to our purposes, the thematic schema had to be devel- 6 19 0 0.000
oped nearly from scratch. Using existing schemas [17, 30] 7 19 0 0.000
as well as analyses of sample scientific articles as a start- 11 9 0 0.004
ing point we compiled a coding schema of originally 71 top- 12 6 18 0.011
ics such as method, history, and inducements. By 20 9 20 0.001
applying the schema to a wide range of documents, it was Table 5: Differently interpreted categories
extended to presently 121 different topics. Some of these
categories represent very subtle semantic differences (see ta-
ble 3), which made it necessary to accomplish the annotation culated with data from the first annotations of each coder.
task manually. The annotation itself is done by hand in an The rather low kappa coefficients led to the question of why
XML-format in the style of [25]. A small part of an anno- such a low agreement was obtained and, in turn, where the
tated document is shown in figure 2. causes for the lack in agreement could be found.
Guidelines defining the topics and clarifying problematic cases Step 4: Identifying Sources of Unreliability As our results
were written. At the beginning of the annotation process from the first evaluation cycle show interpretations turned out
the quality, measured in terms of inter-coder agreement, was to differentiate considerably. The Stuart-Maxwell test was
very low. We obtained agreement rates between κ = .09 and highly significant (χ2 = 90.42; p < 0.001). The McNemar-
κ = .50 (m = .22) for two coders each annotating the same Text for single categories showed that in the first evaluation
six documents. Since the annotation quality did not improve cycle eight categories were interpreted differently by the two
much during the following annotation sessions we attributed coders (table 5). Two differences, however, occurred be-
the problem to the annotation schema itself. Since we did not cause coder 1 introduced new topics which was therefore not
want to reduce our schema in order to retain as much infor- known to coder 2 (categories 6 and 7).
mation as possible we decided to develop a methodology to
improve the usability of the coding schema instead. By further checking the types and number of categories an-
notated by both coders we found the effect that the first coder
First Evaluation Cycle annotated many more different categories than the second
Steps 1 and 2: Annotations As in the project three separate coder. In text 1 and text 2 the first coder annotated 71 and
annotation levels are used, the number of annotators for each 51 categories, respectively, whereas coder 2 chose between
level was kept to the minimum of two annotators each. In 39 categories in text 1 and 34 categories in text 2. In this light
order to meet the requirements for kappa (see above) and to the higher TRR-values of coder 2 do not seem so surprising
ensure a more or less even distribution in the probability of any more.
occurrences of topics we decided to annotate two complete
scientific articles for each evaluation cycle. The chosen arti- Step 5: Adjustment of the Schema Starting from the statis-
cles for the first annotation cycle contained between 102 and tical evidence, we now began to adjust our coding schema.
192 segments to be annotated leading to an average number First, we discussed the problematic categories from table 5
of 293 annotated topics for each coder. The two coders anno- with the annotators to clarify their understanding. Defini-
tated independently from each other. The second annotation tions were adjusted and fixed in the annotation guidelines
was done approximately two weeks after the first. like in case of category 23 (table 6). The two newly invented
categories 6 and 7 were dropped because discussion showed
Step 3: Calculating Reliability For the two documents in our that they could be subsumed in two existing categories. The
first evaluation cycle we obtained kappas at the slight to mod- differences in annotation behavior of the two coders concern-
erate level of agreement (see table 4), which clearly could not ing the use of a different amount of categories were also dis-
be considered as satisfying. Inter-coder agreement was cal- cussed and more rigorous guidelines established.
In these situations, it does not matter
whether he or she mentions any information during discussion.
However, Stasser(1988;
see also Stasser et al., 1989) identified a types of information distribution in which the
best decision is not apparent to members prior to discussion.
This is termed a hidden profile.
Table 2: Part of an annotation at the thematic level
Category Definition Number Significance
textual (old) statements of the author’s intentions or Category Coder 1 Coder 2 Level
about the organization of text or text parts 12 0 15 0.000
textual (new) statements of the author’s intentions or 23 1 9 0.011
about the organization of text or text parts, 25 10 2 0.021
also information for further reading; table
captions are excluded Table 8: Differently interpreted categories
Table 6: Adaptation of definition for category 23
ers and has set up a large corpus of annotated speech files.
TRR These were annotated by six coders using a six-tier coding
ICA coder 1 coder 2 schema. On the first tier, type of phrases (e.g. complete, in-
text 1 .44 .80 .55 terrupted) and intervening non-speech events such as laugh-
text 2 .40 .74 .64 ter and noise are coded. The second tier consists of an or-
mean .42 .77 .60 thographic annotation of words. On the third tier, syllables
ICA: inter-coder agreement; TRR: test-retest reliability are annotated in SAMPA [34], and on the fourth tier vowel
Table 7: Degrees of agreement at the second evaluation cycle
and consonant boundaries are annotated. On tier 5, tones are
[kappa-values]
annotated using the ToBI [26] system, and on the sixth tier,
initial highs, final lows and intermediate highs and lows of
pitch are marked. For one speech file, an average of 1000
Second Evaluation Cycle annotations are carried out. All annotators were trained for
After the modifications of the coding schema a new evalua- two months at the beginning of the project.
tion cycle started, which included the same steps as described
above. In the second evaluation cycle we obtained the results For the calculation of inter-coder agreement one speech file
stated in table 7. ICA values were nearly twice as high than in consisting of 368 words was annotated separately by three
cycle 1. Also TRR values for coder 1 increased considerably. to four annotators. For a measure of overall agreement the
(Data for the second annotation of coder 2 was not available median of all pairwise comparisons per tier (kappa-values)
in time, but will be ready in short.) According to [19] the was calculated. Since orthographic environment, i.e. words
test-retest reliability for coder 1 could now be considered as and syllables can not be considered as categories, no agree-
substantial to almost perfect, indicating that the schema may ment was calculated for the second and third tier. The results
be used consistently over time by a single coder. Inter-coder of pairwise and overall agreement for each tier are shown
agreement turned from fair to moderate. in table 9. Kappa-values clearly indicate that certain tiers are
more difficult to annotate in agreement than others, e.g. tones
The test for marginal homogeneity still was highly signifi- and phrases. These differences seem attributable mainly to
cant (χ2 = 153.02; p < 0.001). The comparison of sin- the complexity of the underlying schemas as the number of
gle categories, however, showed that three instead of the for- categories from tier 1 to tier 6 are seven (phrases), three
mer eight categories were not used in accordance (table 8). (vowels), 34 (tones), four (pitch). For the calculation of retest-
Hence, other evaluation cycles will follow in the near future reliability the first file annotated was annotated again two
to further improve the coding schema. years later by each coder. Results for tier 1 and tier 4 show
that kappa-values are on a moderate level of agreement (ta-
CASE 2: EVALUATION OF ANNOTATIONS OF SPEECH ble 10). In the light of the long period of time that elapsed
DATA between the first and the second annotation this must still be
We also tested our methodological approach for coding schema seen as a rather good result.
evaluation with data from another project. The LeaP (http://
leap.lili.uni-bielefeld.de) project is concerned An evaluation of the reasons for disagreement will be pre-
with the acquisition of prosody by foreign language learn- sented here only for the pair coder 1 – coder 3 in the first tier
Coder Pair Coder Pairs
Tier 1-2 1-3 1-4 2-3 2-4 3-4 Median Tier 1-2 1-3 1-4 2-3 2-4 3-4 Median
1- phrases .40 .39 .43 .57 .63 .60 .50 1- phrases .86 .92 .88 .89 .93 .90 .90
4- vowels .46 .46 .52 .46 .46 .49 .46 4- vowels .99 1.00 .99 .99 1.00 .99 .99
5- tones .21 .20 .29 .30 .35 .25 .27 5- tones .44 .44 .58 .56 .58 .51 .54
6- pitch .58 .68 – .62 – – .62 6- pitch .96 .94 – 1.00 – – .96
Table 9: Inter-coder agreement at different annotation tiers Table 11: Inter-coder agreement in case 2 [corrected
[kappa-values] kappa-values]
Coder Kappa as Measure of Reliability
Tier 1 2 3 4 One of the major problems when employing kappa is that
1- phrases .53 .24 .51 .65 the coefficient depends on the actual marginal distribution
4- vowels .58 .35 .46 .53
[11, 5]. In cases with heterogeneous marginal distributions
Table 10: Retest-reliability at different annotation tiers kappa may not have the originally intended range of −1 to
[kappa-values] +1, but a more restricted one. This will not only reduce the
kappa-values obtained, but also the interpretability of the co-
efficient, since rules of thumb for interpreting the goodness
(inter-coder agreement), since this is the pair with the low- of the coefficient [19] do not apply anymore.
est agreement on this level. Procedure and interpretation are
identical to those described in case 1. As the only tenden- In this case some authors suggest the calculation of the pos-
tially significant Stuart-Maxwell Test (χ2 = 12.404; p < sible maximum that kappa can reach (κmax ) with the given
0.05) proposes, the overall interpretation of categories can marginal distribution [10, 1]. The expression κ/κmax will
be considered as nearly identical. This leads to the conclu- then lead to a corrected κ with the original range of −1 to
sion that the differences are attributable primarily to a sys- +1 [10, 1]. Even though this procedure would have the big
tematic variance in assigning objects to different categories. advantage of not only tremendously improving the kappa-
Additionally, however, the McNemar-Test reveals that there values (see table 11 for an example), but also of restoring the
is one category in the schema (category 2) that is interpreted original interpretation of kappa, we refrained from using it
differently (χ2 = 7.36; p < 0.05). The implication in this in the context of our framework. Severe aberrations from the
case would be to first clarify the definition of the problematic homogeneity of marginal distributions often indicate under-
category with both coders, and then to resume training with lying problems with the use of the categories. By correcting
the aim of improving the differentiation between objects. kappa, valuable information would be discarded.
CONCLUSIONS
PRACTICAL PROBLEMS
The aim of the work presented here was to present hands-
In applying the methodology to the two projects described on experience with the development of highly complex cod-
above we encountered some practical problems, which might ing schemas for manual annotations of linguistic data. The
be worth noting, since they are likely to occur in other appli- methodological framework we created in order to solve our
cations as well and in quite a similar way. problems with poor annotation quality because of the high
complexity of the annotation task proved fruitful not only in
Coder Characteristics the context of our original project aiming at the semantic an-
notation of text documents, but also in translating it to the
In our case studies we assumed that coder characteristics
annotation of speech data. We therefore feel confident that
were stable or had no direct influence on annotation qual-
the systematic and iterative process presented here can prof-
ity. This of course is an overly optimistic view. Individual
itably be applied in other annotation projects, where complex
characteristics of coders such as familiarity with the mate-
coding schemas have to be developed and evaluated.
rial, amount of former training, but also motivation and in-
terest may clearly have a varying impact on their work. In REFERENCES
both studies we tried to keep these variables as stable as pos- 1. Brennan, R.L. and Prediger, Dale J. (1981). Coefficient
sible by providing equal training for every coder, choosing kappa: Some uses, misuses, and Alternatives. Educational
annotators familiar with the subject or material and giving and Psychological Measurement, 41, 687-699.
guidelines for the annotation process aimed at reducing ef-
fects of fatigue (e.g. restricting the annotation time to max- 2. Bruce, R. and Wiebe, J. (1999). Recognizing subjectivity: A
imally three hours per session). Nonetheless, as interaction case study in manual tagging. Natural Language Engineer-
ing, 5(2), 187-205.
effects of coder characteristics and coding task cannot be ex-
cluded, the choice of a group of similar coders should be 3. Bulyko, I. and Ostendorf, M. (2002). A bootstrapping ap-
aspired. proach to automating prosodic annotation for limited-domain
synthesis. Proceedings of the IEEE Workshop on Speech Syn- 22. Maxwell, A. Comparing the classification of subjects by two
thesis, 11-13 September, Santa Monica, California USA. independent judges. British Journal of Psychiatry, 116, 651-
4. Butler, T., Fisher, S., Coulombe, G., Clements, P., Brown, S., 655.
Grundy, I., Carter, K., Harvey, K. and Wood, J. (2000). Can 23. McNemar, Q. Note on the sampling error of the difference be-
a team tag consistently? Experiences on the Orlando project. tween correlated proportions or percentages. Psychometrika,
Markup Languages, 2(2), 111-125. 12, 153-157.
5. Byrt, T., Bishop, J. and Carlin, J.B. (1993). Bias, prevalence 24. Ng, H.T., Lim, C.Y. and Foo, S.K. (1999). A Case Study on
and kappa. Journal of Clinical Epidemiology, 46(5), 423-429. Inter-Annotator Agreement for Word Sense Disambiguation.
Proceedings of the ACL SIGLEX Workshop: Standardizing
6. Cantor, A.B. (1996). Sample-size calculations for Cohen’s
Lexical Resources, College Park, Maryland, USA, 21-22 June
kappa. Psychological Methods, 1(2), 150-153.
1999, 9-13.
7. Carletta, J. (1996). Assessing agreement on classification
25. O’Donnell, M. RST-Tool 2.4 - A Markup Tool for Rhetorical
tasks: The kappa statistic. Computational Linguistics, 22(2),
Structure Theory. Proceedings of the International Natural
249-254.
Language Generation Conference (INLG’2000), Mitzpe Ra-
8. Carletta, J., Isard, A., Isard, S., Kwotko, J.C., Doherty- mon, Israel, 12-16 June 2000, 253-256.
Sneddon, G. and Anderson, A.H. (1997). The reliability of
26. Silverman, K., Beckman, M., Pitrelli, J. Ostendorf, M.,
a dialogue structure coding scheme, 23(1), 13-31.
Wightman, C., Price, P., Pierrehumbert, J. and HirschbergJ.
9. Carmines, E.G. and Zeller, R.A. (1979). Reliability and va- (1992). ToBI: A standard for labeling English prosody.
lidity assessment. Sage Publications: Beverly Hills and Lon- Proceedings of the 1992 International Conference on Spo-
don. Paper series on Quantitative Applications in the Social ken Language Processing, Denver Colorada, USA, 16-20
Sciences, 07-017. September 1992, 867 – 870.
10. Cohen, J. (1960). A coefficient of agreement for nominal 27. Staab, S., Maedche, A. and Handschuh, S. (2001). Creating
scales. Educational and Psychological Measurement, 20(1), metadata for the semantic web: An annotation framework
37-46. and the human factor. Technical Report 412. Institute AIFB,
11. Feinstein, A.R. and Cicchetti, D.V. (1990). High agreement University of Karlsruhe.
but low kappa: I. The problem of two paradoxes. Journal of 28. Stuart, A. (1955). A test for homogeneity of the marginal dis-
Clinical Epidemiology, 43(6), 543-549. tributions in a two-way classification. Biometrika, 42, 412-
12. Flack, V.F., Afifi, A.A., Lachenbruch, P.A. and Schouten, 416.
H.J.A. (1988). Sample size determinations for the two rater 29. Teufel, S. (1999). Argumentative zoning: Information extrac-
kappa statistics. Psychometrika, 53(3), 321-325. tion from scientific text. PhDThesis, University of Edinburgh.
13. Hanley, J.A. (1987). Standard error of the kappa statistic. Psy- 30. Teufel, S., Carletta, J. and Moens, M. (1999). An annotation
chological Bulletin, 102(2), 315-321. scheme for discourse-level argumentation in research articles.
14. Helmstadter, G.C. (1964). Principles of psychological mea- Proceedings of the Ninth Conference of the European Chap-
surement. Meredith Publishing: New York. ter of the Association for Computational Linguistics (EACL-
99), Bergen, 8-12 June 1999.
15. Hollnagel, E. (1993). Human Reliability Analysis Context
and Control. Academic Press: London. 31. Uebersax, J. (2001). Statistical Methods for Rater Agreement.
online available: http://ourworld.compuserve.com/
16. Hoyt, W. and Kerns, M.-D. (1999). Magnitude and modera- homepages/jsuebersax/agree.htm.
tors of bias in observer ratings: A meta-analysis. Psychologi-
cal Methods, 4(4), 403-424. 32. Veronis, J. (2000). Sense tagging: Don’t look for the mean-
ing but for the use. Workshop on Computational Lexicog-
17. Kando, N. (1997). Text-level structure of research papers: raphy and Multimedia Dictionaries (COMLEX’2000), 22-23
Implications for text-based information processing systems. September 2000, Patras, Greece, 1-9.
Proceedings of the British Computer Society Annual Collo-
quium of Information Retrieval Research, Aberdeen, Scot- 33. Vorsterman, A., Martens, J.-P. and Coile, B. van. (1996). Au-
land, 8-9 April 1997, 68-81. tomatic segmentation and labeling of multi-lingual speech
data. Computational Linguistics, 19(4), 271-293.
18. Krippendorff, K. (1980). Content analysis: An introduction.
Sage Publications: Beverly Hills and London. 34. Wells, J.C., Barry, W., Grice, M., Fourcin, A. and D. Gibbon.
(1992). Standard Computer-Compatible Transcription. SAM
19. Landis, J.R. and Koch, G.G. (1977). The measurement of ob- Stage Report Sen.3 SAM UCL-037, University College Lon-
server agreement for categorical data. Biometrics, 33(1), 159- don.
174.
35. Wiebe, J.M., Bruce, R.F. and O’Hara, T.P. (1999). Develop-
20. Maier, E. (1997). Evaluating a Scheme for Dialogue Annota- ment and use of a gold standard data set for subjectivity clas-
tion. V ERBMOBIL Report 193. DFKI GmbH, Saarbrücken. sifications. Proceedings of the 37th Annual Meeting of the
21. Marcu, D., Romera, M. and Amorrortu, E. (1999). Experi- Association for Computational Linguistics (ACL-99), 20-26
ments in Constructing a Corpus of Discourse Trees: Prob- June 1999, University of Maryland, 246-253.
lems, Annotation Choices, Issues. The Workshop on Levels of
Representation in Discourse, Edinburgh, Scotland, 71-87.