English. We present the results of prototypical experiments conducted with the goal of designing a machine translation (MT) based system that assists the annotators of learner corpora in performing orthographic error annotation. When an annotator marks a span of text as erroneous, the system suggests a correction for the marked error. The presented experiments rely on word-level and characterlevel Statistical Machine Translation (SMT) systems.
Italian. Presentiamo i risultati degli
esperimenti prototipici condotti con lo
scopo di creare un sistema basato sulla
traduzione automatica (MT) che assista
gli annotatori dei corpora degli
apprendenti di lingue durante il processo di
annotazione degli errori ortografici.
Quando un annotatore segna un segmento di
testo come errato il sistema suggerisce
una correzione dell’errore segnato. Gli
esperimenti presentati utilizzano dei
sistemi statistici di traduzione automatica
(SMT) al livello di parole e di caratteri.
Manual error annotation of learner corpora is a
time-consuming process which is often a
bottleneck in learner corpora research. “Computer
learner corpora are electronic collections of
authentic FL/SL textual data assembled according
to explicit design criteria for a particular
SLA/FLT1 purpose. They are encoded in a
stand1FL: foreign language, SL: second language, SLA:
second language acquisition, FLT: foreign
language teaching
ardised and homogeneous way and documented
as to their origin and provenance” (Granger,
2002). Error-annotated learner corpora serve the
needs of language acquisition studies and
pedagogy development as well as help the creation of
natural language processing tools such as
automatic language proficiency level checking
systems
This paper is organized as follows: we review existing approaches to automatic error correction (Section 2), introduce our experiments (Section 3), present the data we used (Section 4), describe and discuss the performed experiments (Section 5) and conclude the paper (Section 6). 2
Orthographic errors are mistakes in spelling,
hyphenation, capitalisation and word-breaks
SMT and NMT can be easily adapted to new
languages, but their performance depends on the
amount and quality of the training data. In order
to make up for lack of parallel corpora of texts
containing language errors and their correct
equivalents, various techniques for resource
construction have been suggested, such as using the
World Wide Web as a corpus
In recent years learner corpora of German have
been used for the creation of systems for
automatic German children’s spelling errors
correction
The particularity of our work is that we focus on a specific use-case where annotators are assisted in error-tagging newly created learner corpora. To ensure the relevance of our system and limit false positives that would hinder its adoption, the targeted use-case is to only suggest corrections while leaving the task of selecting the error to the linguist. Aforementioned GEC systems take as input text containing language errors and produce corrected text. Thus, they may introduce changes in any part of the text, even where no errors are observed. In order to prevent such behavior, we only submit to our system spans of text marked as erroneous by annotators, while leaving out spans of text not containing errors. Therefore, our system is not directly comparable to existing GEC systems.
A given language error may have more than one
possible correction, but in the presented research
we limit ourselves to orthographic errors that in
most cases have only one correction
As our systems are not directly comparable to GEC systems, the usual metrics used to evaluate GEC systems are not fully adequate, because they target a similar but different use case. We thus evaluate our systems according to their accuracy that we define as a ratio between the number of suggestions matching the target hypothesis present in the test data (TH)3 and the whole number of annotated errors. However, accuracy is not the only criteria as it is also important not to disturb the annotators with irrelevant suggestions: it is better not to suggest any TH than to suggest a wrong one. In order to control the ratio between right and wrong suggestions, we also evaluate our systems according to their precision. We define precision as a ratio between the number of suggestions matching the TH and the whole number of suggestions, correct and incorrect, thus excluding the errors for which the system was consulted, but no correction was suggested. Precision is mainly used as a quality threshold which should remain high, whereas our main performance measure is accuracy. 4
Our experiments rely on three error-annotated
learner corpora: KoKo, Falko and MERLIN.
KoKo is a corpus of 1.503 argumentative essays
(811.330 tokens) of written German L14 from
high school pupils, 83% of which are native
speakers of German
The Falko corpus consists of six subcorpora
The MERLIN corpus was compiled from
standardized, CEFR6-related tests of L2 German,
Italian and Czech
Due to the differences in content and format, we do not use all three learner corpora in all the experiments. KoKo is our main corpus, because of its larger size, easy to use format and detailed orthographic error annotation. We use it in training, validation and testing of our SMT systems. Falko is smaller and its format does not allow an easy alignment of orthographic errors, we thus only use it in some experiments as part of the training corpus (Sections 5.1 and 5.2). MERLIN was annotated similarly to KoKo, therefore error-correction results obtained for these two corpora are easily comparable. Furthermore, MERLIN is representative of different levels of language mastery. We thus use it for testing some of our systems (Section 5.2).
As the language model for our character-based SMT systems cannot be generated from the limited amount of data provided by learner corpora, for that purpose we used 3.000.000 sentences of a German news subcorpus from the Leipzig Corpora Collection7.
5second language, foreign language 6Common European Framework of Reference for
Languages 7http://hdl.handle.net/11022/0000-0000-2417-E 8 The computational results presented have been achieved in part using the Vienna Scientific Cluster (VSC). learner texts and their corrected versions from Falko and KoKo. In each fold of the 10-fold validation, 1/10 of KoKo is taken out of the training corpus and used as a validation corpus.
Since our objective was to only observe the overall adequateness of the SMT models, we only attempted to optimise the way the SMT models were used at a later stage (see Section 5.4). These prototypical experiments showed that all the SMT models have a rather high precision and that, for this amount of training data, the SMT model that performed best is the word 5-gram model. It yielded an encouraging result of 39% of accuracy and 89% of precision, which is far better than the 11% of accuracy and 8% of precision originally obtained with Hunspell. However, 39% of accuracy were obtained by training on Falko and 9/10 of KoKo and validating on 1/10 of KoKo, which would be the configuration we would have towards the end of the annotation of a new learner corpus. We thus proceeded with our experiments by testing how the SMT models would perform at an earlier stage.
1 word-grams 3
5 Prec. 84%
87% 89% Acc. 32% 37% 39%
character-grams 10 84% 38% 6
10 83% 86% 16% 21% 15 87% 29% At an early stage of the annotation of a new learner corpus, an error-correction system could be trained on an already existing corpus. We thus tried to apply the different models trained on Falko, KoKo and the newspapers to MERLIN. However, none of the 7 models presented in the previous section achieved more than 13% of accuracy and 70% of precision on the whole MERLIN corpus. Despite that, these experiments highlighted an interesting aspect: all the models performed better on MERLIN texts of higher CEFR levels compared to MERLIN texts of lower CEFR levels (Table 2). We suspect this phenomenon to be due to the fact that the level of language mastery of MERLIN texts of higher CEFR levels is closer to the level of language mastery of KoKo and Falko texts. This observation indicates that the training and test data must attest to the same level of language mastery, because mistakes made by beginner language learners tend to differ noticeably from mistakes made by advanced language learners. Therefore, 5 5.1
We started by testing SMT word and characterbased language models with various numbers of n-grams in order to understand which one could suffer less from data scarcity and thus best suit our data8 (Table 1). We used Moses default values for all the other parameters. The systems were trained on a parallel corpus composed of using existing learner corpora as training data is a difficult task as most of them target different types of learners with different profiles and bias towards specific kinds of errors.
Prec.
Acc.
A1 The results of the previous experiments incited us to train an SMT model on a small part of a corpus and test it on a bigger part of the same corpus in order to observe how an SMT model would behave when trained on an already annotated part of a new learner corpus. We thus performed 3-fold validation experiments with a word 5-gram model taking 1/3 of KoKo as training data and 2/3 of KoKo as test data and obtained 30% of accuracy9. This result was much better than 13% of accuracy we had obtained by training SMT systems on KoKo and Falko and testing them on MERLIN. We thus decided to pursue our experiments with KoKo as both training and test data.
In order to observe the evolution of the system’s performance with the growth of the corpus, we also trained it on 2/3 of KoKo and tested it on 1/3 of KoKo. Augmenting the training corpus size did not change the system’s performance (Table 3, line 1). Such results tend to indicate that most of the performance can be obtained at an earlier stage of the annotation process. 5.4
After evaluating the impact of the training data on the system’s performance, we switched our focus to the optimisation of the way SMT models were used. First of all, we tried to take into account not only the highest-ranked suggestion of Moses, that in many cases was equal to the error text (i.e. no correction was suggested), but also the lower-ranked suggestions in order to find the highest-ranked suggestion that was different from the error text. This change considerably improved the accuracy for both corpus sizes and 9We also calculated the BLEU score for this model and obtained 95%. This result shows that the BLEU score is irrelevant for the evaluation of error correction systems such as ours that cannot introduce errors in error-free spans of text. only slightly deteriorated the precision (Table 3, line 2).
In order to further improve the performance, we decided to combine the word-based and character-based systems. For this first experiment we chose the best-performing of the word-based systems which is the word 5-gram model and the second best performing of the character-based systems which is the character 10-gram model. We chose the character 10-gram model for practical reasons: it is considerably less resourceconsuming than the character 15-gram model. By applying both the word 5-gram and the character 10-gram models to the same data and comparing the overlap in their responses, we verified their degree of complementarity. This experiment showed that only in 18% of cases the word-based and character-based models both suggest a correction (corresponding or not to the TH). In 39% of cases only the word-based system suggests a correction and in 5% of cases only the characterbased system suggests a correction. It means that by combining the two systems it is possible to improve the overall performance. We calculated the maximum theoretical accuracy10 of such a combined system and came to a conclusion that it cannot exceed 53% when trained on 1/3 of KoKo and 60% when trained on 2/3 of KoKo (Table 3, line 3).
By simply giving preference to the word-based model before consulting the character-based model, we almost achieved the maximum theoretical accuracy (Table 3, line 4).
However, we realised that by augmenting the training corpus size, we augmented the accuracy, but slightly deteriorated the precision.
By analysing the performance of different
modules (word 5-gram highest-ranked suggestions,
word 5-gram lower-ranked suggestions,
character 10-gram) on different kinds of errors, we
could observe that their performance differs
according to types of errors. For example, the
lower-ranked suggestions of the word-based model
introduce a lot of mistakes in the correction of
errors where one word was erroneously written
as two separate words (e.g. Sommer fest instead
10The maximum theoretical accuracy would be
achieved if it was possible to always choose the
right system to consult for each precise error
(word-based or character-based) and never
consult the system that gave a wrong result when the
other system gave a correct result. In that case the
maximum potential of both systems would be
used.
of Sommerfest). We tried to prevent such false
corrections by not consulting the lower-ranked
suggestions of the word-based model for errors
containing spaces. By introducing this rule we
succeeded in improving the precision at the cost
of loosing some accuracy (Table 3, line 5). This
experiment showed that add-hoc rules might not
be a workable solution and a more sophisticated
approach should be considered if we intend to
dynamically combine several systems. In order
to obtain better results combining two or more
word-based and character-based systems, further
experiments should be conducted.
1 word highest-ranked corr.
30% (88%) 30% (88%)
2 word lower-ranked corr.
48% (84%) 55% (83%)
Our preliminary experiments brought us to the
conclusion that a SMT system trained on a
manually annotated part of a learner corpus can be
helpful in error-tagging the remaining part of the
same learner corpus: it is possible to train a
system that would propose the right correction for
half of the orthographic errors outlined by the
annotators while proposing very few wrong
corrections. Such results are satisfactory enough to
start integrating the system into the annotation
tool we use to create learner corpora
The combination of a word-based and a
character-based systems gave promising results,
therefore we intend to continue experimenting with
multiple combinations of word-based and
character-based systems. We are also considering the
possibility to rely on other technologies
The choice of training data for our experiments was dictated by the availability of high-quality resources. In future experiments we would like to enlarge the spectrum of resources considered for our experiments and work with other languages, in particular with Italian and English.
We would like to thank the reviewers as well as our colleagues Verena Lyding and Alexander König for their useful feedback and comments.