Improvements to Korektor: A Case Study with Native and Non-Native Czech

Abstract

We present recent developments of Korektor, a statistical spell checking system. In addition to lexicon, Korektor uses language models to find real-word errors, detectable only in context. The models and error probabilities, learned from error corpora, are also used to suggest the most likely corrections. Korektor was originally trained on a small error corpus and used language models extracted from an in-house corpus WebColl. We show two recent improvements: • We built new language models from freely available (shuffled) versions of the Czech National Corpus and show that these perform consistently better on texts produced both by native speakers and nonnative learners of Czech.

-

The idea of using the context of a misspelled word to improve the performance of a spell checker is not new [ 10 ]. Moreover, recent years have seen the advance of contextaware spell checkers such as Google Suggest, offering reasonable corrections of search queries.

Methods used in such spell checkers usually employ the noisy-channel or window-based approach [ 4 ]. The system described here also belongs to the noisy-channel class. It makes extensive use of language models based on several morphological factors, exploiting the morphological richness of the target language.

Errors detected by such advanced spell checkers have a natural overlap with those of rule-based grammar checkers – grammatical errors are also manifested as unlikely n-grams. Using language models or even complete SMT approach [ 8 ] for grammatical error correction is also becoming more common, however all the tasks and publications on grammar correction we have seen so far expect pre-corrected text in terms of spelling. See also [ 15 ] and Table 1 in [ 14 ] for what types of errors were subject to correction at the CoNLL 2013 and 2014 Shared Tasks on English as a Second Language.

We make no such optimistic expectations. As we show in Section 2 there are many types of spelling errors both in native speakers’ texts and in learner corpora. The error distributions are slightly different, though.

Richter [ 12 ] presented a robust spell checking system that includes language models for improved error detection and suggestion. To improve the suggestions further, the system employs error models trained on error corpora. In this paper we present some recent improvements to Richter et al.’s work in both respects: improved language models in Section 3 and task-dependent, adapted error models in Section 4. We apply native and non-native error models on both native and non-native datasets in Section 5. We analyze a portion of the systems output in Section 6 and provide some insight into the most problematic errors that various models make. Finally, we summarize our work and list potential scope for further improvements of Korektor components in Section 7. 2

Error Distribution for Native vs

Non-Native Czech Richter [11, p. 33] presents statistics of spelling errors in Czech, based on a small corpus of 9500 words, which is actually a transcript of an audio recording of a novel. The transcription was done by a native speaker. Following [ 1 ], the error analysis in Table 1 is based on the classification of errors into four basic groups: substitution, insertion, deletion/omission and swap/transposition/metathesis. Although the figures may be biased due to the small size of the corpus and the fact that it was transcribed by a single person, we still find them useful for a comparison with statistics of spelling errors made by non-native speakers.

In Table 2 the aggregate figures from Table 1 (in the last column headed by “Native”) are compared with figures from an automatically corrected learner corpus (“SGT”, or CzeSL-SGT) and a hand-corrected learner corpus (“MAN”, or CzeSL-MAN). The taxonomy of errors is derived from a “formal error classification” used in those two corpora, described briefly in Section 4. 1 In this table we follow [ 3 ] in treating errors in diacritics as dis1See [ 7 ] for more details about the classification and the http://utkl. ff.cuni.cz/learncorp/ site, including all information about the corpora.

Error Type

Substitution – horizontally adjacent letters – vertically adjacent letters – z → s – s → z – y → i – i → y – non-adjacent vocals – diacritic confusion – other cases Insertion – horizontally adjacent letter – vertically adjacent letter – same letter as previous – other cases Deletion – other cases Swap letters TOTAL tinct classes, adding their statistics on native Brazilian Portuguese for comparison in the “PT” column.

The high number of errors in diacritics in non-native Czech and native Portuguese in comparison with native Czech can be explained by the fact that native speakers of Czech are aware of the importance of diacritics both for distinguishing the meaning and for giving the text an appropriate status. The high number of errors in diacritics in learner texts is confirmed by results shown in Table 3, counted on the training portion of the “CzeSL-MAN” corpus by comparing the uncorrected and corrected forms, restricted to single-edit corrections.2 The distribution is shown separately for the two annotation levels of CzeSLMAN: somewhat simplifying, L1 is the level where nonwords (forms spelled incorrectly in any context) are cor2I.e., without using the “formal error types” of [ 7 ].

Error type

Substitution3

– Case – Diacritics Insertion Deletion Swap Total rected, while L2 is the level where real-word errors are corrected (words correct out of context but incorrect in the syntactic context). For more details about CzeSL-MAN see Section 4.1.

As an illustration of the prevalence of errors in diacritics in non-native Czech, see Table 4, showing the 12 most frequent substitution errors from L1 in Table 3. There is only one error which is not an error in a diacritic (the use of the i homophone instead of y). 3

Current Improvements for Native Czech Spelling Correction The original language model component of Korektor [ 12 ] was trained on WebColl – a 111 million words corpus of primarily news articles from the web. This corpus has two issues: (i) the texts are not representative and (ii) the language model from this data could not be distributed freely due to licensing issues. To obviate this, we evaluate Korektor using two new language models built from two corpora available from the Czech National Corpus (CNC): (i) SYN2005 [ 2 ] and (ii) SYN2010 [ 9 ]. Both have the size of 100 million words each and have a balanced representation of contemporary written Czech: news, fiction, professional literature etc.

We use the error model and the test data (only the Audio data set) described in [ 12 ]. Audio contains 1371 words with 218 spelling errors, of which 12 are real-word errors.

3The two error types below are actually subtypes of the substitution error. For the CNC corpora, we build 3rd order language models using KenLM [ 6 ].

The spell checker accuracy is measured in terms of standard precision and recall. The precision and recall measures are calculated at two levels: (i) error detection and (ii) error correction. These evaluation measures are similar in spirit as in [ 17 ]. For both levels, precision, recall and other related measures are calculated as: Precision(P) = T P+FP , Recall(R) = T P+FN , and F − score(F 1) = 2P∗+P∗RR ,

T P T P where, for error detection, • TP – Number of words with spelling errors that the spell checker detected correctly • FP – Number of words identified as spelling errors that are not actually spelling errors • TN – Number of correct words that the spell checker did not flag as having spelling errors • FN – Number of words with spelling errors that the spell checker did not flag as having spelling errors and for error correction, • TP – Number of words with spelling errors for which the spell checker gave the correct suggestion • FP – Number of words (with/without spelling errors) for which the spell checker made suggestions, and for those, either the suggestion is not needed (in the case of non-existing errors) or the suggestion is incorrect if indeed there was an error in the original word • TN – Number of correct words that the spell checker did not flag as having spelling errors and no suggestions were made • FN – Number of words with spelling errors that the spell checker did not flag as having spelling errors or did not provide any suggestions

The results for error detection and error correction are shown in Tables 5 and 6, respectively. Maximum edit distance, i.e., the number of edit operations per word is set to values from 1 to 5. In the case of error detection, the best overall performance is obtained for the SYN2005 corpus when the maximum edit distance parameter is 2, and there is no change in results for the edit distance range from 3 to 5. Of the two CNC corpora, SYN2005 consistently provides better results than SYN2010 corpus. Differences in the vocabulary could be the most likely reason.

Even in the case of error correction, the best overall performance is obtained for SYN2005 with 94.5% F1-score. We can also see that WebColl performs better in 3 out of 5 cases, but we should also note that this happens when we include top-3 suggestions in the error correction. Otherwise the SYN2005 model consistently provides better scores. We have also experimented with pruned language models and obtained similar results.

Work in Progress for Improving Spelling Correction of Non-Native Czech One of the main hurdle in obtaining a new error model is the availability of annotated error data for training. Many approaches are available to somehow obtain error data automatically from sources such as the web [ 16 ]. The error data obtained from the web may be good enough for handling simple typing errors, but not for the more complicated misspellings a learner/non-native speaker of a language makes. However, these approaches can be successfully used to obtain general purpose spell checkers. One resource which could be of some value to spell checking is the learner corpus. Unlike native error corpus, the learner corpus of non-native or foreign speakers tend to have more errors ranging from orthographical, morphological to realword errors. In this work, we try to address whether error models from texts produced by native Czech speakers can be applied to errors from non-native Czech texts and vice versa. We also derive error analysis based on the results. 4.1

CzeSL — a Corpus of Czech as a Second

Language A learner corpus consists of language produced by language learners, typically learners of a second or foreign language. Deviant forms and expressions can be corrected and/or annotated by tags making the nature of the error explicit. The annotation scheme in CzeSL is based on a twostage annotation design, consisting of three levels. The level of transcribed input (Level 0) is followed by the level of orthographical and morphological corrections (Level 1), where only forms incorrect in any context are treated. The result is a string consisting of correct Czech forms, even though the sentence may not be correct as a whole. All other types of errors are corrected at Level 2.4

This annotation scheme was meant to be used by human annotators. However, the size of the full corpus and the costs of its manual annotation have led us to apply automatic annotation and find ways of its improvement.

The hand-annotated part of the corpus (CzeSL-MAN) now consists of 294 thousand word tokens in 2225 short essays, originally hand-written and transcribed.5 A part of the corpus is annotated independently by two annotators: 121 thousand word tokens in 955 texts. The authors are both foreign learners of Czech and Czech learners whose first language is the Romani ethnolect of Czech.

The entire CzeSL corpus (CzeSL-PLAIN) includes about 2 mil. word tokens. This corpus comprises transcripts of essays of foreign learners and Czech students with the Romani background, and also Czech Bachelor and Master theses written by foreigners.

The part consisting of essays of foreign learners only includes about 1.1 word tokens. It is available as the CzeSLSGT corpus with full metadata and automatic annotation, including corrections proposed by Korektor, using the original language model trained on the WebColl corpus.6 In the annotation Korektor detected and corrected 13.24% incorrect forms, 10.33% labeled as including a spelling error, and 2.92% an error in grammar, i.e. a ‘real-word’ error. Both the original, uncorrected texts and their corrected version was tagged and lemmatized, and “formal error tags,” based on the comparison of the uncorrected 4See [ 5 ] and [ 13 ] for more details.

5For an overview of corpora built as a part of the CzeSL project and relevant links see http://utkl.ff.cuni.cz/learncorp/.

6See http://utkl.ff.cuni.cz/~rosen/public/2014-czesl-sgt-en.pdf. and corrected forms, were assigned. The share of ‘out of lexicon’ forms, as detected by the tagger, is slightly lower – 9.23%. 4.2

The CzeSL-MAN Error Models We built two error models from the CzeSL-MAN corpus – one for Level 1 (L1) errors and another for Level 2 (L2) errors. As explained in Section 4.1 above, L1 errors are mainly non-word errors and L2 errors belong to real-word and grammatical errors, but still include form errors that are not corrected at L1 because the faulty form happens to be spelled as a form which would be correct in a different context. Extracting errors from the XML format used for encoding the original and the corrected text at L1 is straightforward. The only thing needed is to follow the links connecting tokens at L0 (the original tokens) and L1 (the corrected tokens) and to extract tokens for which the links are labeled as correction links. In the error extraction process, we do not extract errors that involve joining or splitting of word tokens at either level (Korektor does not handle incorrectly split or joined words at the moment).

L2 errors include not only errors identified between L1 and L2 but also those identified already between L0 and L1, if any. This is because L2 tokens are linked to L0 tokens through L1 tokens, rather than being linked directly. For example, consider a single token at Levels L0, f ormSingCh,incorBase L1 and L2: všechy (L0) −−−−−−−−−−−−→ všechny (L1) agr −−→ všichni (L2). The arrow stands for a link between the two levels, optionally with one or more error labels. For the L1 error extraction, the extracted pair of an incorrect token and a correct token is (všechy, všechny) with the error labels (formSingCh, incorBase), and for the L2 error extraction, the extracted error and correct token pair is

Error

single-edit multi-edit (všechy, všichni) with the error labels (formSingCh, incorBase, agr). For the L2 errors, we project the error labels of L1 onto L2. If there is no error present or annotated between L0 and L1, then we use the error annotation between L2 and L1. The extracted incorrect token is still from L0 and the correct token from L2.

Many studies have shown that most misspellings are single-edit errors, i.e., misspelled words differ from their correct spelling by exactly one letter. This also holds for our extracted L1 and L2 errors (Table 7). We train our L1 and L2 errors on single-edit errors only, thus the models are quite similar to the native Czech error model described in [ 11 ]. The error training is based on [ 1 ]. Error probabilities are calculated for the four single-edit operations: substitution, insertion, deletion, and swap. 5

Experiments with Native and Non-Native Error Models For the native error model (webcoll), we use the same model as described in [ 12 ]. For the non-native error models, we create two error models as described in Section 4.2: (i) czesl_L1 – trained on the L1 errors (CzeSL-L1 data in Table 8) and (ii) czesl_L2 – trained on the L2 errors (CzeSL-L2 data in Table 8). We partition the CzeSL-MAN corpus in the 9:1 proportion for training and testing.

The non-native training data include more errors than those automatically mined from web. The training of nonnative error models is done on single-edit errors only (refer Table 7 for the percentage of errors used for training). For the language model, we use the best model (SYN2005) that we obtained from Section 3.

We perform evaluation on all kinds of errors in test data. We also set the maximum edit distance parameter to 2 for all our experiments. We arrived at this value based on our observation in various experiments. We run our native and non-native models on the test data described in Table 9, and their results are given in Table 10. Error correction results are shown for top-3 suggestions.

In error detection, in terms of F1-score, czesl_L2 model posts better score than the other two models for both native and non-native data sets. When it comes to error correction, the native model webcoll seems to perform better in 2 out of 3 data sets, and the next better performer being the czesl_L2 model. One has to note that, the non-native models are not tuned to any particular phenomenon such

Train data

WebColl

CzeSL-L1 CzeSL-L2

Corpus size 111M 383K 370K #Errors as capitalization or keyboard layouts, so there is still some scope for improvements on the non-native error models. While webcoll and czesl_L2 models help each other in the opposite direction, i.e., the performance of native model on the non-native data and vice versa, the czesl_L1 model works better only on the CzeSL-L1 dataset. In other words, since L1 error annotation did not involve complete correction of the test data of CzesL-MAN, they can be used, for instance, the correction of misspellings that do not involve grammar errors. 6

Discussion We manually analyzed a part (the top 3000 tokens) of the output of Korektor for the CzeSL-L2 test data for all the three models. We broadly classify the test data as having form errors (occurring between the L0 and L1 level), grammar (gram) errors (occurring between L1 and L2) and accumulated errors (form+gram, where errors are present at all levels – between L0 and L1, and L1 and L2). The CzeSL-L2 test data can include any of the above types of errors. About 23% of our analyzed data include one of the above errors. More than half of the errors (around 62%) belong to the form errors and about 27% belong to the gram class. The remaining errors are the form+gram errors.

In the case of form errors, both the native (webcoll) and the non-native models (czesl_L1 and czesl_L2) detect errors at the rate of more than 89%. Form errors may or may not be systematic and they are easily detected by all the three models. Most of the error instances in the data can be categorized under either missing/addition of diacritics, or they can occur in combination with other types of errors, for instance, prˇítelkyneˇ was incorrectly written as prˇatelkine. Model webcoll czesl_L1 czesl_L2 Error correction

CzeSL-L1

In the case of gram errors, most of the errors are undetected. Out of 193 gram errors in our analyzed data, the percentage of errors detected by the models are: webcoll (15.5%), czesl_L1 (9.3%) and czesl_L2 (15.0%). Most of the grammar errors involve agreement, dependency and lexical errors. The agreement errors are shown in Table 11. Except for a few pairs such as jedné → jednou (incorrect → correct), meˇ → mé, který → kterˇí, teplí → teplý, most of the error tokens involving agreement errors have not been recognized by any of the three models.7

Dependency errors (e.g. a wrongly assigned morphological case, missing a syntactic governor’s valency requirement) such as rokuGEN → roceLOC ‘year’, kolejACC → kolejiLOC ‘dormitory’, rokuSG → rokyPL ‘year’, restauraciLOC → restauraceNOM ‘restaurant’ have not been recognized by any of the models. The pair miDAT → meˇACC ‘me’ has been successfully recognized by all the three models and the correct suggestion listed in the top: Error label: "gram:dep" Error token: mi Gold token: mě webcoll: <suggestions="mě|mi|ji|mu|i"> czesl_L1: <suggestions="mě|mi|ji|mu|si"> czesl_L2: <suggestions="mě|mi|ji|mu|ho">

For instance, the pair ve → v ‘in’ (vocalized → unvocalized) has been recognized by the webcoll and czesl_L2 models, but not by the czesl_L1 model. When it comes to grammar errors, webcoll and czesl_L2 have better performance than czesl_L1. It was expected, because the czesl_L1 model was not trained on grammar errors.

When the error involved a combination of form and gram errors, all the three models tend to perform better. Most of the form+gram errors were recognized by 7The category glosses should be taken with a grain of salt: many forms can have several interpretations. E.g. oslaviliMASC.ANIM → oslavilaNEUT ‘celebrated’ could also be glossed as oslaviliPL,MASC.ANIM → oslavilaSG,FEM. incorrect usage bavímSG bylSG bylSG Chci1ST chodímSG ChteˇlaFEM dívatINF dobréFEM dobrýMASC druhýNOM hezkéPL jeSG jednouINS jichGEN jsemSG jsmePL jsouPL kterýSG leželiMASC.ANIM malýSG malýchGEN méACC MeˇPERS.PRON miluju1ST mohliMASC.ANIM nemocníPL nichLOC oslaviliMASC.ANIM pracovníNOM pracuji1ST StudovaliMASC.ANIM tepleADV cc teplíPL trˇíGEN tuhleFEM typickéFEM velkéPL correct usage bavímePL bylyPL BylyPL Chce3RD chodímePL ChteˇlMASC dívá3RD dobrˇíMASC.ANIM dobráFEM druhéhoGEN hezkýSG jsouPL jednéLOC jeACC jsmePL jsemSG jeSG kterˇíPL leželyFEM maléPL maléACC míNOM MéPOSS.PRON miluje3RD mohlyFEM nemocnýSG neˇACC oslavilaNEUT pracovnímINS pracuje3RD studovalyFEM tepléADJ teplýSG trˇiACC TenhleMASC typickáNEUT velkýSG category number number number person number gender verb form gender gender case number number case case number number number number gender number case number POS person gender number case gender case person gender POS number case gender gender number gloss enjoy was → were was → were want → wants walk wanted to see → sees good good 2nd, other nice is → are one them am → are are → am are → is which lay small small my me → my love → loves could ill them celebrated work-related work → wants studied warmly → warm warm three this typical big all the three models: webcoll (85%), czesl_L1 (86%) and czesl_L2 (89%). For instance, the error pair *zajímavy → zajímavé ‘interesting’ that was labeled at both L1 and L2 level was successfully recognized by all the models, and the correct suggestions were listed in the top. There were many errors that were successfully recognized, but the correct suggestions did not appear in top-3, such as, *nechcí → nechteˇl ‘didn’t want’, *mym → svým ‘my’, *kamarad → kamaráda ‘friend’, *vzdeˇlany → vzdeˇlaná ‘educated’.

Based on the results in Table 10 and the manual error analysis in this section, we can make the following general observations: • Non-native Czech models can be applied to native test data and obtain even better results than the native Czech model (Table 10). • From the manual analysis of the test outputs of both native and non-native Czech models, the most problematic errors are the grammar errors due to missed agreement or government (valency requirements). Some of the grammar errors involve most commonly occurring Czech forms such as jsme, byl, dobrý, je, druhý. • Both native and non-native error models perform well on spelling-only errors. • The CzeSL-MAN error data include errors that involve joining/splitting of word forms that we did not handle in our experiments. We also skipped word order issues in the non-native errors which are beyond the scope of current spell checker systems. 7

Conclusions and Future Work We have tried to improve both the language model and the error model component of Korektor, a Czech statistical spell checker. Language model improvements involved the employment of more balanced corpora from the Czech National Corpus, namely SYN2005 and SYN2010. We obtained better results for the SYN2005 corpus.

Error model improvements involved creating non-native error models from CzeSL-MAN, a hand-annotated Czech learner corpus, and a series of experiments with native and non-native Czech data sets. The state-of-the-art improvement for the native Czech data set comes from the nonnative Czech models trained on L1 and L2 errors from CzeSL-MAN. Surprisingly, the native Czech model performed better for non-native Czech (L2 data) than the non-native models. This we attribute to the rich source of learner error data, since the texts come from very different texts: Czech students with Romani background, as well as learners with various proficiency levels and first languages. Another potential reason could be the untuned nature of the non-native error models that may require further improvement.

As for future work aimed at further improvements of Korektor, we plan to explore model combinations with native and non-native Czech models. We would also like to extend Korektor to cover new languages so that more analysis results could be obtained. To improve error models further, we would like to investigate how the more complex grammar errors such as those in agreement and form errors such as joining/splitting of word forms can be modeled. Further, we would like to analyze non-native Czech models, so that Korektor can be used to annotate a large Czech learner corpus such as CzeSL-SGT more reliably.

References

[1] Church , K. , Gale , W. : Probability scoring for spelling correction . Statistics and Computing 1 ( 7 ) ( 1991 ), 93 - 103

[2] Cˇ ermák , F., Hlavácˇová , J., Hnátková , M. , Jelínek , T. , Kocek , J. , Koprˇivová, M., Krˇen , M. , Novotná , R. , Petkevicˇ , V., Schmiedtová , V. , Skoumalová , H. , Spoustová , J. , Šulc , M. , Velíšek , Z. : SYN2005: a balanced corpus of written Czech, 2005

[3] Gimenes , P. A. , Roman , N. T. , Carvalho , A. M. B. R. : Spelling error patterns in Brazilian Portuguese . Computational Linguistics 41 ( 1 ) ( 2015 ), 175 - 183

[4] Golding , A. R. , Roth , D.: A window-based approach to context-sensitive spelling correction . Machine Learning 34 ( 1999 ), 107 - 130 10.1023/A: 1007545901558 .

[5] Hana , J. , Rosen , A. , Škodová , S. , Štindlová , B. : Errortagged learner corpus of Czech . In: Proceedings of the Fourth Linguistic Annotation Workshop , Uppsala, Sweden, Association for Computational Linguistics, 2010

[6] Heafield , K. : KenLM: faster and smaller language model queries . In: Proceedings of the EMNLP 2011 Sixth Workshop on Statistical Machine Translation , 187 - 197 , Edinburgh, Scotland, United Kingdom, 2011

[7] Jelínek , T. , Štindlová , B. , Rosen , A. , Hana , J.: Combining manual and automatic annotation of a learner corpus . In: Sojka, P. , Horák , A. , Kopecˇek , I., Pala , K. , (eds.), Text, Speech and Dialogue - Proceedings of the 15th International Conference TSD 2012, number 7499 in Lecture Notes in Computer Science , 127 - 134 , Springer, 2012

[8] Junczys-Dowmunt , M. , Grundkiewicz , R. : The AMU System in the CoNLL-2014 Shared Task: Grammatical error correction by data-intensive and feature-rich statistical machine translation . In: Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task , 25 - 33 , Baltimore, Maryland, Association for Computational Linguistics, 2014

[9] Krˇen , M. , Bartonˇ , T., Cvrcˇek , V., Hnátková , M. , Jelínek , T. , Kocek , J. , Novotná , R. , Petkevicˇ , V., Procházka , P. , Schmiedtová , V. , Skoumalová , H.: SYN2010: a balanced corpus of written Czech, 2010

[10] Mays , E. , Damerau , F. J. , Mercer , R. L. : Context based spelling correction . Information Processing & Management 27 ( 5 ) ( 1991 ), 517 - 522

[11] Richter , M.: An advanced spell checker of Czech . Master's Thesis , Faculty of Mathematics and Physics , Charles University, Prague, 2010

[12] Richter , M. , Stranˇák , P. , Rosen , A. : Korektor - a system for contextual spell-checking and diacritics completion . In: Proceedings of the 24th International Conference on Computational Linguistics (Coling 2012 ), 1019 - 1027 , Mumbai, India, ( 2012 ), Coling 2012 Organizing Committee

[13] Rosen , A. , Hana , J. , Štindlová , B. , Feldman , A. : Evaluating and automating the annotation of a learner corpus. Language Resources and Evaluation - Special Issue: Resources for language learning 48 (1) ( 2014 ), 65 - 92

[14] Rozovskaya , A. , Chang , K. -W., Sammons , M. , Roth , D. , Habash , N. : The Illinois-Columbia System in the CoNLL2014 Shared Task . In: CoNLL Shared Task, 2014

[15] Rozovskaya , A. , Roth , D. : Building a state-of-the-art grammatical error correction system, 2014

[16] Whitelaw , C. , Hutchinson , B. , Chung , G. Y. , Ellis , G. : Using the web for language independent spellchecking and autocorrection . In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing - Volume 2 , EMNLP' 09 , 890 - 899 , Stroudsburg, PA, USA, Association for Computational Linguistics, 2009

[17] Wu , S. -H. , Liu, C. -L. , Lee , L. -H.: Chinese spelling check evaluation at SIGHAN Bake-off 2013 . In: Proceedings of the Seventh SIGHAN Workshop on Chinese Language Processing , 35 - 42 , Nagoya, Japan, Asian Federation of Natural Language Processing , 2013