In this paper we present our attempt to im- Translation of named entities consists of several prove machine translation of named entities by using Wi- subtasks. NEs have to be identified in the source text kipedia. We recognize named entities based on categories and their translations must be proposed. These have to of English Wikipedia articles, extract their potential trans- be appropriately incorporated into the sentence translations from corresponding Czech articles and incorporate lation - the sentence context must match the NE and them into a statistical machine translation system as trans- vice versa. lation options. Our results show a decrease of translation quality in terms of automatic metrics but positive results For the English-Czech language pair, matchfrom human annotators. We conclude that this approach ing NEs to the sentence context consists mainly of can lead to many errors in translation and therefore should inflection of NE words. For example, while “London” always be combined with the standard statistical translation translates to Czech as “Londy´n”, in the context of model and weighted appropriately. a more comlex NE, the name has to be inflected in Czech, such as “London airport” ! “Londy´nsk´e letiˇste” (Londonadj airport). 1 Introduction Matching the sentence context to the named entity is needed when some information, such as the grammatical gender, comes from the NE. For example, Czech verbs in past tense have different forms for each gender - the verb “came” has to be translated as “priˇsel” when the subject is masculine, as “priˇsla” for feminine and as “priˇslo” for neuter subject. This information needs to be taken into account in translation: “Jeffry came.” ! “Jeffry priˇsel.”.
looked problem of today’s machine translation (MT). Particularly, most statistical systems do not handle named entities explicitly, simply relying on the model to pick the correct translation. Since most of NEs are rare in texts, statistical MT systems are incapable of producing reliable translations of them.
Moreover, many NEs are composed of ordinary words, such as the term “Rice University”. In the attempt to output the most likely translation, a statistical system would translate this collocation word by word.
In this paper, we attempt to address this problem by using Wikipedia1 to translate NEs and present them already translated to the MT system.
The set of named entities is unbounded and there are many definitions of named entities. In our project, we work with a vague definition of a named entity being a word or group of words which, when left untranslated, are a valid translation anyway (despite the fact that a “real” translation is usually better if it exists; however, it does not exist in many cases). 1.2
Named entity recognition is done in two steps. First, all potential NEs are recognized using a simple recognizer with a low precision but with a high recall. Then, confirmation/rejection of named entities is done — if there is an article with the corresponding title in English Wikipedia, we try to confirm the potential NE as a true NE based on the categories of the article.
The translation of a NE is done by looking up the Czech version of the English Wikipedia article about the named entity. Its title is considered the “base translation”. Other potential translations (in our case this means simply various inflected forms) are then extracted from the text of the Czech article. Each named entity found in the input text is then replaced with a set of its potential translations, from which the MT system then tries to choose the best one.
The matching of the sentence context to the NE is not handled explicitly. We rely on target-side language model to determine the most appropriate option.
find as many potential NEs as possible (i.e. we favour higher recall at the expense of precision), because the candidates for NEs are still to be confirmed or rejected in the next step. Thanks to the external world knowledge provided by Wikipedia, our task is not a typical NER scenario. NE recognition is not the focal point of our experiment, so we limit ourselves to using two tools for recognition of potential NEs: our simple named entity recognizer and Stanford named entity recognizer. 2.1
We created a simple rule-based named entity recognizer for selecting phrases suspected to be named entities. It looks for capitalized words and uses a small set of simple rules for beginnings of sentences — most notably, the first word of a sentence is a potential NE if the following word is capitalized (except for words on a stoplist, such as “A”, “From”, “To”. . . ). Sequences of potential NEs are always considered as a single one multiword potential NE. 2.2
The Stanford NER [
To measure the precision of a NE recognizer, we count the NEs on which the tool agrees with the standard annotation and divide it by the total number of NEs recognized by the tool. Similarly, the recall is measured as the number of NEs confirmed by the standard divided by the number of NEs in the standard.
The performance of the two aforementioned tools measured on the evaluation text is shown in Table 1.
Recognizer Simple NER Stanford NER
Precision Recall F-measure 0.57 0.73 0.64 0.70 0.49 0.58
Our Simple NER has a significantly higher recall than Stanford NER; it is actually capable of delivering most of the named entities. Its low precision is not an issue for our experiment since in the next step we confirm the named entities by using Wikipedia categories. Its F-measure is also higher than that of Stanford NER, suggesting the Simple NER suits our NE definition better.
Since the Stanford NER results are well documented, we assume that its poor results in our experiment are mainly caused by a different NE definition and the recognition model used — in this setup Stanford NER recognizes only people, locations and organizations, but e.g. named entities from the software class (names of programs, programming language functions etc.) are left out from the recognition.
On the other hand, with Stanford NER we are capable of correctly recognizing complex named entities, and the recall of recognition of named entities at sentence beginnings is higher than that of Simple NER.
ing of 255 sentences rich in named entities, originally
collected for a quiz-based evaluation task [
We first performed a human annotation of NEs in the evaluation text, where two annotators marked NEs in the text according to our NE definition. The interannotator agreement F-measure3 was only 83%, which sets an upper bound on the value for our automatic recognizers. We then picked one annotation as a standard, according to which we compare outputs of the NE recognition tools.
<?xml version="1.0"?> <api> <query> <pages> <page pageid="25813" ns="0" title="Rice University"> <categories> <cl ns="14" title="Category:Association of American Universities" /> <cl ns="14" title="Category:Educational institutions established in 1891" /> . . .
The following categories are considered to indicate additional translation options for the decoder. This
NEs: can be generally done in several ways, such as by
extending the parallel data, by adding new entries into
– Places the translation model (i.e. the phrase table), or by
pre– People processing the input data.
– Organizations We use the Moses [
To get the information from Wikipedia we use the lations for sequences of words and even to assign the
Wikimedia API [
When scoring hypotheses, Moses uses several translation model scores, namely p(ejf ), p(f je), 4 Wikipedia translation lex(ejf ) and lex(f je), i.e. translation probabilities in both directions (where f stands for “foreign” For each English Wikipedia article about a NE we (English in this case) and e stands for Czech) and look if there is a corresponding Czech article (this is lexical weights. The value specified in the markup (or 1 provided by Wikipedia under the page section “Lan- if omitted) replaces all of these scores. guages”). If there is one, we use its title as the base Pre-processing of the input data also has the adtranslation. vantage of not requiring to retrain or modify exist
We then try to find all inflected forms of the base ing translation models. Fully trained MT systems can translation in the text of the Czech article to use as therefore be easily extended to take advantage of our alternative translations. method.
For each word in the base translation, we trim its Moses can treat the translation suggestions as eilast three letters, keeping at least the first three letters ther exclusive or inclusive. If set to exclusive, only opintact. This is considered a “stem”. tions suggested in the input markup are considered
Then, the Czech article is fetched using Wikime- as translation candidates. With the inclusive setting, dia API and wiki markup is stripped. We then search these options are included among the suggestions from the article text for sequences of words with the same the translation model, competing with them for the stems. If we find a match, we consider it an inflected highest score. Depending on the quality of the transform of our base translation and include it in the list lation model and the external translation suggestions, of potential translations. this setting can either improve or hurt translation per
Finally, we estimate the probability of the various formance. forms from their counts of occurrences.
When estimating the probability of our translations, we distribute the whole probability mass among 5 Translation process them. The scores of translation suggestions provided by the translation model are typically much lower.
In order to utilize the retrieved translation sugges- However, target language model usually has a signifitions, we had to find a way of incorporating them as cant impact on hypothesis scoring, so even if the ex
They moved to <name translation="Londyn||Londyna" probs="0.6||0.4">London</name> last year. ternal translation scores are set to unrealistically high Since manual evaluation would benefit from data values, the language model makes the “competition” rich in terms of named entity occurrences, we used with translation model options reasonably fair. the same set of sentences as in NER evaluation. These
The default settings for common language models, sentences cover quite a wide range of topics, so they
such as SRILM or KenLM, as used in Moses, assign seem suitable even for translation evaluation.
zero log-probability (i.e. the probability of 1) to
unknown tokens instead of the intuitive 1. In most
cases, training data of the language model for the tar- 6.2 Tools
get language also include the target language part of We used the common pipeline of popular tools for
the translation model parallel data, so this is not an phrase-based statistical MT, namely the Moses
deissue. However, our translation suggestions often con- coder and toolkit, SRILM language modelling
tain tokens unseen in any data, including some noise tool [
results are summarized in Table 2. The main goal of these experiments was to determine which components of our pipeline are actually important for achieving
We began with a simple scenario, only using the
tiWe conducted a series of translation experiments, eval- tles of the articles for translation (i.e. inflected
occuruating various setups of our method. We also carried rences of the title were not available to the decoder)
out a blind manual evaluation, in which the annota- and forcing Moses to use only our suggestions when
tors compared outputs of two MT setups which used translating a NE in a sentence.
our method and of the baseline MT system. In the very first case, we also kept unknown named
entities in their original form — by an unknown NE we
6.1 Data sources understand an entity for which the corresponding
English Wikipedia article exists and its categories imply
We used CzEng 0.9 [
Regular Translations Excluded Excluded Included Included Included
Baseline
Unknown NEs Preserved Translated Translated Translated Translated
NER Simple Simple Simple Simple Stanford potential translations suggested by our tools (albeit able to avoid some errors in each of the steps that, probabilistically weighted). On the other hand, pro- when combined, resulted in a loss in BLEU score. moting phrase table entries too eagerly would result A detailed analysis of errors is provided in Section 6.5. in undesirable translations in some cases, for example On the other hand, we also achieved several nowhen a named entity is composed of common words. table improvements in translation quality even in the
It is also not surprising that keeping unknown en- CzEng test set, some of which are shown in Figure 3. tities untranslated hurts (automatically estimated) translation performance, as Czech tends to translate most of frequent foreign names, and even NEs which 6.4 Manual evaluation are used in their original form are usually inflected in Czech. NEs that would remain completely unchanged We had four annotators evaluate 255 sentences rich in are quite rare. Sentences with some NEs left untrans- named entities, using QuickJudge4 which randomized lated may be more understandable, even considered the input. In the input sentences there were approxbetter translations in some cases, but BLEU score is imately 400 named entities, but the translations difnecessarily worse. fered only in 78 sentences. QuickJudge automatically
When we allowed translation model entries to com- skips sentences with identical translations, so the anpete with our suggestions, the score improved further notators only saw these 78 sentences. to 25.80. The target language model was apparently Three setups were evaluated: the “Baseline” unable to promote options from the phrase table in spite modified Moses system, and two modifications of that of their low translation model scores compared to our system, “Translate” and “Keep unknown”. The syssuggestions (see Section 5). tem marked as “Translate” corresponds to the best
Our translations could have been inadequate for performing setup, not using Stanford NER. “Keep untwo main reasons in this scenario: known” is the same system, however, unknown NEs are handled differently — if a potential NE is con– Lexically incorrect translation, firmed by Wikipedia, but a Czech translation does not – Wrong surface form (only title translation used). exist, it is kept untranslated in the output. The annotators were presented with the source
Adding a full list of all inflected forms of NEs along English sentence and with three translations coming with their estimated probabilities improved the trans- from the three different setups. Then they assigned lation quality slightly, presumably because the target marks 1, 2 and 3 to them. Ties were allowed and only language model was able to determine which of our relative ranking, i.e. not the absolute values, was consuggestions fitted best into the sentence translation. sidered significant.
We can therefore conclude that our approach to in- Table 3 summarizes the results. The values suggest corporating named entity translations works success- a large number of ties — this is not surprising since fully — the outputs contained some direct translations differences between systems were small, their outputs of article titles, some inflected forms extracted from often differed only in 1 word or inflection of a named the article content and some phrase table entries. entity.
Using Stanford named entity recognizer brought We find it promising that our setups won accordno further gains. The recognizer marked a differ- ing to all annotators. The inter-annotator agreement ent (albeit smaller) set of NEs, but further filtering was however surprisingly low — even though in tobased on Wikipedia article categories and the absence tal, the annotators’ preferences match, the individual of many Czech equivalent articles made the difference sentences that contributed to the results differ greatly negligible. among them. All annotators agreed on a winner in
Finally, all our scenarios scored worse than the only 25% sentences. baseline in terms of BLEU. While we believe that the motivation behind our method is valid, we were not 4 http://ufal.mff.cuni.cz/euromatrix/quickjudge/
Source It was Nova Scotia on Wednesday.
Baseline bylmasc to nova scotia ve stredu.
Our setup to byloneut nove skotskoneut ve stredu.
Source In August, 1860, they returned to the Victoria Falls.
Baseline v srpnu, 1860, se k vyjadren falls.
Our setup v srpnu, 1860, se na viktoriiny vodopady. (NE is left untranslated) (correct NE translation and gender agreement) (\Victoria" is left out, \falls" kept untranslated) (correct translation extracted from Wikipedia)
Confirming our intuition, annotators usually pre- Suffix trimming error Suffix trimming also occaferred to keep unknown entities untranslated. The fact sionally matched words or word sequences completely that all of the annotators speak English certainly con- unrelated to the article name. As an example, the tributed to this result, however we believe that keeping name of the company Nestl´e matched the word “neunknown NEs in the original form is often the best so- spra´vne” (“incorrectly”) in the Czech article. Because lution, especially in terms of preserved information. this word is quite common, the language model score Imagine a translation of a guidebook, for example — ensured it to appear in the final translation. A simiif an MT system correctly detects NEs and keeps un- lar example was matching “pole” (“field”) in the arknown ones untranslated, the result is probably better ticle about Poland (“Polsko” in Czech). We decided than if it attempts to translate them. Thanks to the to match case-insensitively in order to cover cases of NER enhanced by Wikipedia, our system would pro- named entities that do not begin with a capital letter duce more informative translations than a standard in Czech (such as “Gulf War”, “v´alka v Z´alivu”). SMT system, which tends to translate NEs in various undecipherable ways.
Wrong named entity form There are two possible causes for an error of this kind — either the Czech article did not contain the inflected form needed in the translation, or the language model failed to enforce the correct option, mainly because the NE contained words unknown to the model (never seen in the monolingual training data).
Since BLEU does not differentiate between a wrong word suffix and a completely incorrect word translation, these errors are equally severe in terms of automatic evaluation.6 On the other hand, human annotators consider a mis-inflected (otherwise correct) translation to be better than a completely untranslated named entity. 7
5 It is however noteworthy that the in ected form of this particular name was always chosen correctly.
Incorrect Wikipedia translation Quite often, the into MERT, we also used a contrastive setup with an
Wikipedia article contained information about a dif- alternative phrase table instead of the XML markup of
ferent meaning of the term. When translated to Czech, input sentences. The decoder was then working with
the difference in the meaning became apparent. For two translation tables — the standard one, generated
example, the default Wikipedia article on “Brussels” by GIZA++ from the parallel corpus, and the new
discusses the whole “Brussels Region”, therefore the one, created by our tools. As is shown in Figure 4,
Czech translation is “Bruselsky´ region”. This word
appeared several times in the test data and the default
interpretation was wrong in all cases.5
6 Metrics with paraphrasing (e.g. Meteor [
Regular Translations Included Included
Unknown NEs Translated
Translated there are two scores in our table — the first one is the system should be used for all named entities, or only probability assigned by our tools (based on number for entities not present (or very rare) in the training of occurrences of the form in the text of the Czech data.
Wikipedia article) and the second one is the “penalty” We described two methods of mixing the newly for using our NE translation.7 It is up to MERT to proposed translations and the default translations of estimate the weight to assign to our translations. the MT system. We studied the XML-input method more and learned that it faces an imbalance in scoring of hypotheses from the two sources. We also report LLoonnddoonn |||||| LLoonnddyynna |||||| 00..42 22..771188 pdreecloimdiinngarpyatrhess,ualtllsowofintghteheotmheordeml ettohcohdo:oaseltethrneabteivset Fig. 4. Example of phrase table entries. balance automatically. While the automatic scores for the second method increased slightly, the results are not yet stable and a further analysis is needed. 7.1 Results In sum, we have shown that Wikipedia can serve as a valuable source of bilingual information and there is an open space for incorporating this information into machine translation. However, Wikipedia should not serve as the only source of information, and the extracted information should be confirmed e.g. by analysis of some other monolingual data.
Although the results of this experiment look promising, they have not been fully evaluated yet and are therefore only preliminary. There is an improvement in BLEU score (see Table 4), but it is not a result of better NE translation. The unstability of MERT process results in different weights in both translations, causing the baseline translation and our experiment outputs to differ significantly in whole sentences, not only in NE translation. Futher analysis and experiments are therefore needed.
There are two results reported in Table 4 because two different versions of the inflector were used to get the inflected forms. The “old” one uses all text data from the body of the article (including e.g. external links), while the “new” one looks for the inflected form only in the text of the article. 8
of named entities based on Wikipedia texts leads to drop in automatic evaluation but to a slight improvement in manual evaluation of MT quality. Part of this improvement is due to not translating identified entities at all.
While some deficiencies of the proposed method of NE translation can be hopefully mitigated (poor suffix trimming and search for various forms of targetside NEs), the incorrectness of some Wikipedia translations is not easy to solve. It is therefore questionable whether the named entity translations provided by our 7 This penalty is used in all Moses phrase tables; it is the : same for all entries and equals 2:718 = exp(1) = e.