In this paper, we describe a tool for debugging the output and attention weights of neural machine translation (NMT) systems and for improved estimations of confidence about the output based on the attention. The purpose of the tool is to help researchers and developers find weak and faulty example translations that their NMT systems produce without the need for reference translations. Our tool also includes an option to directly compare translation outputs from two different NMT engines or experiments. In addition, we present a demo website of our tool with examples of good and bad translations: http://attention.lielakeda.lv.
As one of the primary use-cases for the modern computer - automated translation
of texts from one language into another or machine translation (MT) has evolved
vastly since its early days in the 1950s. There have been several large paradigm
shifts that have greatly impacted the field of MT - rule-based MT (RBMT),
statistical MT (SMT) and neural network MT (NMT) [
In this paper, we propose a tool for browsing, inspecting and comparing translations specifically designed for NMT output. The tool uses the attention weights that correspond to specific token pairs which are generated during the decoding process, by turning them into one of several visual representations that can help humans better understand how the output translations were produced. Aside from just visualizing attention alignments, the tool also uses them to estimate the confidence in translation, which allows to distinguish acceptable outputs from completely unreliable ones. For this no reference translations are required.
The structure of this paper is as follows: Section 1.1 summarizes related work on tools for inspecting translation outputs and alignments; Section 2 introduces some concepts of the baseline tool - how it scores translations and displays the visualizations in different environments, as well as outlines the improvements made to make it more useful for debugging machine translation output. In section 3 we give an overview of how to make the most use of our tool in finding odd translations, what to look for when comparing them and possible causes of errors. Finally, we conclude in Section 4 and introduce plans for directions of future work and research in the area. 1.1
Related Work
The foundation of our tool is based on the paper of Rikters et al. [
In contrast, both iBLEU [
NMT frameworks like Nematus [
The basis of our visualization tool is described in full detail in the baseline paper
[
repository2 and open-sourced with the MIT License. In the further subsections of the paper, we will outline only core components and focus more on highlighting improvements and differences.
In addition to Nematus, Neural Monkey and Marian3 [
Confidence Scores This section outlines how the confidence scores are calculated and outlines what is how the final score differs from the baseline.
The four main metrics that we use for scoring translations are: – Coverage Deviation Penalty (CDP) penalizes attention deficiency and excessive attention per input token.
CDP = 1
X log 1 + 1 Ls j
X i ji 2! – Absentmindedness Penalties (APout, in) penalize output tokens that pay attention to too many input tokens, or input tokens that produce too many output tokens. (1) (2) (3) APout = APin = 1 X X Ls i 1 X X Ls j j i ji log ji ij log ij – Overlap Penalty (OP) penalizes translations that copy large fractions from source sentences. A stronger penalty is allocated to longer sentences that copy large amounts from the source while shorter ones get more tolerance (e.g., the three-word English sentence “Thanks Barack Obama." can be perfectly translated into “Paldies Barack Obama." although 2/3 of words in the translation are the same in the source).
OP = (0:8 + (Lt 0:01)) (3 ((1
S) 5)) (0:7 + S) tan(S) (4) – Confidence is the sum of the three main metrics – CDP, APin and APout and the similarity penalty, when the similarity between input and output sentences is high (similarity > 0.3) .
SoftAlignments 3 Marian: https://github.com/marian-nmt/marian 4 Sockeye: https://github.com/awslabs/sockeye
Visualizations: https://github.com/M4t1ss/ Source: Kepler measures spin rates of stars in Pleiades cluster Hypothesis: Kepler measures spin rates of stars in Pleiades cluster Reference: Keplers izm¯era zvaigˇzn¸u grieˇsana¯s ¯atrumu Pleja¯des zvaigzna¯ja¯.
In all of the metrics Ls is the length of the source sentence; Lt - length of the target sentence; S - similarity between the source sentence and the translation on the scale of 0 - 1; ji - the attention weight between source token i and translation token j.
Changes have been introduced to the final confidence score by first calculating the similarity ratio between input and output sentences and then adding a further penalty only if the similarity is high enough. The similarity is calculated by finding the longest contiguous matching subsequence.
Since the baseline confidence score considered only the attention alignments when calculating the final value, examples like shown in Figure 1 received particularly high values due to consistent one-to-one attention alignments. The updated score takes care of this problem by penalizing hypothesis sentence that is overly similar to the input source. The web interface is the primary point of interaction with the tool. Aside from browsing visualizations, ordering data sets by confidence scores and exporting visualizations as images, that are all clarified in the baseline paper, we introduce several significant changes to the system. The first one is a technical update on how data is served loading is performed asynchronously in the background and thereby eliminating long wait times to view the proceeding sentences in a large data set. The three major additions are: – the addition of source-translation overlap percentage alongside the four base scores (Section 2.3); – the ability to provide reference translations, if available, to display next to the hypothesis and calculate BLEU scores (Section 2.4); – the ability to directly compare translations and alignments from two different
NMT systems (Section 2.5). 2.3
Overlap As mentioned in Section 2.1, the updated confidence score considers hypotheses translations that are long and have a significant overlap with the source sentence as a worse translations, while tolerating considerable overlap for shorter sentences. In addition to contributing to the final confidence score, the overlap ratio has been added as an individual score for sorting, navigating and comparing sentences from a data set as shown in Figure 2. The system also underlines the longest matching substring between the source and translation in cases where the overlap is high enough (over 10%). An example is shown in Figure 2, where the overlap ratio is 20.19%.
Source: see 0,2 mg/ml kuni 0,8 mg/ml ( 0,9 mg/ml Ku¨prosel ) ning mo˜nedes riikides ei tohi so˜iduki juhtimise ajal veres u¨ldse alkoholi olla.
Hypothesis: на 0,2 mg/ml до 0,8 mg/ml ( 0,9 mg/ml на Кипре ) , и в некоторых странах в крови не может быть алкоголя.
Match: 0,8 mg/ml ( 0,9 mg/ml We believe that simply displaying the reference next to the hypothesis is helpful more often than not. Having provided references also allows to calculate BLEU scores for the translations, providing yet another dimension for sorting (Figure 2). Unlike overlap, the BLEU scores do not influence the overall confidence scores. 2.5
Comparing Translations The final major addition to the tool is the option to directly compare two translations of the same source sentence. To perform the comparison, all source sentences for both input data sets must match, but the target sentences may differ in output token order as well as count. Comparisons may be performed between translations obtained from any two of the five currently supported NMT frameworks (Nematus, Neural Monkey, OpenNMT, Marian and Soceye) or even an arbitrary input file, as long as it’s formatted according to the specification provided in the readme 5.
Figure 3 shows an example comparison of a sentence translated by two different NMT systems. On the top row is the source text and the bottom rows represent output from each individual NMT system color-coded to match the colors of the alignment lines. The second hypothesis (in green) exhibits stronger and more reliable output alignments to the content words while the first shows strong alignments coming from the stop sign. In this example neither hypothesis matches the reference, but since it is only two words long for a source sentence of triple the length, it can hint to an oversimplified translation by the translator (assuming English was the original) and does not mean that both hypotheses are completely wrong. In fact, the second hypothesis is a fairly decent representation of the source sentence.
Figure 4 illustrates another example with strong attention alignments and a high overlap ratio (94.03%) between source and translated sentences from one system compared to a weak, but at least better translation from another system. The final confidence score for the second translation is strongly influenced by the high overlap, even though the sentence is not particularly long. In similar conditions, the confidence score of the second hypothesis calculated by the baseline system would be very close to 100% due to its complete disregard for the actual words of the source and hypothesis sentences. 3
In this section we summarise several tips and tricks that may come in handy when using the tool to look for faulty translations of various kinds. Here we also list common causes associated with the problems. Some peculiarities to pay attention to may include: – Short sentences with a low confidence, CDP, APin or APout
All of the metrics do not necessarily need to be low, but translations that exhibit at least one of them to be under 30% are often worth looking into. 5 Using other input formats - https://github.com/M4t1ss/SoftAlignments# how-to-get-alignment-files-from-nmt-systems Source: the loss was by the team.
Hypothesis 1: zaud¯ejums bija komandas biedrs.
Hypothesis 2: ˇsis zaud¯ejums bija komandai.
Reference: zaud¯e komanda.
As stated before, for short, several words long sentences it may be completely normal to have an overlap of 50% or more, but if it occurs in sentences that are 10 or more words long, it may indicate that the system has only partially translated the source or not translated anything at all. When completely untranslated sentences are found, it is worth checking the training data for any source-target sentence pairs that are equal. Removing them from the training data should help. – Sentences with a low BLEU score, but normal or even high confidence, CDP, APin and APout The BLEU metric has its flaws and one of them is comparing each hypothesis to only one reference, while it is often possible to translate the same sentence in several different ways. In cases when the only low-scoring metric output by the tool is the BLEU score, it is often that the translation is perfectly good, but just different from the reference. Such sentences are often useful examples to show that lower BLEU scores of neural MT systems do not necessarily represent lower quality translations and are cheaper to find than performing full manual human evaluations.
A separate recommendation specifically for comparing two translations is to look at the attention alignment lines and try to find ones with source tokens having strong alignments to different hypothesis tokens, while maintaining relatively similar confidence scores. Such translations are often synonyms. Source: they did so just in time as Hindes emerged.
Hypothesis 1: vin¸i to dar¯ıja tikai toreiz , kad para¯d¯ıj¯as hinduisti.
Hypothesis 2: it did so just in time as Hindes emerged.
Reference: vin¸iem tas izdeva¯s p¯ed¯eja¯ br¯ıd¯ı. In this paper, we described our conversion of a visualization tool into an instrument for debugging output form neural machine translation systems by improving the attention alignment scoring and confidence estimation of the baseline. The tool is intended to help researchers better understand how their systems perform by enabling to quickly locate better and worse translations in a arbitrary test sets. Compared to other similar tools, ours relies on the confidence scores and does not require reference translations to facilitate this easier navigation, but it only benefits with additional features that are enabled when the references are provided. This allows to integrate it, for example, in an NMT system with a web interface, providing users with an explanation for the result of a specific translation.
In a future version of the system we may include other reference-based MT
scoring metrics for more variety of scoring and sorting. Some examples of metrics
may include chrF [
In addition to the reference-based metrics, there still are some
referenceless approaches yet to be utilised. For instance, borrowing ideas from parallel
corpora filtering [
Another ongoing challenge is to find a way of better representing attention
alignments generated by multi-layer neural networks. While in recurrent neural
network NMT systems this is rarely a problem, more modern approaches like
convolution neural networks [
The research has been supported by the European Regional Development Fund within the research project ”Neural Network Modelling for Inflected Natural Languages” No. 1.1.1.1/16/A/215.