In the process of building a recommender system based on Hindi tweets for a project, we want to determine whether raw Machine Translation (MT) results could be useful. We collected 100K such tweets and experimented on 200 of them as a preliminary step. Less than 50% of the machine-translated tweets were understandable by English speakers, while at least 80% understandability seems to be required for MT to be included as a useful feature in this context. We then post-edited the MT results and observed that understandability reached 70%, while post-editing time was 5 times less than human translation time. We outline a scenario to produce a specialised MT system that would be able to translate (fully automatically) 70% to 80% of the tweets in Hindi into understandable English.
The operational architecture of a Machine
Translation (MT) system is determined by
precise conditions of the use and development of the
system. For instance, the architecture changes
depending on the role of MT system users (say,
authors, professional translators), the language
pairs involved, or when availability of resources
is a primary constraint. When the task is simply
to help people understand an unknown or little
known language, the design of the MT system
is driven by coverage and automaticity rather
than by the output quality, while only the gist
of a translation is to be conveyed
An interesting case is that of multilingual recommender systems relying on information mined from tweets in regional languages. The user of the system, for instance a tourist, might like to have a look at the top five translated tweets having influenced the recommendation (summarized as usual by 0 to 5 ”stars”). A tweet translation system providing an operational quality output could be suficient in such cases.
Keeping in mind the above context, we make
a preliminary study of the understandability of
tweet translations from Hindi to English, before
and after post-editing them. For that, we
randomly selected 200 tweets from the 100K
collected, had them translated by Google Translate
(GT), evaluated their understandability as is by
English (non-Hindi) speakers, and asked a few
Hindi speaking colleagues to post-edit the MT
results (which we call ”pre-translations”) using
the iMAG/SECtra
We then asked the same English (non-Hindi) speakers to evaluate again the proportion of understandable tweets. That rate rised from less than 50% before post-editing to more than 70% after post-editing. In the context of a recommender system and of the scenario sketched above, if more than 20% (or perhaps 30%) of the (translated) tweets are ununderstandable, the usage value of the MT system would be null, because users would simply stop looking at the tweets. On the other hand, if only 1 out of 5 tweets is ununderstandable, they would continue to look at them when they are curious about the reason for a particularly good or bad recommendation, so that the usage quality of the MT system might be judged good enough or useful or only usable. Our real distinction is whether the MT results would be used, even sparingly, or not at all.
While the value of the minimal rate of understandability certainly depends on each person, we could not yet set up an experiment with many tweet users, as we wished. In fact, the above value of about 70% has been obtained by asking only 2 English-only readers.
In the following section, we elaborate on the data collection and preprocessing. Section 3 explains the experimental setup and procedure. Experimental observations and a scenario for building a good enough specialized MT system follow in the last two sections. 2 2.1
There are numerous services presently
available for providing customised social content
data, including tweets
The search API returns a collection of tweets corresponding to the requested query and the specified query filters. As we want to investigate tweet translations from Hindi to English, and make use of the search API under the AuA mode with a query containing the language filter ’lang:hi’. The query allows us to extract Hindi (translation source language) tweets within the rate-limit specified by the API. We used an interactive Python programming environment for data preprocessing and development to collect 100K tweets in Hindi. 2.2
The preprocessing of our data involved formatconversion of the tweet dataset into HTML files as required by the iMAG1 framework. We also had to normalize a subset of characters (in particular, emojis) to avoid potential systemic problems on account of data encoding and decoding.
The extracted tweets are in the JSON2 format that contains the metadata and the textual content of each tweet. We kept only the textual content (’text’ field) and the tweet identifier 1interactive Multilingual Access Gateway 2JavaScript Object Notation (’id_str’ field) of each tweet. We finally converted the messages to a set of HTML files, each containing a table of 100 rows and 3 columns as shown in Figure 1. A third column with ’enum’ ifeld is added programmatically during conversion for enumeration.
In order to verify data robustness and systemic consistency for further experiments, we set up an existing iMAG for a few files. During the process, we identified a problem that manifested in the form of emoji(s) and emoticons which are frequently used in tweet texts. The incorrect handling of the UTF-8 mapping scheme for those Unicode points that code these emojis caused the setup to fail.
Our solution was to normalise, during the
preprocessing step, a range of such special
occurrences. We identified and converted characters
in the following Unicode point ranges
For instance, the character ’\U0001F44C’ is converted to ’%%EMOJI-0001f44c’. An example can be seen in row 4 of Figure 1. 3 3.1
About iMAG/SECTra
iMAG/SECTra is a post-editing framework
which internally employs GT by default (and
any number of available MT servers) and allows
integration of specialised MT systems. The
system provides pre-translations to the post-editor
and allows post-editing in various modes. It also
allows post-editors to grade the quality of
posteditions and record total time for post-editing
(T petotal). In iMAG/SECTra, each segment has
a reliability level 3 and a quality score between
0 and 204
The quality of the PE of a segment is deemed to be good enough if its quality score is higher or equal to 12/20. 3.2
Our experimental procedure has two parts: 1. evaluating the understandability of pretranslations 2. post-editing pre-translations and estimating the output quality in relation with the post-editing times recorded by the posteditors.
First, we randomly selected two Hindi tweet datasets containing 100 tweets each (twTxtSet1 and twTxtSet2) and then we set up an iMAG/SECTra for post-editing the tweets.
In order to determine the proportion of understandable pre-translations (that is, tweets translated by GT), 2 participants speaking English and no Hindi were selected. Each participant was asked to give a score of 1 if a (translated) tweet was found to be understandable and 0 otherwise. The proportion of understandable tweets was recorded as 39% for twTxtSet1 and 45% for twTxtSet2.
3* for dictionary-based translation, ** for MT output, *** for PE by a bilingual contributor, *** for PE by a professional translator, and ***** for PE by a translator ”certified” by the producer of the content.
410: pass, 12: good enough, 14: good, 16: very good, 18: exceptional, 20: perfect. 8-9: not satisfied with something in the PE. 6-7: sure to have produced a bad translation! 4-5: the PE corresponds to a text difering from that of the source segment. That happens when a sentence has been erroneously split into 2 segments and the order of words is diferent in the 2 languages. 2: the source segment was already in the target language.
Speakers of both Hindi and English were selected to post-edit the pre-translations, thereby jotting down their T petotal for each tweet. To be in line with the envisaged scenario, where a tweet reader knowing both languages might conceivably (but rarely) correct a translation (contribute in Google’s words), while no other reader would independently contribute on the same tweet, each tweet was post-edited only once. Monolingual and bilingual participants were then asked to score the post-edited pretranslations for understandability.
Even though post-editing was to be done in a minimal time possible, the post-editor was allowed to quickly label a tweet with a single word which would help human understanding and further elicit the context of certain tweets. This label was meant to be added but without taking much time. For instance, spam or derogatory tweets could be labeled as ”((??spam??))”, and code-mixed tweets could be labeled as ”((??mixing??))”. The label delimiters were pre-decided to separate them from the original tweet text. No set of labels was prepared beforehand. Labels were introduced by the post-editors themselves. The most frequent were {”news”, ”philosophy”, ”politics”, ”sports”, ”joke”, ”humour”, ”sarcasm”, ”quote”} 4 4.1
Table 1 shows the total post-editing times (in mins) for twTxtSet1 and twTxtSet2. We observe and note additional statistics (given in the term definitions). More importantly, we obtain the quality measure which stands at 56.1% for twTxtSet1 and 73.6% for twTxtSet2.
Dataset #logical- #segments #source- T petotal mn pages words TwTxtSet1 17 331 1843 162.6 TwTxtSet2 18 356 1780 93.7
In Table 2, the quality formula used (NII5
lecture notes
We proceed to add a few illustrative examples with descriptions to better visualise pertinent stages of our experimental procedure and observations. 5National Institute of Informatics, Japan 4.1.2
From the observations at hand, and
knowing that specializing a MT system to a
restricted sublanguage can dramatically increase
all quality indicators
The idea is to include the MT cross-lingual access facility in the recommender system almost from the start, but not to make it accessible immediately, in order not to discourage forever tweeters to use it. There will be a phase whose length will depend on the number of bilingual Hindi-English speakers contributing to the building of a specialized hi-en tweet-MT system.
As has already been done successfully for
French-Chinese
Initially, we will populate it using parts of some genuine hi-en corpus, if any, and, if none is available, a part of the CFILT6 en-hi corpus. Even if inverted translations are notoriously not translation examples, an inverted parallel corpus is better than nothing. That will be the basis for building version 0:1 of a Moses-based specialized system, say, twMT-hi-en-0.1.
The contributors team will then post-edit what it can, working some time every day. Incremental improvement will be performed a certain number of times7after each new batch of good enough post-editions will become available, giving twMT-hi-en-0.1 … twMT-hi-en-0.20 if there are 20 incremental improvement steps. Version 1:1 (twMT-hi-en-1.1) will then be produced by full recompilation, and the whole process will be iterated.
6Centre for Indian Language Technology, IITB, India 7Experiments on French-Chinese have shown that improvement levels out after 10-20 incremental improvement steps. It is then necessary to recompile the full system, and that is also an appropriate time to modify the learning set by including all good enough post-editions, say, N pe bisegments, and keeping only Sizetw N pe of the unspecialized parallel corpus.
The PE interface will systematically propose the results of the current twMT-hi-en-x.y version in the PE area of each segment, but results produced by GT and if possible other systems (Systran, Bing, Indian systems) will also be visible, with a button to reinitialize the PE area with each of them. No development is needed, as this is a standard feature of the SECTra interface since 2008. The quality measure used to determine when the specialized system will be good enough to open the MT cross-lingual access facility to tweeters. There will be a first period during which twMT-hi-en-x.y will remain inferior to GT, that is, will require more PE time.8
After a certain version (a:b), the PE time for results of twMT-hi-en-x.y with x:y a:b will be less than that for GT, but results will still not be understandable enough. How to know if and when this will happen?
The experiment described in this paper shows that PE of current MT results allows to almost get to the required understandability level of 70%-80%, with a PE time of 12mn/p. We hope that MT outputs needing only 5mn/p of PE to reach 90% understandability will be understandable enough (70%-80%) without PE.
The idea is that, if that correlation holds, which we will verify by testing it every time a new version (x.1) is issued, we will open the MT cross-lingual access facility to tweeters when this minimal understandability threshold will have been attained through this supervised learning process.
Another worry will then be to ensure
nonregressivity. It is expected that some continuous
human supervision will remain needed, and that
no dedicated contributors group will be
maintainable. Then, some self-organizing community of
contributors (post-editors) should emerge,
somewhat like what has happened for many open
source software localization projects. Another
encouraging perspective is the announcement of
a new kind of web service such as SYNAPS
(
8The PE time for GT outputs will be estimated without any supplementary human work because experiments show a very good correlation between our mixed PE distance ∆m(mt; pe) and T petotal(mt).