English. We present an approach to improve the selection of complex words for automatic text simplification, addressing the need of L2 learners to take into account their native language during simplification. In particular, we develop a methodology that automatically identifies 'difficult' terms (i.e. false friends) for L2 learners in order to simplify them. We evaluate not only the quality of the detected false friends but also the impact of this methodology on text simplification compared with a standard frequency-based approach.
Italiano. In questo contributo presentiamo un approccio per selezionare le parole complesse da semplificare in modo automatico, tenendo conto della lingua madre dell’utente. Nello specifico, la nostra metodologia identifica i termini ‘difficili’ (falsi amici) per l’utente per proporne la semplificazione. In questo contesto, viene valutata non soltanto la qualita` dei falsi amici individuati, ma anche l’impatto che questa semplificazione personalizzata ha rispetto ad approcci standard basati sulla frequenza delle parole.
The task of automated text simplification has been
investigated within the NLP community for
several years with a number of different approaches,
from rule-based ones
Along this research line, we present in this
paper an approach to perform automated lexical
simplification for L2 learners, able to adapt to the user
mother tongue. To our knowledge, this is the first
work taking into account this aspect and
presenting a solution that, given an Italian document and
the user’s mother tongue as input, selects only the
words that the user may find difficult given his/her
knowledge of another language. Specifically, we
detect and simplify automatically the terms that
may be misleading for the user because they are
false friends, while we do not simplify those that
have an orthographically and semantically similar
translation in the user native language (so-called
cognates). In multilingual settings, for instance
while teaching, learning or translating a foreign
language, these two phenomena have proven to be
very relevant
We compare our approach to the selection of words to be simplified with a standard frequencybased one, in which only the terms that are not listed in De Mauro’s Dictionary of Basic Italian1 are simplified, regardless of the user native 1https://dizionario.internazionale.it/ language. Our experiments are evaluated on the Italian-French pair, but the approach is generic. 2
Given a document Di to be simplified, and a native language L1 spoken by the user, our approach consists of the following steps: 1. Candidate selection: for each content word2 wi in Di, we automatically generate a list of words W1 L1 which are orthographically similar to wi. In this phase, several orthographical similarity metrics are evaluated.
We keep the 5 most-similar terms to wi. 2. False friend and cognate detection: for each of the 5 most similar words in W1, we classify whether it is a false friend of wi or not. 3. Simplification choice: Based on the output of the previous steps, the system marks wi as difficult to understand for the user if there are corresponding false friends in L1. Otherwise, wi is left in its original form. When a word is marked as difficult, a subsequent simplification module (not included in this work) should try to find an alternative form (such as a synonym, or a description) to make the term more understandable to the user. 2.1
Candidate Selection
A number of similarity metrics have been
presented in the past to identify candidate cognates
and false friends, see for example the evaluation
in Inkpen and Frunza (2005). We choose three of
them, motivated by the fact that we want to have at
least one ngram-based metric (XXDICE) and one
non ngram-based (Jaro/Winkler). To that, we add
a more standard metric, Normalized Edit Distance
(NED). The three metrics are explained below:
XXDICE
2Content words are words that have a meaning such as
names, adjectives, verbs and adverbs. To extract this
information, we use the POS tagger included in the Tint pipeline
3An extended bigram is an ordered letter pair formed by deleting the middle letter from any three letter substring of the word. strings S1 and S2. The formula is:
XX(S1; S2) =
P
2
B 1+(pos(x) pos(y))2
xb(S1) + xb(S2)
where B is the set of pairs of shared extended
bigrams (x; y), x in S1 and y in S2. The
functions pos(x) and xb(S) return the
position of extended bigram x and the number of
extended bigrams in string S respectively.
NED, Normalized Edit Distance
Jaro/Winkler
Each of these three measures has some
disadvantages. For example, we found that
Jaro/Winkler metric boosts the similarity of words
with the same root. On the other hand, applying
NED leads to several pairs of words having the
same similarity score. As a result, two words that
are close according to a metric can be far using
another metric. To overcome this limitation, we
balance the three metrics by computing a weighted
average of the three scores tuned on a training set.
For details, see Section 3.
As for false friend and cognate detection, we rely
on a SVM-based classifier and train it on a single
feature obtained from a multilingual embedding
space
In our experiments, we consider a setting in which French speakers would like to make Italian documents easier for them to read. Nevertheless, the approach can be applied to any language pair, given that it requires minimal adaptation.
In order to tune the best similarity metrics combination and to train the SVM classifier, a linguist has manually created an Italian-French gold standard, containing pairs of words marked as either cognates or false friends. These terms were collected from several lists available on the web. Overall, the Ita-Fr dataset contains a training set of 1,531 pairs (940 cognates and 591 false friends) and a test set of 108 pairs (51 cognates and 57 false friends).
For the candidate selection step, the goal is to
obtain for each term wi in Italian, the 5 French
terms with the highest orthographic similarity.
Therefore, given wi, we compute its similarity
with each term in a French online dictionary4
In Table 1 we report the percentage of times in which the cognate or false friend of wi in the training set would appear among the 5 most-similar terms extracted from the French online dictionary according to the three different scores in isolation: XX for XXDICE, JW for Jaro/Winkler and NED for Normalized Edit Distance. We also report the best configuration of the three metrics with the corresponding weight to maximise the presence of a cognate or false friend among the 5 most similar terms. We observe that, while the three metrics in isolation yield a similar result, combining them effectively increases the presence of cognates and false friends among the top candidates. This confirms that the metrics capture three different types of similarity, and that it is recommended to take them all into account when performing candidate selection: an approach where evey metric contributes to detecting false friend / cognate candidates outperforms the single metrics.
XX 1.0 0.2
JW
1.0
0.4
NED 1.0 0.4
For false friends and cognates detection, we proceed as follows. Given a word wi in Italian, we identify the 5 most similar words in French using the 0.2-0.4-0.4 score introduced before. In case of ties in the 5th positon, we extend the selection to all the candidates sharing the same similarity value.
Each word pair including wi and one of the
5 most similar words is then classified as false
friend or cognate with a SVM using a radial kernel
trained on the 1,531 word pairs in the training set.
For the multilingual embeddings used to compute
0.375 when the two strings are not normalized and 0.125
when they are.
the semantic similarity between the Italian words
and their candidates, we use the vectors from
Bojanowski et al. (2016)6 trained on Wikipedia data
with fastText
We perform two types of evaluation. In the first one, the goal is to assess whether the system can correctly identify false friends and cognates in a text. In the second one, we want to check what is the difference between the terms simplified by a system with our approach compared with a standard frequency-based simplification system.
For the first evaluation, we manually create a set of 108 Italian sentences containing one false friend or cognate for French speakers taken from the test set. On each term, we run our algorithm and we consider a term a false friend according to two strategies: a) if all 5 most similar words in French are classified as false friends, or b) if the majority of them are classified as false friends. Results are reported in Table 2.
false friends (a) false friends (b)
The evaluation shows that the two settings lead to two different outcomes. In general terms, the first strategy is more conservative and favours Precision, while the second boosts Recall and F1.
As for the second evaluation, on the same set of sentences, we run our algorithm again, this time trying to classify any content word as being a false friend for French speakers or not. We evaluate this component as being part of a simplification system that simplifies only false friends, and we compare this choice with a more standard approach, in which only ‘unusual’ or ‘unfrequent’ terms are simplified. This second choice is taken by com6https://github.com/facebookresearch/ fastText/blob/master/pretrained-vectors. md 7http://dizionari.corriere.it/ paring each content word with De Mauro’s Dictionary of Basic Italian and simplifying only those that are not listed among the 7,000 entries of the basic vocabulary.
This evaluation shows that out of 1,035 content words in the test sentences, our simplification approach based on a) would simplify 367 words, and 823 if we adopt the strategy b). Based on De Mauro’s dictionary, instead, 240 terms would be simplified. Furthermore, there would be only 76 terms simplified using both strategy a) and De Mauro’s list, and 154 overlaps for strategy b). This shows that the two approaches are rather complementary and based on different principles. This is evident also looking at the evaluated sentences: while considering frequency lists like De Mauro’s, terms such as accademico and speleologo should be simplified because they are not frequently used in Italian, our approach would not simplify them because they have very similar French translations (acade´mique and spe´le´ologue respectively), and are not classified as false friends by the system. On the other hand, vedere would not be simplified in a standard frequency-based system because it is listed among the 2,000 fundamental words in Italian. However, our approach would identify it as a false friend to be simplified because vider in French (transl. svuotare) is orthographically very similar to vedere but has a completely different meaning. 5
In this work, we have presented an approach supporting personalized simplification in that it enables to adapt the selection of difficult words for lexical simplification to the native language of L2 learners. To our knowledge, this is the first attempt to deal with this kind of adaptation. The approach is relatively easy to apply to new languages provided that they have a similar alphabet, since multilingual embeddings are already available and lists of cognates and false friends, although of limited size, can be easily retrieved online.8
The work will be extended along different
research directions: first, we will evaluate the
approach on other language pairs. Then, we will add
a lexical simplification module selecting only the
words identified as complex by our approach. For
8See for example the Wiktionary entries at
https://en.wiktionary.org/wiki/Category:
False_cognates_and_false_friends
this, we can rely on existing simplification tools
This work has been supported by the European Commission project SIMPATICO (H2020-EURO6-2015, grant number 692819). We would like to thank Francesca Fedrizzi for her help in creating the gold standard.