English. In this work, we analyse whether Wikipedia can be used to leverage simplification pairs instead of Simple Wikipedia, which has proved unreliable for assessing automatic simplification systems, and is available only in English. We focus on sentence pairs in which the target sentence is the outcome of a Wikipedia edit marked as 'simplified', and manually annotate simplification phenomena following an existing scheme proposed for previous simplification corpora in Italian. The outcome of this work is the SIMPITIKI corpus, which we make freely available, with pairs of sentences extracted from Wikipedia edits and annotated with simplification types. The resource contains also another corpus with roughly the same number of simplifications, which was manually created by simplifying documents in the administrative domain.
Italiano. In questo lavoro si analizza la possibilita` di utilizzare Wikipedia per selezionare coppie di frasi semplificate. Si propone questa soluzione come un’alternativa a Simple Wikipedia, che si e` dimostrata inattendibile per studiare la semplificazione automatica ed e` disponibile solo in inglese. Ci concentriamo soltanto su coppie di frasi in cui la frase target e` indicata come il frutto di una modifica in Wikipedia, indicata dagli editor come un caso di semplificazione. Tali coppie sono annotate manualmente secondo una classificazione delle tipologie di semplificazione gia` utilizzata in altri studi, e vengono rese liberamente disponibili nel corpus SIMPITIKI. La risorsa include anche un secondo corpus, contenente circa lo stesso numero di semplificazioni, realizzato intervenendo manualmente su alcuni documenti nel dominio amministrativo.
In recent years, the shift of interest from
rulebased to data-driven automated simplification has
led to new research related to the creation of
simplification corpora. These are parallel
monolingual corpora, possibly aligned at sentence level,
in which source and target are an original and a
simplified version of the same sentence. This kind
of corpora is needed both for training automatic
simplification systems and for their evaluation.
For English, several approaches have been
evaluated based on the Parallel Wikipedia
Simplification corpus
Our methodology can be summarised as follows: we first select the edited sentence pairs which were commented as ‘simplified’ in Wikipedia edits, filtering out some specific simplification types (Section 3). Then, we manually check the extracted pairs and, in case of simplification, we annotate the types in compliance with the existing annotation scheme for Italian (Section 4). Finally, we analyse the annotated pairs and compare their characteristics with the other corpora available for Italian (Section 5). 2
Given the increasing relevance of large corpora
with parallel simplification pairs, several efforts
have been devoted to develop them. The most
widely used corpus of this kind is the
Parallel Wikipedia Simplification corpus
The extraction of the pairs has been performed using the dump for the Italian Wikipedia available on a dedicated website.1 This huge XML file (more than 1 TB uncompressed) contains the history of every operation of editing in every page in Wikipedia since it has been published for the first time. In particular, the Italian edition of Wikipedia contains 1.3M pages and is maintained by around 2.500 active editors, who made more than 60M edits in 15 years of activity. The Italian language is spoken by 70M people, therefore there are on average 35 active editors per million speakers, giving to the Italian Wikipedia the highest ratio among the 25 most spoken languages around the world.
We parse the 60M edits using a tool in Java developed internally and freely available on the SIMPITIKI website.2 The user who edits a Wikipedia page can insert a text giving information on why he or she has modified a particular part of the article. This action is not mandatory, but it is included most of the times. We first select the edits which description includes word such as “semplificato” (simplified), “semplice” (simple), “semplificazione” (simplification), and similar. Then, the obtained set is further filtered by removing edits marked with technical tags such as “Template”, “Protected page”, “New page”. This eliminates, for instance, simplifications involving the page template and not the textual content. The text in the Wikipedia pages is written using the Wiki Markup Language, therefore it needs to be cleaned. We use the Bliki engine3 for this task. Finally, the obtained list of cleaned text passages is parsed using the Diff Match and Patch library,4 identifies the parts of each article where the text was modified. With this process, we obtain a list of 4,356 sentence pairs, where the differences between source and target sentence are marked with deletion and insertion tags (see Figure 1). 4
We manually annotate pairs of sentences through a web interface developed for the purpose and freely available for download.2 Differently from 1https://dumps.wikimedia.org/ 2https://github.com/dhfbk/simpitiki 3http://bit.ly/bliki 4http://bit.ly/diffmatchpatch corpora specifically created for text simplification, in which modifications are almost always simplifications, annotating Wikipedia edits is challenging because the source sentence may undergo several modifications, being partly simplifications and partly other types of changes. Therefore, the interface includes the possibility to select only the text segments in the source and in the target sentence that correspond to simplification pairs, and assign a label only to these specific segments. It also gives the possibility to skip the pair if it does not contain any simplification.
A screenshot of the annotation tool is displayed in Figure 1. On the left, the source sentence(s) are reported, with the modified parts marked in red (as given by the Diff Match and Patch library). On the right, the target sentence(s) were displayed, with segments marked in green to show which parts were introduced during editing. A tickbox next to each red/green segment could be selected to align the source and target segments that correspond to a modification. The annotation interface provides the possibility to choose one of the simplification types proposed in a dropdown menu (‘Conferma’), or to skip the pair (‘Vai Avanti’). The second option was given to mark the sentences where a modification did not correspond to a proper simplification. For example the last edit shown in Fig. 1 reports in the original version ‘Contando esclusivamente sulla capacita` del mare’, which was modified into ‘Contando soprattutto sulla capacita` del mare’. Since this change affects the meaning of the sentence, turning exclusively into mainly, but not its readability, the pair was not annotated.
In order to develop a corpus which is compliant with the annotation scheme already used in
Split Merge Reordering
Subject Other Verb Subject Other Lexical substitution (word) Lexical substitution (phrase) Anaphoric replacement Verb to Noun (nominalization) Noun to Verb Verbal voice
Verbal features
previous works on simplification, we followed the
simplification types described in
So far, annotators viewed 2,671 sentence pairs, 2,326 of which were skipped because the target sentence was not a simplified version of the source one. 345 sentence pairs with 575 annotations are currently part of the SIMPITIKI corpus, and all
Subject
Other
Verb
Subject
Other
Lexical Substitution (word level)
Lexical Substitution (phrase level)
Anaphoric replacement
Noun to Verb
Verb to Noun (nominalization)
Verbal Voice
Verbal Features
phenomena presented in the annotation scheme
proposed by
As a comparison, we analyse also the content
of the annotated corpora described in
In Fig. 2 we compare the distribution of the
different simplification types across the four corpora.
The graph shows that the same phenomena such as
subject deletion, nominalizations, transfer of
verbal voice tend to be rare across the four datasets.
Similarly, the three top-frequent simplification
types, i.e. delete-other, word transformation and
phrase transformation, are the same across the four
datasets. However, in the Wikipedia-based corpus,
word transformation is less frequent than in the
other document types, while phrase
transformation is much more present. This may show that the
‘controlled’ setting, in which the Terence and the
Teacher corpora were created, may lead educators
to put more emphasis on word-based
transformations to teach synonyms, while in a more
‘ecological’ setting like Wikipedia the performed
simplifications are not guided or constrained, and
phrasebased transformations may sound more natural.
As for the PA documents, transformation
phenomena are probably very frequent because of the
technical language characterised by domain-specific
words, which tend to be replaced by more
common ones during manual simplification. In this
corpus, noun-to-verb transformations are
particularly frequent, since nominalizations are
typical phenomena of the administrative language
affecting its readability
While the Terence corpus contains on average 2.1 annotated phenomena per sentence pair, Teacher 2.8 and the PA corpus 2.9 , the Wikipediabased corpus includes only 1.6 simplifications for each parallel pair. As expected, corpora that were explicitly created for simplification tend to have a higher concentration of simplification phenomena than corpora developed in less controlled settings.
As for non simplifications discarded during the Terence
Teacher
Wiki
Split
r the Dtehleerte-Verb
Delete-Subject
Delete-O
Transf-W
Tranosrfd-Phrase
Anaph-Replace
Noun-To-Verb
Verb-To-N
Traonusfn-VVoice
Transf-VFeature creation of the Wikipedia-based corpus, they include generalizations, specifications, entailments, deletions, edits changing the meaning, error corrections, capitalizations, etc. (see some examples in Table 3). These types of modifications are very important because they may represent negative examples for training machine learning systems that recognize simplification pairs. 6
We presented a study aimed at the extraction and annotation of a corpus for Italian text simplification based on Wikipedia. The work has highlighted the challenges and the advantages related to the use of Wikipedia edits. Our goal is to propose this resource as a testbed for the evaluation of Italian simplification systems, as an alternative to other existing corpora created in a more ‘controlled’ setting. The corpus is made available to the research community together with the tools used to create it. The SIMPITIKI resource contains also a second corpus, of comparable size, which was created by manually simplifying a set of documents in the administrative domain. This allows cross-domain comparisons of simplification phenomena.
In the future, this work can be extended in several directions. We plan to use the simplification pairs in this corpus to train a classifier with the goal of distinguishing between simplified and notsimplified pairs. This could extend the gold standard with a larger set of “silver” data by labelling all the remaining candidate pairs extracted from Wikipedia. Besides, the SIMPITIKI methodology is currently being used to create a similar corpus for Spanish, using the same annotation interface. The outcome of this effort will allow multilingual studies on simplification.
Finally, we plan to evaluate the Ernesta system
for Italian simplification
The research leading to this paper was partially supported by the EU Horizon 2020 Programme via the SIMPATICO Project (H2020-EURO-6-2015, n. 692819).