Cross-lingual information extraction, the task of extracting information from multiple-multilingual sources, can benefit from the availability of a corpus of equivalent documents in various languages. We present a dataset of pairs of summaries in Spanish and English in various application domains and demonstrate its use in information extraction experiments. The dataset has been manually annotated with semantic information.
Cross-lingual information extraction, the task of extracting information from multiplemultilingual sources, is a problem which has received considerably less attention than extraction from mono-lingual sources. In this paper, we are concerned with the creation of a dataset for the development and evaluation of cross-lingual information extraction systems. Our corpus is a set of pairs of summaries in Spanish and English in various domains. An example of the dataset is shown below: 17 julio 2006 Isla de Java: un maremoto de magnitud 7,7 Richter de magnitud provoca un ’tsunami’ que caus´o la muerte de 596 personas.
On 17 July at 03:19:25 p.m. local time an earthquake measuring 7.7 on the Richter scale struck offshore immediately south of West Java at a depth of 10 km. The areas affected by the earthquake and resultant tsunami included the districts of Taskimalaya, Ciamis, Sukabumi and Garut in West Java province, Cilacap, Kebumen and Banyumas in Central Java and the Gunung Kidul and Bantul districts in the province of Yogyakarta. No. Deaths 500.
These elements in the dataset are nontranslated equivalent summaries which have been found on the Web. They report on the same event, in this case an earthquake, but because they are not translations of one another, they contain different information, for example the Spanish summary reports 596 people dead while the English summary reports 500 people dead. The English summary is more verbose and contains information about the time of the event and various locations affected by the tremor thus being the two elements complementary. The dataset can be used for training information extraction systems, studying template-totext bilingual generation, and automatic knowledge modelling.
This paper gives an overview of the dataset and initial experiments showing its potential application. The rest of this paper is structured as follows: Section 2 we explain related work and then, in Section 3 we describe the data set created. After that, in Section 4 we illustrate how we have used the corpus and in Section 5 we present our conclusions. 2
There are various multilingual datasets in
the machine translation field such as the
Europarl Multilingual Corpus (Koehn, 2005) or
the United Nations Parallel Corpus
The dataset under development is a comparable corpus of Spanish and English summaries for four different domains: aviation accidents, rail accidents, earthquakes, and terrorist acts; this later subset is still under development. Further domains will be incorporated in the future for researchers interested in evaluating the robustness and adaptation capabilities of different natural language processing techniques. In order to collect the summaries, a keyword search strategy was used to search for documents on the Internet using Google Search. Keywords per domain were defined and used to select a set of Web pages in Spanish, for example the keywords “lista de terremotos” could be used to search for documents in the earthquake domain. The pages returned by the search engine were examined to verify if they actually contained an event summary and in that case a document was created for the summary (it is not unusual to find multiple summaries in a single Web page). The documents were given names indicating the type of event and the date of the event/incident. A set of around 50 summaries per domain in Spanish were collected in this manner. After this, for each event summary originally in Spanish the Internet was searched for an equivalent English summary (not a translation) using keywords in English, this time manually derived from the Spanish summary. For example if an earthquake event mentioned a particular date and intensity, then those elements were used as keywords. Following this procedure we found equivalent English summaries for most of the Spanish ones.
For each domain (event or incident) a set of semantic components (i.e., slots) were identified based on intuition and on the actual data observed in a set of summaries for the domain. The slots/components making
Airline Cause DateOfAccident Destination FlightNumber NumberOfVictims Origin Passenger Place Survivors Tripulation TypeOfAccident TypeOfAircraft Year
Cause DateOfAccident Destination NumberOfVictims Origin Place Survivors TypeOfAccident TypeOfTrain up the templates which model the domain are shown in Table 1.
Corpus examples (pairs of summaries in the two languages) for the three domains are shown in Table 2. In order to manually annotate the summaries with semantic information, we have used the GATE annotation framework (Maynard et al., 2002). To facilitate the annotation process an annotation schema was used so that in the GATE Graphical User Interface the target text span to be annotated can be selected, and annotated with one valid category from the annotation schema. The summaries are annotated by one person, however a second person checks the annotations for any inconsistency. Note that because we are dealing with short texts, the annotation process is easier than that of annotating a full event report.
The number of event components found in the set of summaries is reported in Tables 3, 4 and 5. 4
We have started using the corpus in
monolingual as well as in cross-lingual information
extraction. Information extraction is the
mapping of natural language texts (e.g. news
articles, web pages, e-mails) into predefined
structured representations or templates
(Grishman, 1997) such as those we defined in
Table 1. Various techniques have been used
in the development of information extraction
systems including rule-based approaches
relying on robust partial syntactic analysis
Semantic Schema Airline; Cause; DateOfAccident; Destination; FlightNumber; Origin; Passenger; Place; Survivors; Tripulation; TypeOfAccident; TypeOfAircraft; Victims; Year Cause; DateOfAccident; Destination; Origin; Passenger; Survivors; TrainLine; Tripulation; TypeOfAccident; TypeOfTrain; Victims; Year City; Country; DateOfEarthquake; Depth; Epicentre; Fatalities; Homeless; Injured; Magnitude; OtherPlacesAffected; Province; Region; Survivors; TimeOfEarthquake; TotalVictims
Aviation Accident 2009 30 de junio: el vuelo 626 de Yemenia choc´o en cercan´ıas a Comoras, en el Oc´eano Indico. 2009 June 30 Yemenia Flight 626, an Airbus A310-300 flying from Sana’a, Yemen to Moroni, Comoros, crashes into the Indian Ocean with 153 people aboard; one 12-year-old is found clinging to the wreckage.
Railway Accident 12 enero 1997 8 muertos y 25 heridos en el descarrilamiento del tren r´apido Mil´an-Roma en las proximidades de Piacenza (Italia).
January 12, 1997 A Pendolino train derails just before a train station at Piacenza, Italy, killing 8 people and injuring 29 others.
Earthquake 27 mayo 2006 Isla de Java (Indonesia): un terremoto de magnitud 6,2 Richter causa al menos 6.234 muertos, 20.000 heridos y 340.000 desplazados.
May 27, 2006 A powerful earthquake struck Indonesia’s
central province of Java early Saturday morning at 0554 Hrs
local time (26 May 2254 Hrs GMT), flattening buildings and
killing over 4900 people.
multilingual sources can lead to improved
semantic indexing
Experiments Our cross-lingual information extraction experiments involve the use of a system trained in a source language to extract information from translations from another language.
However, to test how useful the dataset is, we
have started with monolingual experiments
per domain and language (e.g., six systems in
total). The systems are a pipeline of text
processing tools followed by a process of token
classification based on Support Vector
Machines (Li et al., 2002). The machine learning
component was adjusted through testing and
evaluation cycles. The text analysis
components are as follows:
• For English: we used default
processors from the GATE system: tokenizer,
parts-of-speech tagger, rule-based
morphological analysis, dictionary lookup,
and named entity recognition and
classification;
Event
Train Accident Spanish
Train Accident English
Aviation Accident Spanish
Aviation Accident English
Earthquake Spanish
Earthquake English
• For Spanish: we used the TreeTagger
software
Basic linguistic features were used to train Spanish and English extraction systems. Both the Spanish and English systems use for each token to be classified a context window of five positions containing the following token features: orthography (e.g., word capitalization), word root, partsof-speech, named entity type, and dictionary (gazetteer lookup) information.
Because each dataset is relatively small, we have performed 10-fold cross-validation experiments reporting here aggregated precision, recall, and f-score figures. Table 6 presents the results. The English extraction system performs better than the Spanish system in the train and aviation accident domains, while the Spanish system performs better than the English one in the earthquake domain. This could be due to the fewer human annotations in the English earthquakes compared to the Spanish counterpart. It is worth noting that the English summaries are more verbose in this domain making extraction more difficult. Although the obtained results are modest, they have to be assessed taken into account the limited syntactic and semantic information available from the text processors. In order to test how the systems cope with noisy data we have translated the Spanish summaries into English and the English summaries into Spanish using Google Translator and have applied the information extraction systems to each translation. In these experiments, for each translation T in a domain D, the extraction system is trained with all documents except the document which is equivalent to T and the resulting system is applied to summary T.
Evaluation metrics are also computed and
aggregated over all documents. In these
experiments we have obtained in most
domains and languages f-scores over 0.60 which
although not directly comparable with the
mono-lingual results are certainly
encouraging, full details on these experiments can be
found in
In this paper we have presented an overview of a dataset with potential interest for crosslingual natural language processing applications. To the best of our knowledge this is one of the few datasets in this field for the pair Spanish/English. We have shown information extraction and cross-lingual extraction as potential applications of the dataset. Our current work involves the expansion of the dataset to cover additional domains such as terrorism and sports. In future work we will address automatic domain modelling from summaries and information extraction induction. We also plan to use the cross-lingual extraction results to improve mono-lingual mono-document extraction. Nation Documents. En Nicoletta Calzolari (Conference Chair) Khalid Choukri Bente Maegaard Joseph Mariani Jan Odijk Stelios Piperidis Mike Rosner, y Daniel Tapias, editores, Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10), Valletta, Malta, may. European Language Resources Association (ELRA).
Freigtag, D. y A. K. McCallum. 1999.
Information Extraction with HMMs and Shrinkage. En Proceesings of Workshop on Machine Learnig for Information Extraction, p´aginas 31–36.
Grishman, R. 1997. Information extraction: Techniques and challenges. En Maria Teresa Pazienza, editor, Information Extraction: A Multidisciplinary Approach to an Emerging Information Technology, International Summer School (SCIE-97), volumen 1299 de Lecture Notes in Computer Science, p´aginas 10– 27, Frascati, Italy, Jul. Springer Verlag. Hakkani-Tu¨r, D., Heng Ji, y R. Grishman. 2007. Using Information Extraction to Improve Cross-lingual Document Retrieval. En Proceedings of the 1st Intl. Workshop on Multi-source Multi-lingual Information Extraction and Summarization Workshop.
Knight, K. y M. Marcu. 2000. Statisticsbased summarization - step one: Sentence compression. En AAAI/IAAI, p´aginas 703–710, Austin, Texas.
Koehn, Philipp. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. En Conference Proceedings: the tenth Machine Translation Summit, p´aginas 79–86, Phuket, Thailand. AAMT, AAMT.
Leek, T.R. 1997. Information Extraction Using Hidden markov Models. Informe t´ecnico, University of California, San Diego, USA.
Li, Y., H. Zaragoza, R. Herbrich, J. ShaweTaylor, y J. Kandola. 2002. The Perceptron Algorithm with Uneven Margins. En Proceedings of the 9th International Conference on Machine Learning (ICML2002), p´aginas 379–386.