This paper describes the participation of the FORMAS research group with the systems ICEIS17, InferPortOIE, and PragmaticOIE in the Iberian Languages Evaluation Forum 2019. Our activities have focused on the \General Open Relation Extraction" task of relation extraction for Portuguese texts. We present our choices on this challenge, as well as the performance of our systems and their results.
Information Extraction (IE) emerged as a research area to identify relevant
patterns in large quantities of textual documents [
R. Glauber et al.
where arg1 and arg2 are nominal phrases in a sentence and rel establishes
a relationship between arg1 and arg2 through a verb phrase. Open IE
systems are useful in web-scale issues such as question answering and document
ltering systems [
We describe our Open IE systems and their results, as well as the choices and problems faced to perform this task. Our systems were implemented based on machine learning, inference, and handcrafted rules to extract facts from Portuguese sentences. We participate with three of our systems: ICEIS17, InferPortOIE, and PragmaicOIE.
This paper is organized as follows: section 2 describes the problem statement; Section 3 presents our methods ICEIS17, InferPortOIE, and PragmaticOIE; Section 4 describes our Setup, and section 5 presents our evaluation. Section 6 presents the results, and we conclude in Section 7. 2
The organization of the IberLEF (Iberian Languages Evaluation Forum) forum proposes a task that involves the automatic extraction of any relation descriptor expressing any semantic relation between a pair of entities or concepts mentioned in Portuguese sentences. In this task, the coordinators consider a relation description as a text chunk that describes the explicit semantic relation, occurring between two entities or noun phrases in a sentence.
The task was divided into two di erent tests. In the rst test, participants extract the relation descriptors between NP pairs from data provided by the coordinators. This data was annotated with NP information, and as a consequence, do not need to employ a NER system by participants. In the second test, the data provided was not annotated with NP information. The goal of the task was to extract and classify the NPs from the test sentences, and then extract the relation descriptors between pairs of the NPs. We submitted our methods to both Test 1 and Test 2 of Task 3. 3
We participate in Task 3 with three systems: 3.1
Our method called ICEIS17 [
InferPortOIE [
Our PragmaticOIE method achieves a rst pragmatic level. Our rst pragmatic
level copes with inferential, contextual, and intentional aspects. The inferential
module in PragmaticOIE has inherited from our previous work [
All systems, ICEIS171, InferPortOIE2 and PragmaticOIE3, employed to perform the Task 3 are available for download on FORMAS website. Our systems generate an output le in comma-separated values (CSV) format. For Test 1, each system extracted the facts contained in the test sentences. Then, each pair of NP contained in the test le is compared with the arguments of the facts extracted by all systems. For the comparison between the arguments of the extracted facts and the NPs of the test le, the following characters were ignored: \ , . ( ) [ ] ? !. Moreover, to avoid minor divergences in the comparison of strings, we removed a set of stopwords4. The text fragment corresponding to the relationship is chosen 1 http://formas.ufba.br/dclaro/tools.html\#sgs\_iceis 2 http://formas.ufba.br/dclaro/tools.html\#inferportoie 3 http://formas.ufba.br/dclaro/tools.html\#pragmaticoie 4 List of stopwords at https://github.com/stopwords-iso/stopwords-pt/blob/ master/stopwords-pt.txt as a result of Test 1 when the pair of arguments in the output le is similar to those NP pair of the test le.
Test 2 follows the free form suggested by the Open IE task. After running all systems through the test sentences, the next step was to convert our output format from CSV to the required format of IberLEF 2019. 5
Two scores were considered for the evaluation of Task 3: a completely correct
relations score and a partially correct relations score [
The extractions of the systems were matched against the relationship in Test 1 golden dataset, and metrics of exact Precision (EP), exact Recall (ER), partial Precision (PP), and partial Recall (PR) were calculated. Exact and partial Fmeasure are identi ed by (EF) and (PF). 5.2
Since Open Relation Extraction recognizes all possible information, and the sentences adopted in Test 2 are the same as Test 1, we did four di erent evaluations to provide a full panorama of the performance of our systems: { Considering only the relationships in Test 2 golden dataset; { Considering the relationships in Test 2 golden dataset and disregarding the relationship in the training dataset; { Considering the relationships in Test 2 golden and Test 1 golden dataset and disregarding the relationship in the training dataset; { Considering the relationships in all three datasets;
All datasets used are available at http://www.inf.pucrs.br/linatural/
wordpress/iberlef-2019/. The details of the performed measures and datasets
are described in [
We organized the results of Task 3, considering the values obtained in both Tests 1 and 2. Table 1 exhibits the results achieved by all systems considering the exact measure in Test 1. Values for all measures are not very expressive. It is noteworthy that ICEIS17 has a slight advantage when compared with the other systems. Both InferPortOIE and PragmaticOIE systems obtained null values for the exact score. R. Glauber et al.
(a) Exact - Evaluation 1 ICEIS17-EPICEIS17-ERICEIS17IN-EFFERPORTINOFIEE-REPPORTINOFIEE-REPRORPTROAIEG-EMFATIPCROAIEG-MEPATICPROAIEG-MERATICOIE-EF (b) Exact - Evaluation 2 0.04 0.03 0.02 0.01 0.00 0.08 0.06 0.04 0.02 0.00
(c) Exact - Evaluation 3
(d) Exact - Evaluation 4
Open Information Extraction" task through Test 1 and Test 2. In particular, the ICEIS17 system presented the best results (especially, when we isolate precision). When the values for Test 2 are analyzed, it becomes more evident.
The approach used in the evaluated systems demonstrated a low performance for the proposed task. While Open IE systems prioritize the identi cation of a high number of facts in sentences, our methods, that utilize shallow analyzers, have been little e cient.
This study was nanced in part by the Coordenaca~o de Aperfeicoamento de Pessoal de Nivel Superior - Brasil (CAPES) - Finance Code 001.
(a) Partial - Evaluation 1 ICEIS17-PPICEIS17-PRICEIS17IN-PFFERPORTINOFIEE-RPPPORTINOFIEE-RPPRORPTROAIEG-PMFATIPCROAIEG-MPPATICPROAIEG-MPRATICOIE-PF (b) Partial - Evaluation 2 0.25 0.20 0.15 0.10 0.05 0.00 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00
(c) Partial - Evaluation 3
(d) Partial - Evaluation 4 Precision Recal F-measure Precision Recal F-measure 0.30 0.25 0.20 0.15