This paper describes the participation of the DependentIE and DptOIE systems in the Iberian Languages Evaluation Forum 2019. Our activities have focused on the \General Open Relation Extraction" task of relation extraction for Portuguese texts. We describe the choices adopted during the challenge, as well as the systems performed and their results.
Extract information from large repositories of texts within di erent domains is a
hard task for humans. While the quantity and diversity of textual content grow
on the Web, the traditional IE tools have low coverage in this scenario [
which included Open IE task [
In this work, we describe our Open IE systems and their results, as well as the choices and problems faced to perform this task. Our systems were based on dependency analysis and handcrafted rules to extract facts from Portuguese sentences. We participate with two of our systems: DependentIE1 and DptOIE2.
This paper is organized as follows: section 2 describes the problem statement; Section 3 presents our methods DependentIE and DptOIE; Section 4 describes our setup and section 5 presents our evaluation. Section 6 presents our results and we conclude in Section 7. 2
The organization of the IberLEF (Iberian Languages Evaluation Forum) forum proposes a task that involves the automatic extraction of any relation descriptor expressing any semantic relation between a pair of entities or concepts mentioned in Portuguese sentences. In this task, the coordinators consider a relation description as a text chunk that describes the explicit semantic relation, occurring between two entities or noun phrases in a sentence.
The task was divided into two di erent tests. The rst one, the participants must extract the relation descriptors between NP pairs from data provided by the coordinators. This data was annotated with NP information, and as a consequence, do not need to employ a NER system by participants. The second one, the data provided was not annotated with NP information. The goal of the task was to extract and classify the NPs from the test sentences, and then extract the relation descriptors between pairs of the NPs. We submitted our methods to both Test 1 and Test 2 of Task 3. 3
We participate in the IberLEF 2019 with two Open IE systems. The rst of
them, the DependentIE is an Open IE system for Portuguese sentences [
The second system DptOIE is an evolution of DependentIE system [
Both systems, DependentIE4 and DptOIE5, used to perform this task are available for download on FORMAS website. Our systems generate an output le in comma-separated values (CSV) format. For Test 1, each system extracted the facts contained in the test sentences. Then, each pair of NP contained in the test le is compared with the arguments of the facts extracted by both systems. For the comparison between the arguments of the extracted facts and the NPs of the test le, the following characters were ignored: \ , . ( ) [ ] ? !. Moreover, to avoid minor divergences in the comparison of strings we removed a set of stopwords6. When identifying a pair of arguments in the output le of systems similar to an NPs pair of the test le, the text fragment corresponding to the relationship is selected as a result of Test 1.
Test 2 follows the free form suggested by the Open IE task. After running both systems for the set of test sentences, the next step is to convert our output format from CSV to the required format of IberLEF 2019. 5
Two scores were considered for the evaluation of Task 3: a completely correct
relations score and a partially correct relations score [
All datasets used are available at http://www.inf.pucrs.br/linatural/
wordpress/iberlef-2019/. The details of the performed measures and datasets
are described in [
We organized the results of Task 3, considering the values obtained in Tests 1 and 2. Table 1 presents the results obtained by both systems performing the exact measures in Test 1. The values for all measures are not very expressive. Still, DptOIE has a slight advantage in comparing both systems. Next, we present the results for the partial measures in Table 2. Although the values obtained with the partial measures are better for both systems, Test 1 proved to be challenging to solve. The values obtained were very low for both systems in any of the experimental setup. The activity of identifying entities that are part of the arguments of a fact extracted by an Open IE system was the cause of part of the errors introduced. The arguments of the facts are NPs that contains other fragments of the sentence. Even when removing stopwords, other lters should be considered.
Another critical aspect in Test 1 is that the attempt to improve the measures, with a partial score, generated a little impact on the outcome. The increase in the values of Precision, Recall, and F-measures was small by the scale of the values presented.
Figures 1 and 2 present the results obtained for Test 2 in the four setups proposed by the coordinators. In this test, we were able to identify the best performance of DptOIE in performing the task. The di erence is more signi cant when comparing the values for the partial measures between both systems. DptOIE presents higher values for precision and Recall, in addition to greater harmony between these measures, which generated higher values of F-measure.
For Test 2, the execution of partial scores generates a signi cant impact on the outcome. There's an improvement in the results for the DptOIE. In addition, the DependentIE precision result gets a large increase. Although this type of evaluation approach generates better results, one aspect should be considered: the absence of some terms in the arguments of the facts extracted by the Open IE systems may indicate invalid facts, and this cannot be discarded from a more fair evaluation. 7
This paper described the participation of the DependentIE and DptOIE systems in IberLEF 2019. Both systems were submitted to the \General Open Information Extraction" task through Test 1 and Test 2. In particular, the DptOIE system presented the best results. When the values for Test 2 are analyzed, it becomes more evident.
In general, the Open IE task presents the trade-o of approaches that prioritize greater coverage. While discovering a more signi cant number of relationships, the precision values obtained by Open IE systems are low. The results obtained in the participation of the DependentIE and DptOIE systems con rm this problem.
This study was nanced in part by the Coordenaca~o de Aperfeicoamento de Pessoal de Nivel Superior - Brasil (CAPES) - Finance Code 001.
(a) Exact - Evaluation 1 DPTOIE-EF
(b) Exact - Evaluation 2 DPTOIE-EF 0.25 0.20 0.15 0.10 0.05 0.00 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00
(c) Exact - Evaluation 3 DPTOIE-EF
(d) Exact - Evaluation 4 DPTOIE-EF Precision Recal F-measure Precision Recal F-measure 0.25 0.20 0.15 0.10 0.05 0.00 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0.30 0.25 0.20 0.15 0.10 0.05 0.00 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00
(a) Partial - Evaluation 1 Precision Recal F-measure
(c) Partial - Evaluation 3
(d) Partial - Evaluation 4 DPTOIE-PF 0.30 0.25