We discuss several challenges of evaluating information extraction patterns, using the DHBB corpus, a public resource for the Dicionar´io Hisotr´ico-Biogarfic´o Brasileiro. Our goal is to stress both the limitations and the advantages of using a corpus-based approach for the task of identifying political families in Brazilian society. It is often mentioned that in Brazil family ties matter a lot for success in politics [12, 7]. However, this is not easy to measure and therefore confirm. But given the availability of the DHBB corpus, we decided to extract all family relationships there mentioned, and assess whether they concerned family relationships among politicians. This can be considered a kind of distant reading for History [2, 10], and it highlighted the need to be very concrete as to what exactly one is evaluating. We begin by explaining how we annotated family ties in the DHBB corpus, then how we annotated that a particular name was already a biographee in DHBB, and then how the family relations were extracted. Then we discuss how to evaluate the result, and show that there are several ways one can evaluate the resulting concordances. In our understanding, a politician is someone who is invested in his or her position through election, nomination or designation, usually members of the executive and legislative branches4. Positions that serve merely for bureaucratic
functions, such as technical advisers and consultants, whether executive,
legislative, judiciary branches or military, are generally not considered politicians,
although they are involved in government decision-making processes.[
DHBB belongs to the AC/DC family of corpora, a project designed to make
available, and searchable, large corpora on the Web [
Contrary to the family semantic domain, there is information that only makes
sense for DHBB, namely the unification of several distinct names as
corresponding to a particular biographee (Lula, Lula da Silva, Luıs´ Ina´cio Lula da Silva, for
example). In fact, each politician who has an entry in DHBB receives an unique
identifier (stored in the id field), and during creation of the corpus we tried to
unify the several diferent ways of referring to the same person, using manually
constructed rules, as described in [
By looking at ten entries whose holders are known to have many family ties with other politicians, the second author devised a set of patterns, divided in ifve groups (not necessarily mutually exclusive), as detailed in [3, page 119]. 1. Relations between the entry politician and other people biographed in DHBB 2. Relations between the entry politician, using the possessive pronoun, assuming that it refers to the biographee, and named politicians. 3. Relations between the entry politician and another non-named politician 4. Relations between two politicians biographed in DHBB, none of them the biographee (This gave us 35 cases, all of them correct.) 5. General family relations described in DHBB between names
Sentences of each group are listed below, preceded by type: – (1) Paulo Maluf, seu padrinho, (...) – (2) Sua esposa era filha de Jo˜ao Alves de Sousa , militar, tenentecoronel e chefe poıtli´co em Patroınc´ io (MG) . – (3) Seu pai foi eleito deputado constituinte (...) 5 More information on the categories can be found in https://www.linguateca.pt/ Gramateca/Familia.html. – (5) (...) os ex-pessedistas, liderados por Crispim Jaques Bias Fortes, secreatr´io de Obras Pu´blicas, filho do ex-governador Joes´ Francisco Bias Fortes (...) Note that the second example illustrates an indirect relationship between politicians: the biographee is son-in-law of JAS (his wife is daughter of JAS). 5
Since the patterns yielded a large number of results, we obtained a sample per kind of pattern, and manually evaluated those 198 cases. We soon noticed that evaluation could be done according to the following criteria: – did the patterns find valid family relations? (criterion 1) – did the patterns find family relations between politicians? (crit 2) – did the patterns find family relations between politicians which were possible to identify in DHBB? (crit 3)
In addition, one could take a strict evaluation of the patterns (and no text around would be included).
– did the patterns extract family relations between politicians? (crit 4) – did the patterns extract family relations between politicians which were possible to identify in DHBB? (crit 5) (only first names are not enough) See, for example, the following cases: – (...) no entanto derrotada dentro do partido que optou por Gleisi Hofmann, esposa do ministro do Planejamento Paulo Bernardo (GH, wife of the minister) – No mˆes seguinte, foi acusado de envolvimento na morte de Severino Alves de Lacerda, filho do ex-prefeito do muniıpc´ io paraibano de Aguiar . (SAL, son of the ex-mayor) The first case allow us to find in the whole sentence a relationship between two politicians (Gleisi Hofman and Paulo Bernardo), but not in the extracted pattern. (it yelds YES for the three first criteria). The second, although it identifies a relationship between two politicians, does not allow us to identify the second politician, and thus yields YES only for for the first, second and fourth criteria.
In Table 1 we show the results of this fivefold evaluation [
What we would like to stress here is that all these numbers are appropriate, but evaluate diferent things. While the first only looks at the precision of family relations in encyclopedic text, the second and fourth measure politician family links, and the third and fifth measure the capability of extracting links among named politicians. The diference between the 2nd and 3rd vs the 4th and 5th deal with the amount of text to be processed. Obviously, the pattern itself is much easier to employ to get triples of the form A-family link-B, that can for example be depicted in a graph.
While analysing the cases obtained, it became clear that the patterns themselves could be significantly improved if we took into consideration the (main) political positions. While reliably annotating the DHBB with all appropriate political positions is not yet performed, we created a short list, improved the rules that mentioned a noun to become a “political position noun”, and got new results.
We were able to get down from 5,017 cases to 2,641 cases, which is almost a half, as is shown in Table 2. Also, the patterns themselves became more reliable, in the sense of yielding the position of the family member much more frequently.
This is confirmed by a new random set which was again humanly reviewed,
see Table 3, and which can be inspected in [
Even though the exercise reported here may seem to exhaust the evaluation possibilities, several interesting issues remain to be solved, such as: full names vs. first names only, concordances where more than one family relationship was present, and the automatic recovery of the possessive pronoun’s referent. And ifnally, the common presence in DHBB of family members with power in Brazil, although not politicians in our sense.