Information Extraction (IE) is a technology dedicated to the extraction of structured information from texts. This technique is used to highlight relevant sequences in the original text or to fill pre-defined templates [1]. Below is the example of a story concerning a terrorist attack in Turkey together with the corresponding entry in the database filled by the IE system.
Semantic knowledge acquisition from texts remains a hard task
even for limited domains. This knowledge is crucial in order to
improve natural language applications like information extraction.
Approaches mixing machine learning (ML) and natural language
processing (NLP) obtain good results in a short development time (we
can cite, among others M. E. Califf [
We present here ASIUM which learns cooperatively semantic
knowledge from texts syntactically parsed without previous manual
processing. This knowledge consists in subcategorization frames of
verbs and an ontology of concepts for a specific domain following
the ”domain dependence” defined by G. Grefenstette 4 [
ASIUM is based on an unsupervised conceptual clustering method and provides an ergonomic user-interface5 to help knowledge acquisition process.
In this part, we will show how ASIUM is able to learn good quality knowledge in a reasonable time from parsed text, even if the syntactic parsing of texts is noisy. Our aim is to learn subcategorization frames of verbs and an ontology for a specific domain, from texts. Actually, existing knowledge bases like EUROWORDNET or WORDNET are frequently over-general for applications in specific domains. These ontologies, although very complete, are not suitable for processing texts in technical languages. On one hand they are not purpose directed ontologies, they may store up to seven meanings and syntactic roles for a word, thus increasing the risk of semantic ambiguity. In a specific domain, the vocabulary as well as its possible usage is reduced, which makes ontologies such 2 2.1
ASIUM learns subcategorization frames like <to drop> <object: Explosive> <in: Public_Place> for the verb to drop. Both couples object: Explosive et in: Public Place are subcategories, object is a syntactic role and in is a preposition but Explosive and Public Place are concepts used as restrictions of selection. More usually, ASIUM learns verb frames like: <verb> <prep.|syntactic role: concept*>*
These frames are more general than the ones defined in the LFG 6 formalism because the subcategories are verb arguments (subject, direct object or indirect object) and adjuncts. In our framework, restrictions of selection can be filled by an exhaustive list of nouns (in canonical form) or by one or more concepts defined in an ontology. The ontology represents generality relations between concepts in the form of a directed acyclic graph (DAG). For example, the ontology could define car, train and motorcycle as motorized vehicle, and motorized vehicle as both vehicle and pollutant. Our method learns such an ontology and subcategorization frames in an unsupervised manner7 from texts in natural language. The concepts formed have to be labeled by an expert. 2.3
as WORDNET overly general. On the other hand, WORDNET may lack some specific terminology of the application domain.
Contrary to any approach of increasing or specializing general
ontologies for a specific domain like R. Basili [
Our previous attempts to automatically revise subcategorization frames and a subset of an ontology acquired by a domain expert have failed. Revision of the acquired knowledge with respect to the training texts required deep restructuring of the knowledge that incremental and even cooperative ML revision methods were not able to handle. The main reason was that the expert built the ontology and the subcategorization frames with too many a priori that were not reflected in the texts. This experiment illustrates one of the limitation of manual acquisition by domain experts without linguists.
The first step of the acquisition process is to automatically extract
syntactic frames from texts. We use the syntactic parser SYLEX
developed by P. Constant [
ASIUM only uses head nouns of complements and links with verbs. Adjectives and empty nouns are not used. Our experiments have shown that these informations were enough to learn semantic knowledge even from a noisy syntactic parsing.
The learning method relies on the observation of syntactic
regularities in the context of words [
Basic classes are then successively aggregated by a bottom-up
breadth-first conceptual clustering method to form the concepts of
the ontology level by level with an expert validation and/or labelling
at each level. Thus a given cluster cannot be used in a new
construction before it has been validated. For complexity reasons, the
number of clusters to be aggregated is restricted to two, but this does
not affect the relevance of the learned concept [
For example, starting with these syntactic frames, <to travel> <subject:[father,neighbour,friend]> <by: [car,train]> <to drive> <subject:[friend,colleague]> <object:[car,motorcycle]> 2.2
Experts have to control the link between the new concept and the verb because the only threshold, fixed by the expert, can not measure the over-generalization risk. This validation process is relatively quick due to the ergonomic user-interface. ASIUM provides to the expert the list of newly covered examples in order to estimate the generality of the proposed concept. Moreover the expert can use functionalities provided by ASIUM in order to divide the learned concept into sub-concepts in case of a proposed concept overly general for the target task. : : : Allemagne de l’Est,N+Loc+Country; via transducers are for most part syntactic structures (the set of expressions equivalent to the notion of ”bombing”) integrating some of the semantic classes furnished by the ASIUM system 10.
The homogeneous semantic lists learned by the ASIUM system are introduced in the INTEX vocabulary. At this level, a manual work is necessary to exploit the semantic classes from ASIUM. These classes are refined (merging of scattered classes, deletion of irrelevant elements, addition of new elements, etc.). About ten hours have been dedicated, after the acquisition process, to the refinement of the data furnished by ASIUM. This knowledge is then considered as a resource for INTEX and is exploited either as dictionaries or as transducers, in function of the nature of the information. If it is a general information that is not domain specific, we prefer to use a dictionary which can be reused, otherwise, we use a transducer.
A dictionary is a list of words or phrases, each one being accompanied of a tag and a list of features11. The first names dictionary or the locations dictionary are generic reusable resources. Below is a sample of the location names dictionary12:
Abidjan,N+Loc+City;
Afghanistan,N+Loc+Country;
Allemagne,N+Loc+Country;
Allemagne de l’Ouest,N+Loc+Country;
As for D. Hindle [
Zelle [
The case roles differs from our subcategorization frames because our prepositions or grammatical functions are replaced by semantic roles like agent or patient. Contrary to the ontology learned by ASIUM, selectional restrictions learned by WOLFIE are attribute-value lists.
An unsupervised learning approach like ASIUM delays concepts
labelling after the learning process and so considerably reduces the
time needed by the expert. After ASIUM learning, the semantic roles
can be labelled by assuming a couple verb+prep./function
represents a specific semantic role. E. Riloff in [
The main advantage is that the number of concepts is not limited to five and we learn in parallel subcategorization frames of verbs without more time-consuming validation needed.
These items structured in a list are convenient for the dictionary format and the semantic lists elaborated from ASIUM complete in an accurate manner the coverage of the initial dictionaries from INTEX.
The transducer format is essentially used for more complex or more variable data where linguistic phenomena such as insertion or optionality may interfere.
Here, the figure LastName presents an example of FirstName a transducer <Person> monsieur de </Person> allowing to mMMa.mdeame UknownWord recognize Transducer "Person". person names such as Monsieur Jean Dupont. the transducer recognizes a sequence composed of a trigger word (Monsieur), a first name (Jean) and a proper name (Dupont). But we must keep in mind that most of these elements can be optional (Monsieur Dupont or Jean Dupont are correct sequences) and that Dupont can be a word that is not listed in any dictionary (it will then be considered as an unknown word).
At this level, one can find two types of transducers: some are generic - as the ”Person” one, and some others are domain-specific and can filled with the semantic knowledge acquired by ASIUM.
The next figure is the illustration of a transducer recog- <We<aepxopnl>osion> de <DET> Explosive </Weapon> nizing explosion Transducer "Weapon". de Det N (explosion of Det N), where the nominal phrase Det N recognizes nominal phases elaborated from the semantic class bombing where the following words appear: bombe (bomb), obus (shell), grenade, etc.
The elaboration of such transducers requires some linguistic expertise to obtain in fine a system recognizing the relevant sequences without too much noise. The architecture of the system is using cascading transducers, it is then important that each level has a good quality in order to allow the following analysis level to operate on a solid background. 3
The Information Extraction system is based on the INTEX tool-box, developed by the LADL laboratory9. INTEX allows a rapid and interactive development of automata and transducers to analyze texts. A linguistic automaton recognizes expressions in texts, whereas a transducer associate specific tags with words in the texts (for example, assign a syntactic category to a word). Transducers are efficient, expressive and sufficient for a local analysis of texts. We chose this approach because it allows the rapid development of an IE system for a given domain with a strictly local analysis limited to the sentence area. Our aim is to develop a highly portable system even if this means using more precise analysis strategies afterwards. To elaborate linguistic resources, we first used the semantic classes defined by the ASIUM system. Before the experiment, the corpus was separated in two different parts : the training set and the test set. The linguistic resources are constantly tested on the training set during the development. This development approach allows to evaluate the performances and to detect possible errors in the grammar (a grammar with too much or not enough constraints which would bring silence or noise during the analysis). The expressions modeled
IE is a now widely spread research domain. The American Message
Understanding Conferences (MUC) provided a formidable
framework for the development of research in this area ([
At least two French-speaking projects have been developed which
are somewhat comparable with MUC systems. these two systems are
the European project ECRAN and the EXIBUM project from the
University of Montreal (Canada). ECRAN developed a generic and
multilingual system tested on different corpora (movie reviews, stories
from the economic area, etc.) [
Several other Information Extraction systems were developed for
specific kind of information (dates, location names, etc.). For
example, D. Maurel [
Recent American work in the area proposed an approach mixing
corpus exploration and knowledge acquisition to feed IE systems. A
first well-known experiment is the AutoSlog from E. Riloff [
In our experiment, we have used a corpus of texts form the French journal ”Le Monde”. Texts indexed by the noun ”terrorist event” have been extracted and manually filtered in order to be sure that they really contain a terrorist event description15. This corpus is of the same kind as the one used for experiments in the ECRAN project, so that we will be able to compare our results.
The time spent on the definition of the linguistic resources with INTEX is estimated to about 15 hours. This duration has to be compared with the two weeks16 needed for the manual resources development of the ECRAN project.
Hundred texts have been used as ”training corpus” and fifteen different texts have been used as ”test corpus”. Texts are first parsed with our system, and then some heuristics allow to fill the extraction template:
Due to the structure of articles of Le Monde, the first date is always the date of the article; we assume that the second date is the one of the terrorist event; the two first occurrences of locations found are stored and usually quite well identify the location of the terrorist event; the first occurrence of a number of victims or injured persons is stored. If a text speaks of more than one terrorist event, we assume that only the first one is relevant. We have chosen short texts to prevent us from this problem inherent to long texts;
only the first weapon linked with the terrorism event is stored.
These heuristics are very succinct and we will have to specialize them to perform information extraction on longer or less-specialized texts. We have used these simple heuristics to evaluate our system and compare it with the ECRAN one. With these heuristics, we obtain good results on our corpus, and most of the extraction systems evaluated in the American MUC conferences used this kind of heuristics in order to solve any parsing problems.
Our results have been evaluated by two human experts who did not follow our experiment. Our performance indicators were defined as:
FALSE for all the other cases.
Using these indicators, we can compute two differents values:
PRECISION1 (P1), ratio between OK and FALSE answers, without taking into account the NONE answers.
PRECISION2 (P2), same as P1 but with the NONE answers.
The next table summarizes results for the different elements of the template.
Date of the story Location Date Nb dead persons Nb persons injured Weapon Average
Semantic knowledge acquisition tools like ASIUM are always very difficult to evaluate. Measuring the quality of an ontology or evaluating an ontology regarding another one is not easy and heavily depends on applications. So, we will only present here some indicators to have an idea on the quality of the acquired knowledge.
Concept quality depends of two different elements. The first one is the distance which computes similarity between classes in order to create relevant concepts and perform relevant inductions. As usual in conceptual clustering, the distance is a parameter of the concept quality and of quantity of expert’s work.
This first qualitative element is very hard to estimate. In our application, 16 of the 19 first classes proposed by ASIUM have been accepted by the expert. 447 inductions have been proposed by ASIUM and 73 % of these inductions have been judged relevant by the expert.
The second element which affects the concepts quality is the level of generality for a concept. When ASIUM proposes a new concept, the expert has to decide from the generality of the concept whether it should be split or not. This work is easy for an expert because he has a very good knowledge of the final application.
For example, if ASIUM proposes the ”Organization” concept, the expert has to decide if it is relevant for the task to identify subconcepts like ”Military org.” and ”Politic org.”.
The generality level in the application highly depends on the
subtlety of the template to be filled by the information extraction system.
Our previous experiments on this domain and on the cooking recipes
domain have shown that this work is simple and that expert choices
really depend on the task. (More explanations on the suitability of
concepts for the main task and unsuitability of these concept for
another task are given in [
The results of the ASIUM system allow to speed up the definition of the paradigmatic classes filling states in the INTEX transducers, even if certain classes need to be manually completed. For example, the ASIUM semantic classes allowed to rapidly complete the graph representing the set of weapons or persons who were implicated in terrorist events. ASIUM provided a class in which terms such as, for example: ”bomb”, ”grenade”, ”explosive” or ”car” could appear, considering that a booby-trapped car is a kind of weapon, etc;
The description language provided by INTEX is richer than the one of ECRAN. The time spent to model the linguistic INTEX transducers was longer than the one spent for ECRAN since the constraints and the empty transitions in automata and transducers have to be manually designed so that the noise is kept at a low level 17.
Such an evaluation, in which we deliberately limited the time spent on the development of linguistic resources, shows the importance of having accurate resources adapted to the task. Moreover, the inescapable incompleteness of the developed resources facing new texts shows that this kind of systems have to integrate dynamic acquisition processes to assist the incremental enrichment of resources, as time goes by.
The experiment was intended to show the time needed for the development of a sufficient set of resources, in order to obtain results equivalent to those of the ECRAN project. That is the reason why we emphasize on an evaluation of the amount of time spent on the task rather than on the improvement potential. That is also the reason why we focused on a limited template that only necessitates a surface analysis. This limitation could certainly be solved if we used more accurately the knowledge acquired by ASIUM. Thus, we plan to take into account a deeper linguistic analysis (anaphora resolution, partial information merging, etc.). pour me´decins sans frontie`res (MSF-Belgique) ont e´te´ blesse´s. (Two Italian doctors working for me´decins sans frontie`res (MSF-Belgique) have been injured.)”, where the passive subject have not been correctly parsed by the system;
The silence for the weapon slot is frequently due to incompleteness of semantic dictionaries.
In this section, we will comment some of the results of this experiment. Results obtained prove the interest of coupling a semantic knowledge acquisition tool with the IE system. But those results are not precise enough to decide about the quality of the semantic knowledge acquisition tool. We will examine here some indicators which allow to judge of the quality of the semantic knowledge learned and next we will present some comments on the information extraction.
The results we obtained during this experiment can be satisfactorily compared with those that we obtained on the same corpus with the ECRAN system, that performed 0.89 precision. Moreover, the results we performed with the new system were obtained after a reduced development phase: about 40 hours for the learning phase with ASIUM and about 15 hours to format the knowledge base as INTEX resources. The following comments can be done on this experiment:
Having a good knowledge of the corpus is indubitably an advantage for the system designer. The fact that one of the author had previously done the same task for ECRAN speeded up the development process, given that the search of relevant syntactic structures was facilitated;
All the knowledge learned by ASIUM is not used in this experiment, especially subcategorization frames. We showed that a surface analysis is sufficient when templates to be filled are not more complex than those of ECRAN. The good quality obtained in a very short time proves this idea.
Nevertheless, in order to extract more specific informations from texts (like the name of the organization that performs the terrorist event, the politic membership of victims or attacker nationality), we think that the use of subcategorization frames could be very useful. Writing syntactic rules in order to perform relevant information extraction becomes very hard because of the multiplicity of the syntactic variations used in texts.
Our current work is to create a cooperative acquisition system to learn resources using the subcategorization frames learned by ASIUM. The expert will be able to express rules using complements of verbs independently of the syntax. Active and passive forms will be given the same representation by the system. For example, the two following sentences will be equivalent: L’action terroriste est revendique´e par le Front populaire de libe´ration de la Palestine (FPLP) (The terrorist event was claimed by the FPLP) or le Front populaire de libe´ration de la Palestine (FPLP) revendique l’action terroriste (The FPLP claimed responsibility of the terrorist event.) One example of rules for this kind of sentences can be:
”Attack” Then the subject is the attacker.
This kind of rule allows to differentiate people claiming terrorism events like in Un groupe terroriste libanais revendique l’attentat antise´mite de Buenos-Aires (A lebanese terrorist group claim anti-semite attack in Buenos-Aires) from an organization claiming for a right like in ”les fondamentalistes musulmans revendiquent le droit de vote (Muslim fundamentalists are claiming voting rights).
Semantic rules allow to make fine differences to accurately fill fine-grain slots. The two next rules fill the field ”Missile” or ”Attacker” regarding the concept (Explosive or Person) learned by ASIUM and used as subject of the verb to kill.
If verb = ”to kill” and subject = Person Then the subject is the attacker. If verb = ”to kill” and subject = Explosive Then The subject is the missile used.
We can see that, even if syntactic parsers generate errors and ambiguities, ASIUM can check texts using the ontology and the subcategorization frames previously learned. Then, the information extraction process will process only on consistent sentences with subcategorization frames. This allows to detect some parsing errors.
The system we are thinking of will process two different parsing steps. First, we will use syntax and concepts learned by ASIUM to pre-fill the frame. Second, we will use our ”conceptual rules” to fill more specifically the frame.
We have described in this article an experiment in which we coupled an information extraction system using INTEX with the machine learning system ASIUM. The development time of the linguistic resources of the information extraction system has been reduced by using the semantic knowledge learned by ASIUM. The quality of the results remains the same as in the European ECRAN project.
The aim of this experiment was to validate our approach. We will now explore a better integration of the two systems and examine how to better use the semantic knowledge learned by ASIUM in order to increase the quality of our results.
The research from Thierry Poibeau is partially funded by a Cifre grant between the Laboratoire Central de Recherches of Thomson-CSF and the Laboratoire d’Informatique de l’Universite´ de Paris-Nord.
The authors want acknowledge M. Rodde (Cristal-Gresec) and A. Balvet (Universite´ Paris X) for their contribution during analysis of the results.