Information extraction from unstructured text data has been used essentially to provide new insights by collecting, storing, and analyzing text data in textual analysis. The research on event extraction has been recently getting more attention in information extraction area since lots of events happens and significantly affect our societies and countries. Related studies on event extraction use rules for identifying and extracting events from texts so far. However, rule based approaches have a limitation in terms of accuracy and rule construction. In this paper, we present an event extraction system which takes advantage of the machine learning method, especially SSVM. The system extracts event triggers and predefined event arguments, while existing rule based systems extract unknown event arguments. Ours provides 60.23 F1 score, which is higher than that of previous researches, in which rule based event extractions were performed. Even though rule-based and machine learning-based approaches cannot be compared against each other completely fairly, what is clear is that for the task in which event arguments are defined in advance, applying machine learning method can make better results.
Recently, research on event extraction has been conducted to rapidly discover
meaningful events with regard to business that are found in a massive quantity of
data. In terms of meaningful information existing in text, an event might be viewed as
a type of group that qualifies for information to be extracted. Data obtained from
named entity recognition, relationship extraction, and event extraction can be
employed as fundamental data for more detailed data analysis and intelligent services,
such as natural language questioning and answering (NLQA) or predictive analytics
(PA) which are emerging technologies presented in Gartner Hype Cycle [
Lots of researches have been conducted on named entity recognition and relation
extraction, and the results have reached over 90% of human cognitive capacity [
This study relates to event extraction as source data required to expand the knowledge database used for NLQA or PA. We aim to accurately extract a trigger and arguments of an event by using a machine learning method with applying a structural support vector machine (SSVM) algorithm. Furthermore, temporal information, another event argument, can be extracted more effectively by applying rules from a text or metadata rather than by applying the machine-learning method because of its various forms of expression. We designed and implemented a module for extracting temporal information. 2
An event is real temporal data, whose temporal sequence is important and might be
infinitely long [
A variety of techniques, such as language processing, text mining, data mining, and machine learning, are widely adopted to extract events from a large quantity of text. Extraction methods are largely divided into a machine-learning method and a rulebased method. The machine-learning method sets the structure of an extraction event in advance, constructs a learning group based on the determined structure, and extracts the event. This method is advantageous for extracting events with a predetermined structure. On the other hand, although the rule-based method also defines the structure of an event in advance, the structure of the event can subsequently be changed easily, thus permitting freely reflecting feedback from the extraction result. However, there exists a limitation to the rule-based method because of a significant number of exceptions. Accordingly, the machine-learning method might be more efficient in extracting events with a predetermined structure. 3
This study focuses on business event extraction reported in the literature. A
business event indicates an event arising from corporate activities, and the major concepts
pertaining to this type of event are as follows [
- Event mention: an event type. Common designation of events that have an identical meaning.
- Event trigger: the most exact term for expressing an event mention. - Event argument: an event attribute associated with an event trigger, such as an entity mention, a temporal expression, or value.
- Event instance: an instance comprised of event arguments associated with an event trigger that is referring to an event mention presented in an identical sentence.
According to the above definition, the event in this study includes an event trigger, event subject, event object, and time. The event subject, event object, and time correspond to an event argument. An event is expressed as shown in (1).
Event mention <event trigger, subject (arg.1), object (arg.2), time (arg.3) > (1)
In this study, business events are limited to ‘Announce’ and ‘Launch’. The subjects of two events are corporation names and the objects are product names which are highly related to the companies. 4
In this study, an event trigger and arguments, with the exception of temporal information, are extracted by the machine learning method. Temporal information is extracted by the rule-based method rather than by the machine learning method because such information can be expressed in various ways.
The machine learning algorithm used is the structural support vector machine (SVM) algorithm. The structural SVM algorithm is a machine learning algorithm extended from the conventional SVM algorithm. A structural SVM supports more general structural issues, for example, sequence labeling, parsing, etc., whereas a conventional SVM supports binary classification and multi-class classification, among other classification methods.
In addition, for the learning of the structural SVM algorithm, this study uses the Primal Estimated sub-GrAdient Solver for SVM (PEGASOS) algorithm among the Stochastic Gradient Descent (SGD) method as an extension of the structural SVM algorithm because the PEGASOS algorithm has shown high performance and rapid learning rate when applied to the SVM.
To extract an event by applying the extended structural SVM algorithm, training
data is required. Training data can be directly constructed manually by domain
experts, but because of limited time and labor, it is preferred to make it automatically.
We also built it semi-automatic method [
Later, domain experts are hired to establish the gold standard through manual verification in order to enhance accuracy. The training data was prepared for two event types and two relation types which are just for the system evaluation. The number of training data of each type is presented in table 1.
With the machine learning, an event trigger, an event subject, and an event object are extracted; temporal information is extracted through an additional designed tool. Fig. 1 illustrates the process of extracting the event temporal information. Each module that comprises the tool is described as follows.
The evaluation of the system which is used for event extraction in this study is performed with F1 score for two business relations. The result of the evaluation is presented in table 2. Our system is well performed in information extraction job, especially relation extraction which is similar with event extraction.
The test data is 10% of the entire training data (table 1), and it is randomly
sampled. The result of business event extraction is shown in table 3. Even though the F1
score of each event type is different, they are all over 60.0. The F1 score of our
system exceeds that of other systems [
Precision
Recall Method Rule-based Rule-based ML
Ours 6
This study aims to extract business events by using machine learning based information extraction system. Different with events defined in other researches, arguments of an event is predefined. Thus, our system is only capable of extracting designated arguments in advance. For extracting the temporal information, one of event arguments, we use rules. The result of system evaluation says that our system accomplishes better performance than other rule based system in terms of F1 score. This means that for the event extraction in which the arguments are defined ahead, machine learning based method makes better results.
Acknowledgement. This work was supported by the IT
R&D program cal-business Strategy Analysis and User-targeting Visual Advertisement Materials for Micro-enterprise Managers]