With the globalization trend there is a big amount of documents written in different languages. If these polylingual documents are already organized into existing categories one can deliver a learning model to classify newly arrived polylingual documents. Despite being able to adopt a simple approach by considering the problem as multiple independent monolingual text classification problems, this approach fails to use the opportunity offered by polylingual training documents to improve the effectiveness of the classifier. This paper proposes a method to combine different monolingual classifiers in order to get a new classifier as good as the best monolingual one having also the ability to deliver the best performance measures possible (precision, recall and F1). The proposed methodology was applied to a corpus of legal documents - from the EUR-Lex site - and was evaluated. The obtained results were quite good, indicating that combining different mono-lingual classifiers may be a promising approach to reach the best performance for each category independently of the language.
Current Information Technologies and Web-based services need to manage, select and filter increasing amounts of textual information. Text classification allows users, through navigation on class hierarchies, to browse more easily the texts of their interests. This paradigm is very effective both in filtering information as in the development of online end-user services.
Since the number of documents involved in these applications is large, efficient and automatic approaches are necessary for classification. A Machine Learning approach can be used to automatically build the classifiers. The construction process can be seen as a problem of supervised learning: the algorithm receives a relatively small set of labelled documents and generates the classifier. Several algorithms have been applied, such as decision trees, linear discriminant analysis and logistic regression, the nave Bayes algorithm and Support Vector Machines (SVM). Besides having a justified learning theory describing its mechanics, SVM are known to be computationally efficient, robust and accurate.
Because of the globalization trend, an organization or individual often generates, acquires and archives the same document written in different languages (i.e., polylingual documents); moreover, many countries adopt multiple languages as their official languages. If these polylingual documents are organized into existing categories one would like to use this set of preclassified documents as training documents to build models to classify newly arrived polylingual documents.
For multilingual text classification (i.e., collections of documents written in several languages), some prior studies address the challenge of crosslingual text classification. However, prior research has not paid much attention to using polylingual documents yet. This study is motivated by the importance of providing polylingual text classification support to organizations and individuals in the increasingly globalized and multilingual environment.
We propose a method that combines different monolingual classifiers in order to get a new classifier as good as the best monolingual one which has the ability to deliver all the best performance measures (precision, recall and F1) possible.
This methodology was applied and evaluated on a set of legal documents from the EUR-Lex site. We collected documents for two anglo-saxon languages (English and German) and two roman ones (Italian and Portuguese), obtaining four different sets. The obtained results were quite good, indicating that combining different monolingual classifiers may be a promising approach to the problem of classifying documents written in several languages.
The paper is organized as follows: Section 2 describes the main concepts and tools used in our approach, Section 3 introduces the methodology for combining monolingual classifiers and Section 4 presents the document collection used for evaluation, describes the experimental setup and evaluates the obtained results. Finally, Section 5 presents some conclusions and points out possible future work.
This section introduces the Automatic Text Classification approach and the classification algorithm and software tool used in this work.
Originally, research in Automatic Text Classification addressed the binary problem, where a document is either relevant or not w.r.t. a given category. However, in real-world situations the great variety of different sources and hence categories usually poses a multi-class classification problem, where a document belongs to exactly one category from a predefined set. Even more general is the multi-label problem, where a document can be classified into more than one category.
In order to be fed to the learning algorithm, documents must by
preprocessed to obtain a more structured representation. The most common
approach is to use a bag-of-words representation
Research interest in this field has been growing in the last years.
Several machine learning algorithms were applied, such as decision trees (Tong,
1994), linear discriminant analysis and logistic regression (
While most text classification studies focus on monolingual documents, some
point to multilingual text classification. From these, the great majority
address the challenge of crosslingual text classification where the classification
model relies on monolingual training documents and a translation
mechanism to classify documents written in another language
Support Vector Machines, a learning algorithm introduced by Vapnik and
coworkers
Kernel techniques comprise two parts: a module that performs a mapping from the original data space into a suitable feature space and a learning algorithm designed to discover linear patterns in the (new) feature space. The kernel function, that implicitly performs the mapping, depends on the specific data type and domain knowledge of the particular data source.
The learning algorithm is general purpose and robust. It’s also efficient
since the amount of computational resources required is polynomial with
the size and number of data items, even when the dimension of the
embedding space grows exponentially
The structural risk minimization (SRM) framework creates a model with
a minimized VC (Vapnik-Chervonenkis) dimension. This developed theory
As classification software we used SVMlight
Having documents in several languages, one can adopt a nave approach by considering the problem as multiple independent monolingual text classification problems. This simple approach only employs the training documents 1 Available at http://svmlight.joachims.org of one language to construct a monolingual classifier for that language and ignores all training documents of other languages. When a new document in a specific language arrives, one select the corresponding classifier to predict appropriate category(s) for the target document. However, the independent construction of each monolingual classifier fails to use the opportunity offered by polylingual training documents to improve the effectiveness of the classifier.
With this bearing in mind, and to get a decision for a new document, monolingual classifiers could be improved up in several ways. We propose the following strategies for the combination system: the sum of SVMs output values the F1 weighted sum of SVMs output values the F1 weighted sum of SVMs decisions The above measures could also be used to draw decisions when considering a voting strategy of the monolingual classifiers.
This section introduces the dataset, describes the experimental setup and presents the obtained results for the legal concepts classification task.
For testing the proposed methodology, experiments were run over a set of European Union law documents. These documents were obtained from the EUR-Lex site2 within the “International Agreements” section, belonging to the “External Relations” subject matter. From all available agreements we chose the ones with full text (not just bibliographic notice) obtaining a set of 2714 documents (dated from 1953 to 2008).
Since agreements are available in several languages we collected them for two anglo-saxon languages (English and German) and two roman ones (Italian and Portuguese), obtaining four different corpora: eurlex-EN, eurlex-DE, eurlex-IT and eurlex-PT. Table I presents the total number and average per document of tokens (running words) and types (unique words).
Each document is classified onto several ontologies: the “EUROVOC descriptor”, the “Directory code” and the “Subject matter”. In all available classifications each document can be assigned to several categories. For our classification problem we used the first level of the “Directory code” classification, considering only categories with at least 50 documents. Table II shows each category along with the number of documents assigned.
2 Available at http://eur-lex.europa.eu/en/index.htm
The experiments were done using a bag-of-words representation of documents, the SVM algorithm was run using SVMlight with a linear kernel and other default parameters and the model was evaluated using a 10-fold stratified cross-validation procedure with significance tests done with a 90% confidence level.
To represent each document we used the bag-of-words approach, a vector space model (VSM) representation where each document is represented by the words it contains, with their order and punctuation being ignored. Document’s representation was obtained by mapping all numbers to the same token and using the tf-idf weighting function normalized to unit length.
To measure learner’s performance we analyzed precision, recall and the
F1 measures
To support our claim, as baseline we have built classifiers for each language. Table III shows the average precision, recall and F1 measures for each corpus and each category (boldface values are significantly worse than the best value obtained). Last line presents the average values over all nine classes.
For the precision values we can notice that the Portuguese dataset has values with no significant difference with the “best” for all classes; all other languages perform worse for some classes (English: c2, c4 and c16; German: c12 and c16; Italian: c2, c7 and c12). With this in mind one can say that the Portuguese language generates the best precision classifiers.
Concerning recall, it’s the English and German languages that consistently present the best values; Italian and Portuguese while equally good for some classes, are worse for others (Italian: c2 and c3; Portuguese: c2, c3 and c4).
The F1 measure presents the same behavior as recall, being the only difference the classes where the Portuguese language performs worse (c2, c3 and c16).
From all possible combiners (see Section 3), there is one that, for all classes, persistently generated the best F1 values: the F1 weighted sum of SVMs decisions.
Table IV shows, for each performance measure its results compared with the “best” monolingual classifiers(boldface values are significantly worse than the corresponding multilingual one): the Portuguese one for precision, and the English and German one for recall and F1. Last line equally presents the average values over all classes.
From the average values, one can easily see that precision is higher than recall and that the best monolingual classifier depends on what performance measure one is considering. Nevertheless, the combined classifier has all performance measures very similar and never significatively worse then the best monolingual classifier.
In fact, significant tests show that, for all classes and all performance measures, there is no significant difference between the “best” monolingual classifier and the corresponding combined classifier.
A proposal to combine monolingual classifiers was presented and evaluated. The proposed methodology uses SVM classifiers to associate concepts to legal documents and uses a decision function that combines them in order to obtain, for each class, a classifier as good as the best monolingual classifier of each performance measure.
The baseline experiments allows one to conclude that some languages generate classifiers with better precision values (Portuguese language) while others generate classifiers with better recall ones (English and German languages). In order to be able to explain and to try to generalise these results further experiments need to be done. For instance, we will need to evaluate this methodology with other collections and domains. Are these results specific for the legal domain? Or only for this collection and topics? Nevertheless, from a linguistic point of view, these results raise quite interesting questions.
By combining all classifiers one obtains a classifier as good as the best monolingual one. This combined classifier can even be considered better than the others since it has the ability to deliver all the best performance measures (precision, recall and F1) unlike using one monolingual classifier.
As ongoing research we intend to use a deeper linguistic representation
of documents and to re-evaluate this methodology. Specifically, we will use
a semantic representation (based on DRS3) of documents and a graph kernel
to create SVM models. In previous work, this approach showed to be able
to improve the bag-of-words result for the Portuguese language. Another
research line is to use legal thesaurus, such as the LOIS4 lexical thesaurus,
to reinforce some features/terms. With this approach we would combine our
proposal with the main ideas of the Wei et al. work