The purpose of this research was to automatically extract catchphrases given a set of Legal documents. For this task, our focus was mainly on the Machine learning approaches: a comparative approach was used between the unsupervised and supervised approaches. The idea was to compare the diferent approaches to see which one of the two was comparatively better for automatic catchphrase extraction given a dataset of Legal documents. To perform this, two open source text mining software were used; one for the unsupervised approach while another one was used for the supervised approach. We then fine tuned some parameters for each tool before extracting catchphrases. The training dataset was used when fine tuning parameters in order to find optimal parameters that were then used for generating the final catchphrases. Diferent metrics were used to evaluate the results. We used the most common measures in Information Extraction which include Precision and Recall and the results from the two Machine learning approaches were compared. In general our results showed that the supervised approach performed far much better than the unsupervised approach.
Automatic keyword or catch phrase extraction is an area of
research that seems like it has not been exploited much. Determining
catchphrases manually can be time consuming, expensive and
usually require expertise to perform the work [
Keyword Extraction involves automatically searching and
identifying keywords within a document that best describes the
subject of the document [
As the name suggests, Simple statistical approaches are very
simple, they do not need any training and are language and
domain independent. Keywords can be identified by using statistics
of the word such as word frequency, word co-occurrences, term
frequency-inverse document frequency (TF-IDF), N-gram statistics.
The disadvantage with using this approach is that in some
domains such as health and medical, the most important keyword
may appear only once [
This paper is organized as follows: We first presented related
work for the supervised and unsupervised Machine learning
approaches mainly focusing on Rapid Automatic Keyword
Extraction, RAKE [
According to Lima et al and Rose et al [
According to Medelyan [
A keyword extraction library called RAKE [
For the IRLeD task, the dataset provided contained the following: (1) Train docs - consisted of 100 case statements. (2) Train catches - contained the gold standard catchwords for each of the 100 case statements provided in the Train docs. (3) Test docs - contained 300 test case statements. For each of these 300 statements, a set of catchphrases was generated.
The training dataset was randomly divided into two groups consisting of 90 documents and 10 documents from the dataset. The 90 documents dataset was only used for training the supervised machine learning approach while the remaining 10 documents dataset were used for testing both the unsupervised and supervised methodologies. 3.3
RAKE consisted of the following parameters which were fine tuned for diferent experiments in order to find the optimal parameter values that yielded the best performance on the training set provided. Table 1 provides more details on parameters experimented with as well as performance results.
(1) The number of character can be varied in order to select keywords with a certain number of characters represented as No of Char/word in Table 1. (2) The number of phrases for each keyword can be tuned to varies words represented as No of word/phrase in Table 1 (3) The number of times a keyword appears in a given text can be limited to a certain number represented as keyword frequency in Table 1. 3.4
As it was done in Section 3.3, parameter turning experiments were performed in order to find the optimal parameters for MAUI. The only parameter tuned for MAUI was to vary the frequency of occurrence of each keyword and discard some keywords based on that. The default MAUI parameter discards any candidate phrase(s) that appeared less than two times. See Table 2. 3.5
RAKE was used to generate catchphrases for the Test documents provided with parameters tuned to 3 3 1: meaning each word had atleast 3 characters, each phrase had at most 3 words and each keyword appeared in the text at least once.
UBIRLeD_1 - Catchphrases were generated for each document together with the corresponding scores for each catchphrase. 3.6
The supervised machine learning approach (MAUI) was used where a classifier was trained by using all the training documents provided with known training catchphrases in the training set. No candidates were discarded prior to training the model. We then used the trained model to generate catchphrases for the test documents.
UBIRLeD_2: 150 catchphrases were generated for each test document. The highest ranked catchphrases appeared first for each test document. 4
Several measures were used to evaluate the results of the two approaches. In this experiments we looked at Recall, Precision and Mean Average Precision among others. 4.1
According to Manning et al [
Recall =
tp tp + f n
Recall@K would be the proportion of relevant catchphrases that
have been retrieved in the top-K.
Precision is described as the fraction retrieved documents which
are relevant according to Manning et al [
Precision =
tp tp + f p
Precision@K, would be the proportion of top-K catchphrases that are relevant. Mean Precision@K, will then cover the Mean of the Precision@K of each test document in the whole collection.
We used Manning et al [
Mean Average Precision (MAP ) value is defined as "the arithmetic
mean of average precision values for individual information needs"
Manning et al [
R
RPrecision = is given in Figure 4, where MAP (Q ) is mean of average precision across the whole collection list of queries being the test documents in this task. Precision(Rjk ) is the precision score of ranked retrieved catchphrases from the top results until position k for test document j. For each of the test documents, a set of ranked catchphrases was produced, which was then used to compute precision and average precision (AP). Average precision is the mean of the precision scores after each relevant catchphrase is retrieved.
MAP (Q ) = 1 jQ j 1 mj
∑ ∑ Precision(Rjk ) jQ j j=i mj k=1 Consider the results displayed in Table 3 UBIRLed_1 and UBIRLed_2 rows contain performance measures obtained after using the generated catchphrases from RAKE and MAUI respectively as mentioned in . Using the performance measures stated in Section 4, we observed that MAUI; the supervised approach, performed far much better than RAKE; the unsupervised approach. Comparing the results based on Mean Precision@10, we discovered that the proportion of top 10 catchphrases which were relevant was more efective using MAUI, MAUI result was 0.254 while RAKE result was 0.013. We also looked at the Mean Recall@100, MAUI still outperformed RAKE by retrieving more relevant catchphrases in the top 100. When finding MAP, the assumption was that we were interested in finding more relevant catchphrases for each test documents and hence we computed the Mean of average precision values of each test documents. The value of MAP obtained for MAUI was higher than the value computed using RAKE results. The Mean R precision value for MAUI had far much better proportion of retrieved catchphrases which were relevant considering the cutof point which was equals the number of relevant catchphrases expected for each and every document provided in the testing dataset. Overall recall, RAKE was better although that was the only measure good compared to MAUI’s performance. UBIRLed_1 UBIRLed_2
In this paper we had proposed and compared two Machine Learning approaches namely: RAKE and MAUI for the unsupervised and supervised approaches respectively. In the proposed approach, ifne tuning parameters before generating candidate catchphrases resulted in obtaining the optimal parameters for each method used. Based on the optimal parameters used for generating the final catchphrases, overall MAUI had high performance compared to RAKE. The diferences in the performance was observed in most areas. RAKE achieved the highest recall but the precision was very low compared to MAUI. We strongly believe that Legal domain is an area which still requires a lot of work on Information Extraction. For the future work, we plan to experiment with diferent techniques used on the supervised approach in Machine learning and evaluate the performance after applying the diferent techniques.