Most keyword extraction algorithms mainly focus on the extraction from a document but not a sentence. A document hosts more information than a sentence, therefore keyword extraction from a sentence is a challenging task. In addition, keyword extraction from a sentence has potential application in many fields, such as question answering systems, text search, recommendation systems, etc. Therefore, this paper proposes the multi-feature fusion TextRank algorithm for sentence-oriented keyword extraction by integrating knowledge of features in the initial score of keywords and calculation of the probability transfer matrix. The initial scores of candidate keywords are adjusted by fusing the term frequency and part of speech features in the sentence. And the probability transfer matrix to calculate the scores is tuned by using the semantic and syntactic features among the candidate keywords. Based on the scores of candidate keywords, the top K words are selected as the keywords of the sentence. The experiments show that our method outperforms in the indices of P, R, and F.
Keyword extraction is one of the popular topics in the field of natural language processing[1]. It is also widely applied in our daily life since keywords help us save time by transferring the main ideas of documents through fewer words.
Keyword extraction from a document but not from a sentence is widely studied, such as from an abstract of a paper[2], news[3], patent texts[4], etc. In document keyword extraction, the knowledge in titles[3], paragraphs[5], and the location of words[6] can be used, and some deep learning methods as the end-to-end extraction are also presented[2].
Keyword extraction methods are divided into supervised and unsupervised types[1]. In supervised type, keyword extraction is regarded as a binary classification or multi-classification task[1]. In unsupervised type, they can be summarized into three kinds: keyword extraction based on statistical features, the topic model and the graph model.
Among the keyword extraction based on statistical features, TF-IDF (Term Frequency-inverse
Document Frequency)[7] is well-known for its simplicity and efficiency. The topic model based methods
extract keywords according to the topic distribution of documents[8]. Among the graph model based
keyword extraction, a popular method is TextRank algorithm[9]. Inspired by PageRank algorithm[10],
TextRank algorithm includes three steps: (
In the improvement from the scores initialization of candidate keywords, many features in document are introduced, such as term frequency[11], the length of words, the position of words, the part of speech[12], the narrative table[11], the importance of words in the document’s title[3]. For the improvement from the construction of the probability transfer matrix, the included features are as follows: the similarity[13] calculated according to Word2Vec, the reduction of the sparsity of matrix[5] based Doc2vec.
Compared with a document, a sentence is shorter, and it does not have adequate structure information.The challenge of sentence keyword extraction is the sparsity of text semantics [5]. Most of the existing keyword extraction algorithms are studied with the background of documents. This paper designs an algorithm suitable for sentence keyword extraction.
However, the TextRank algorithm has the following shortcomings when applied to keyword
extraction from a sentence: (
This paper proposes the multi-feature fusion (MFF) TextRank algorithm for sentence-oriented
keyword extraction to address the above shortcomings. For shortcoming (
The rest of the paper is organized as follows: Section 1 points out the shortcomings of the state of the arts, and reviews the related work on keyword extraction. Section 2 gives the main idea and the key elements of this paper. The experiments are shown in Section 3 and this paper is concluded in Section 4.
This section firstly introduces the main idea of the algorithm. Secondly, a method was proposed to improve candidate keyword scores. Thirdly, the algorithm to improve the construction of the probability transfer matrix is given. Finally, the multi-feature fusion TextRank algorithm for sentence-oriented keyword extraction is presented.
Compared with a document, a sentence has less information that can be made use of. Therefore,
when applying TextRank in sentence-oriented keyword extraction, the information will have to be fully
used. In this paper, we propose two ideas to make use of the information of the sentence: (
The above two ideas are embedded in the scores initialization of candidate keywords and the construction of probability transfer matrix separately, as shown in Figure 1, which gives the framework of MFF TextRank. Six parts are included in this framework, and they are organized according to their relationship in the keyword extraction. Firstly, based on candidate keyword sets and co-occurrence relationships, an undirected graph should be constructed. Secondly, the initial scores of the keywords are assigned. Thirdly, the probability transfer matrix is constructed. Fourthly, iteratively calculate candidate keywords’ scores. Fifthly, rank candidate keywords according to their scores, and at last take the top K candidate keywords with high scores as the result.
The model of TextRank algorithm can be formally expressed as: st (vi ) = (1 − d ) + d ∗ vj∈In(vi )
wji vk∈Out(vj ) wjk
st−1(v j ) Fig.1 MFF TextRank Framework
where In(vi) denotes the set of all nodes (words) that have edges heading to node vi; Out(vj) denotes
the set of nodes that have edges tailing from the node vj; st(vi) denotes the TextRank score of the node
vi in t-th iteration; wji denotes the weight of the edge between the node vj and vi; wji will be taken as the
element value in the probability transfer probability, and d∈[0,1] is a damping factor, and it is generally
set to 0.85 [9]. The vector form of TextRank can be rewritten as:
st = (1− d ) * e + d * M * st-1
(
The ideas to make use of the knowledge in a sentence to initialize the score of s0(vi), and to calculate the probability transfer matrix are explained separately as below.
The canonical TextRank algorithm assigns the initial scores of each candidate keyword to 1 or 1/N (N is the number of candidate keywords) by default, and it ignores the knowledge contained in each candidate keyword. In this paper, we integrate various knowledge and assign different initial scores to each candidate keyword accordingly. The initial score is calculated as:
s0 (vi ) = s1 (vi ) * s2 (vi ) where vi is the i-th candidate keyword; s1(vi) denotes the term frequency of vi; s2(vi) denotes the score of part of speech of vi. The higher s1(vi) is, the more important this word is. Similarly, since the part of speech of the candidate keyword is different, the score should be different. s1(vi) and s2(vi) are described as follows.From the perspective of part of speech, keywords in the sentence are often nouns, verbs, and adjectives[14], therefore s2(vi) is calculated according to the corresponding part of speech as follows: 2 for a noun, 1.5 for a verb, 1 for an adjective, and 0.5 for others.
The initial scores of all candidate keywords can be represented by a vector S0 with dimension n, as
shown in equation (
S0= ( s0 (v1 ) , s0 (v2 ) ,, s0 (vi ) ,, s0 (vn ))
Based on the canonical TextRank algorithm, in terms of semantics, this paper calculates the semantic similarity based on Word2vec. In terms of syntax, the relevance between words is calculated based on dependency parsing. They are described separately as below.
In terms of semantic similarity, if it is high, the weight between the two candidate keywords is high.
For two candidate keywords, vi, vj, their vectors vi, vj can be gained based on Word2vec. The semantic
similarity mij between them can be calculated according to cosine similarity, and the semantic similarity
matrix is shown in equation (
Mα = mij
In terms of syntax, if the dependency relevance[15] between two candidate keywords is high, their weight will be high. Although the Word2vec-based TextRank algorithm[13] can achieve good results on some publicly available datasets, it may not always be valid when it comes to a single sentence. Therefore, from the syntactic perspective, we calculate the relevance by dependency parsing. This paper takes LTP[16] (Language Technology Platform) as the tool for dependency parsing.
Therefore, based on the length of the dependency relation path, the dependency relevance is calculated according to:
where len(vi,vj) denotes the length of dependency relation path between vi and vj. The dependency
relevance matrix is shown in equation (
lij=1/len(vi,vj)
(
Mβ = lij
M = pij = [mij + lij ] = Mα + Mβ where pij denotes the transfer probability from node vi to node vj, and there is nj=1 p ji = 1.
With the initial words’ scores in (
The MFF TextRank algorithm is shown in Algorithm 1. The initial scores are assigned to the candidate keywords according to the word frequency and lexical features. Then the probability transfer matrix is constructed by considering both the semantic relationship characteristics between words and the syntactic relation among words to make the extracted keywords more accurate.
Algorithm 1 Multi-Feature Fusion TextRank
Input: A sentence; the number of keywords to be extracted, K
Output: Top K keywords
Step1: Pre-process the sentence: segment, remove the stop-words, and construct a candidate keyword set;
Step2: Calculate the initial scores for each candidate keyword vi:
(
Step6: Sort the candidate keywords in descending order of score and extract the top K words as the keywords.
The dataset in the experiments is from SogouCA[17], which is in Chinese and its size is about 1.4GB
and covers 18 fields, including military, sports, society, entertainment, etc. The preprocessing of the
dataset includes: (
We cross-labeled 500 sentences of hot topics randomly crawled from Baidu Knows (https://zhidao.baidu.com/) and Zhihu (https://www.zhihu.com/). In order to validate the reliability of the keyword extraction results, the keywords are extracted and manually cross-labeled. In the analysis of the experimental results, the extracted keywords are compared with the manually labeled keywords. The indices used in the experiments include the accuracy rate P, the recall rate R, and the F-measure.
The benchmark algorithms for experimental comparison include the TF-IDF algorithm (A1), the TextRank algorithm (A2), the TextRank algorithm with improved initial scores of candidate keywords (A3), the TextRank algorithm with improved probability transfer matrix by dependency parsing (A4), the TextRank algorithm with improved probability transfer matrix by Word2vec (A5), and the MFF TextRank (A6). When the number of extracted keywords is 1, 2, 3, and 4, the indices P, R, and Fmeasure are calculated, and the experimental results are shown in Table 1 and Figure 2. Table 1 Experimental results
Algorithm P NR=1
A1 0.744 0.200 A2 0.853 0.230 A3 0.867 0.233 A4 0.855 0.230 A5 0.884 0.238 A6 0.894 0.240
It can be seen from Figure 2 that the TextRank algorithm, which extracts keywords through cooccurrence relationships of words in the sentence, outperforms the TF-IDF algorithm, which selects keywords based on word frequency. After assigning the initial scores of candidate keywords based on term frequency and part of speech in TextRank, the performance of the algorithm is improved. It is further improved after integrating the dependency relevance in the probability transfer matrix in TextRank. After integrating the semantic similarity in the probability transfer matrix in TextRank, the algorithm performance becomes better. The MFF TextRank outperforms the above five algorithms in terms of the accuracy rate P, the recall rate R, and the F-measure.
The reasons that our algorithm outperforms other algorithms mainly include that: (
This paper proposes a multi-feature fusion TextRank algorithm for sentence-orient keyword
extraction. To address the shortcomings that the canonical TextRank algorithm ignores the knowledge
of keywords and the relationship between/among keywords, the TextRank algorithm is improved from
two aspects: (
Although the algorithm in this paper outperforms the other five algorithms in terms of P, R, and F, there is still room for improvement in terms of time complexity, and it is the future work to integrate suitable features to make the TextRank algorithm extract sentence keywords with higher performance.
This work was partly supported by the National Natural Science Foundation of China (No. 62077029, 61673196, 62277030), Society Development Foundation of Xuzhou under Grant No. KC19213, Jiangsu Normal University Postgraduate Research and Practice Innovation Program Project (2021XKT1391).
Communications (ICCC). Chengdu, pp. 2109-2113. [14] Zhang, J.-E. (2013) Method for the Extraction of Chinese Text Keywords Based on Multi-Feature
Fusion. Information studies: Theory & Application., 36(
In: Proceedings of the Coling 2010: Demonstrations. Beijing, pp. 13-16. [17] Wang, C., Zhangm M., Ma, S., Ru, L. (2008) Automatic online news issue construction in web environment. In: Proceedings of the Proceedings of the 17th international conference on World Wide Web. Beijing, pp. 457-466.