We present the solution to the CCKS 2017 question entity discovery and linking (QEDL) task. This task is to discover and link entity mentions in natural language questions with their referent entities in a knowledge base (KB). For entity discovery, we devise recognition patterns based on word segmentation and POS tagging. For entity linking, we leverage contextual similarity refined by rich side information contained in the KB. Our solution is fully unsupervised and resource-free, requiring neither labeled data nor auxiliary resources. Experimental results show that our solution is simple yet effective, achieving an F1-score of 44.3% which ranks the third in the QEDL task.
The CCKS 2017 question entity discovery and linking (QEDL) task1 is to
recognize entity mentions from natural language questions, and link them with their
referent entities in a given knowledge base (KB), i.e., CN-DBpedia [
Entity discovery and linking has long been regarded as a challenging task in
natural language processing (NLP) [
To address these challenges, we devise a fully unsupervised method for QEDL. In our approach, entity discovery is conducted based solely on the results of word segmentation and POS tagging. Entity linking is performed by measuring contextual similarity between entity mentions and their referent in the KB. As questions are short with insufficient contexts, we further leverage side information, e.g., titles, types, and primary tags of entities, to refine contextual similarity. Our approach is fully unsupervised and resource-free, requiring neither labeled training data nor auxiliary resources like hand-crafted dictionaries or thesauri. Our approach is simple yet effective, achieving an F1-score of 44.3% which ranks the third in the CCKS 2017 QEDL task. 2
Entity discovery is closely related to named entity recognition (NER) which
recognizes entities of specific types (person, location, and organization). Studies
on NER roughly fall into three categories: 1) rule-based methods which use
handcrafted rules and dictionaries to design recognition patterns; 2) machine
learningbased methods which pose NER as a sequence classification problem, solved by
hidden Markov models or conditional random fields; 3) hybrid methods which
combine rule-based and machine learning-based approaches. For more details
about NER methods, please refer to [
Entity linking is to link textual mentions with their referent entities in a
given knowledge base. Existing approaches can be roughly categorized into two
groups: 1) supervised methods which rely on massive annotated data to learn
how to rank candidate entities for each textual mention; 2) unsupervised methods
which do not require any annotated data to train the ranking model. See [
Given that only a small number of annotated data is provided in the QEDL task, we employ a rule-based method for entity discovery, and an unsupervised method for entity linking. 3
Entity Discovery • Recognition patterns − Word segmentation − POS tagging
Mention: 吴教
Entity Linking • Contextual similarity • Side information − Entity titles − Entity types − Entity primary tags
Candidate Entities 吴教 Xiaomin Wu (professor) Xiaomin Wu (actress) 吴市 Xiaomin Wu (mayor) word segmentation and POS tagging. Entity linking is then performed by measuring contextual similarity between textual mentions and their referent entities, refined by rich side information in the KB. Our approach is fully unsupervised and resource-free, requiring neither labeled training data nor auxiliary resources. 3.1
Entity Discovery Given an input natural language question, we employ the SWJTU Chinese word segmentation system to perform word segmentation and POS tagging. This system supports two segmentation manners, i.e., coarse-grained and fine-grained segmentations. The former uses a longest matching algorithm, and the latter can split the words into smaller units. For example, “新浪微博/Sina Microblog” is segmented into a single word “新浪微博/nt” in coarse-grained segmentation, but two separate words “[新浪/ntc 微博/n]/nt” in fine-grained segmentation. Here, “nt”, “ntc”, and “n” are POS tags.2 After word segmentation and POS tagging, we detect entity mentions as follows (see Table 1 for concrete examples). Rule 1 Words with coarse-grained POS tags of nr (person name), ns (place), and nt (organization) are recognized as entity mentions, e.g., “霍建华”, “宁 波”, and “普陀山”.
Rule 2 Words with coarse-grained POS tags of nz (proper noun) and n (noun) are recognized as entity mentions if they have redirects in the KB, e.g., “湘潭 火车站”. Otherwise, any fine-grained units therein that have redirects are determined as entity mentions, e.g., “中国” in the coarse-grained segmentation “[中国/ns 领土/n]/nz”.
Rule 3 Words with coarse-grained POS tags of nz (proper noun) can further be concatenated with their antecedent or succedent words. If the combined words have redirects in CN-DBpedia, they are also identified as entity mentions, e.g., “qq木马病毒” and “三星note2”. 2 They stand for organization, company name, and noun, respectively. A full description of POS tags is available at http://ics.swjtu.edu.cn. 霍建华/nr 演/v 过/uguo 哪些/ry 电视剧/n 从/p 宁波/ns 到/v 普陀山/ns 怎么走/nz 最/d 方便/a [湘潭/ns 火车站/n]/nz 什么/ry 时候/n 通车/vi 求/v 钓鱼岛/ns 属/v [中国/ns 领土/n]/nz 的/ude 资料/n qq/x [木马/n 病毒/n]/nz 怎么/ryv 编写/v 三星/nz note2/x 电池/n 怎么样/ryv Entity mentions 霍建华 宁波, 普陀山 湘潭火车站 中国, 钓鱼岛 qq木马病毒 三星note2 3.2
Entity Linking Entity linking consists of three modules: candidate entity selection, candidate entity ranking, and NIL (unlinkable entities) detection, detailed as follows. Candidate entity selection. For each recognized mention, we query it directly in the CN-DBpedia search engine3 and retrieve a list of relevant entities. These entities are taken as candidates for that mention.
Candidate entity ranking. We rank candidates for each mention by measuring their contextual similarity. Specifically, given a mention m and a candidate entity e, we construct two feature vectors m and e for them. The former is composed of context words of the mention in the question, and the latter context words of the entity in its abstract. Here only words that are tagged as noun and verb are considered. The contextual similarity between m and e can be calculated as, e.g., the dot product of m and e, i.e., s(m; e) = ⟨m; e⟩. However, since both the question and the entity abstract are short, we might not get enough contextual information in m and e. So we propose to further use side information in CNDBpedia, and calculate a refined contextual similarity s˜(m; e) = w × s(m; e). Candidate with the largest s˜(m; e) score will be selected as the true referent. Three types of side information are considered to calculate the refining factor w, including entity title, entity type, and primary tag.
Entity title is the title of an entity page in CN-DBpedia. The intuition here is that referent entities are usually those that have similar titles with their mentions (string matching). For example, given the mention “格林豪泰”, the entity “格林豪 泰” is more likely to be the true referent than “林豪泰”. So we define the refining factor as string similarity between the mention m and entity title e, i.e., edit(m; e) w1 = 1 − max(|m|; |e|) ; where | · | is the length of a string, and edit(·; ·) the edit distance.
Entity type is the category to which an entity belongs, denoted as type(e). Usually, a mention can only be linked to entities of certain types, e.g., China/ns should be linked to countries. So we specify a type set T for each POS tag, e.g., 3 http://knowledgeworks.cn:30001/?p=** http://knowledgeworks.cn:20313/cndbpedia/api/entity?mention=** T = {Place; Country; City} for ns (space), and define the refining factor as 1;
type(e) ∈ T ; 1; type(e) = ∅; 2; type(e) ̸∈ T and type(e) ̸= ∅; where 0 ≤ 2 < 1 < 1. Here we specify type sets for only three POS tags, i.e., nr (person name), ns (space), and nt (organization).
Primary tags indicate the most popular entities of the given mentions. Candidates with primary tags are more likely to be the true referent. So we define the refining factor according to the presence or absence of primary tags, i.e., where is a parameter in the range of [0; 1). These three weights can further be aggregated together, giving a combined refining factor. For example, aggregating all the three weights gives a combined refining factor of w = w1 × w2 × w3. NIL detection. Not all mentions have referent entities in the KB. To detect such unlinkable mentions, we use a simple heuristic: mentions with no candidates after performing the candidate entity selection module are predicted to be NIL. To yield more accurate NIL, we do this only for mentions discovered by Rule 1. 4
Datasets and evaluation metrics. The training set consists of 1,400
questions with 1,917 mentions manually linked with their referent entities, and 63
mentions labeled as NIL. The test set consists of 749 unlabeled questions. As
our approach is fully unsupervised, we use the training data as a development
set only for parameter tuning. Submissions are finally evaluated on the test set.
Three metrics Precision, Recall, and F1-score are used for the QEDL task.
Implementation details. We use Rule 1, Rule 2, and Rule 3 to detect entity
mentions (Section 3.1). And for entity linking (Section 3.2), we test different
settings. In the calculation of contextual similarity, we use different term weighting
schemes including Boolean, TF, and TF-IDF [
Boolean (Dot) g TF (Cos) ikn TF (Dot) i TF-IDF (Cos) n L y TF-IDF (Dot) ittn TF (Dot)+Title E TF (Dot)+Type
TF (Dot)+Tag TF (Dot)+Title+Tag most entity mentions in short questions. However, the rules we created may still miss some difficult cases such as “红/a 米/n note2/x”, which will be studied in our future work. For linking, we can see that: 1) the TF term weighting scheme combined with the dot product similarity measure performs the best in calculating contextual similarity; 2) incorporating each of the three types of side information alone can further improve contextual matching; 3) among all possible combinations of side information, Title+Type performs the best, achieving an F1-score of 44.3% on the test set;4 4) the performance on the test set is better than that on the training set, which might indicate higher annotation quality of the test data.
This paper introduces our solution to the CCKS 2017 QEDL task. We first devise recognition patterns based on word segmentation and POS tagging to discover mentions. Then, we utilize contextual similarity refined by rich side information for entity linking. Our solution is simple yet effective for short questions, achieving an F1-score of 44.3% which ranks the third in the QEDL task. 4 During the test phase, we refine the outputs using labeled data in the training set.