Wikipedia, the multilingual, free content encyclopedia has evolved as the largest and the most popular general reference work on the Internet. Since the time of commencement of Wikipedia, crowd sourcing of articles has been one of the most salient features of this open encyclopedia. It is obvious that enormous amount of work and expertise goes in the creation of a self-content article. However, it has been observed that the infobox type information in Wikipedia articles is often incomplete, incorrect and missing. This is due to the human intervention in creating Wikipedia articles. Moreover, the type of the infoboxes in Wikipedia plays a vital role in the determination of RDF type inference in the Knowledge Graphs such as DBpedia. Hence, there arouses a necessity to have the correct infobox type information in the Wikipedia articles. In this paper, we propose an approach of predicting Wikipedia infobox type information using both word and network embeddings. Furthermore, the impact of using minimalistic information such as Table of Contents and Named Entity mentions in the abstract of a Wikipedia article in the prediction process has been analyzed as well.
Since the commencement of Wikipedia, it has emerged as the largest multilingual encyclopedias available on the Internet. It is the most widely used general reference non-pro t crowd sourcing project, owned by the Wikimedia Foundation3. A huge amount of expertise and e ort is involved in the creation of Wikipedia articles. Wikipedia articles are generated as an amalgamation of the information contributed by humans in all the di erent segments of the article layout. A typical Wikipedia article comprises both structured and unstructured data. The unstructured data consists of the text describing the article content whereas, structured data is represented in the form of an infobox containing property
value pairs summarizing the content of the article. An infobox is a xed-format
table usually added to consistently present a summary of some unifying aspects
that the articles share and sometimes to improve navigation to other
interrelated articles4. Furthermore, the structured data present in the infoboxes of the
Wikipedia articles are widely used in di erent Knowledge Graphs (KGs) such
as DBpedia, Google's Knowledge Graph, Microsoft Bing's Satori etc. [
The selection of the infobox type is determined collaboratively through
discussion and consensus among the editors. The infobox types or templates are
created and assigned based on categorical type of the articles, i.e. the same
template should be assigned to similar articles. However, no integrity tests are
conducted to check the correctness of the infobox assignments, leading to
erroneous infobox types [
Moreover, RDF type information in KGs such as DBpedia is derived directly
from Wikipedia infobox types by automated information extraction. Therefore,
the completeness and correctness of the infobox type information is of great
importance. Di erent studies [
In this paper, we present a novel approach to predict Wikipedia infobox types by using word embeddings on the text present in theTable of Contents(TOC), the article's abstract, and additionally network embeddings on the Named Entities mentioned in the abstract of the article. The TOC consists of the headings and the subheadings of the article text, summarizing the information content of the text in the section underneath. To the best of our knowledge so far, Wikipedia infobox types have not been predicted using di erent types of embeddings including TOC as one of the features. In this paper, the impact of using a minimalistic yet informative feature such as the TOC in the classi cation process via classical and neural network based classi ers is studied. Additionally, the importance of Named Entities mentioned in the abstract of the article to determine the infobox types has been analyzed.
The rest of the paper is structured as follows. To begin with, a review of the related work is provided in Section 2 followed by a short description of the approach in Section 3. Section 4 accommodates the outline of the experimental setup followed by a report on the results in Section 5. Finally, an outlook of future work is provided in Section 6.
Scope. The aim of this work is to predict the infobox types by leveraging the
Table of Contents, abstract and the Named Entity mentions in the abstract of
the Wikipedia articles. This section presents prior related work on infobox type
prediction. RDF type information in DBpedia is derived from Wikipedia infobox
type information. Therefore, the Wikipedia infobox type prediction problem can
be seen as a closely related task of RDF type prediction, which is covered rst
in the subsequent section followed by Wikipedia infobox type prediction.
RDF Type Prediction. A statistical heuristic link based type prediction
mechanism, SDTyped, has been proposed by Paulheim et al. and was evaluated on
DBpedia and OpenCyc [
As already mentioned, these approaches infer DBpedia RDF type information of entities by taking into account properties present in DBpedia or Wikipedia content. On the contrary, we intend to predict the Wikipedia infobox types by considering the TOC, abstract and Named Entities. Hence, these are worth mentioning but di erent from the proposed work of this paper.
Wikipedia Infobox Type Prediction. One of the initial works in this domain
was proposed by Wu et al.[
Sultana et al.[
Yus et al.[
In contrast to the aforementioned works, we propose a classi cation based infobox type prediction approach by combining word embeddings and network embeddings to generate feature vectors instead of TF-IDF. The proposed method does not focus on creating new infobox templates rather it focuses on correcting and complementing the missing infoboxes in the articles. Furthermore, unlike prior works, the TOC is also considered as one of the features for the classi cation process.
This section contains a detailed explanation of the work ow and the methodologies used for the multi-label classi cation process to predict the infobox type information. The work ow is illustrated in Figure 1. 3.1
Three di erent features are extracted from the Wikipedia articles for the classication task: { TOC5 which is automatically generated based on the section and subsection headers of the Wikipedia articles depicting a summarization of the content in a single word or a short sentence. { Abstract(A) of the Wikipedia articles i.e. the summary of the entire article content. { Named Entities(E) present in the abstract section of the articles are most likely to be related to the article hence assumed to provide more information in the classi cation process. The internal hyperlinks within Wikipedia are used to identify the Named Entities mentioned in the abstract. 3.2
Both word and network embeddings are used to generate features for the classiers.
Word2Vec. Word2vec [
RDF2Vec. RDF2Vec [
to extract the vectors for the Named Entities mentioned in the abstract of the Wikipedia article. The vector length has been restricted to 200 features. Similar to the word2vec word vectors, these vectors are generated by learning a distributed representation of the entities and their properties in the underlying KG. The intuition behind incorporating the vectors of Named Entities mentioned in the abstract of Wikipedia articles into the feature set is to include the features or the properties of the di erent entities from the DBpedia KG into the classi cation process. 3.3
The feature vectors for each of the Wikipedia articles are generated using the following steps: Step 1: Extract word vectors for each word in the TOC as well as the abstract from the Google pre-trained model. Also, extract entity vectors for the Named Entities mentioned in the abstract from the RDF2Vec pre-trained model of DBpedia version 2016-04.
Step 2: Generate an abstract vector for each document by performing vector addition on all the word vectors of the abstract and normalize by the total number of words present in the abstract. Similarly, TOC vectors and entity vectors are also generated.
Step 3: Generate document vectors - Two sets of document vectors are generated for the training of two classi ers. The abstract vector of each document is concatenated separately with the TOC vector and entity vector of the corresponding document to generate the document vectors. 3.4
As already discussed, the Wikipedia infobox type prediction problem can be
reduced to a classi cation of the Wikipedia articles with word vectors and entity
vectors as features. In this work, we have trained the Wikipedia articles using
two classi ers: Random Forest (RF) and Multilabel Convolutional Neural
Network (CNN). For the Random Forest classi er, the aforementioned document
vectors coupled with the labels of the infobox types are used to train the
classier. Random Forest, as an ensemble method is less likely to over t. Moreover,
the subsets of the training set for bagging reduces the e ects of the outliers
in the data, if any. On the other hand, for CNN, the concept of sentence level
classi cation task as discussed in [
This section contains a detailed explanation of the dataset followed by the
generation of ground truth. For the Random Forest classi er, python scikit-learn8
library has been used. For the CNN model, to classify the Wikipedia articles,
based on the sentence classi cation concept as described in [
Wikipedia 201610 version and RDF2Vec pre-trained model for DBpedia version 2016-04 have been used for this work. This version contains around 2000 di erent Wikipedia Infobox types. It has been observed that the frequency distribution of the Wikipedia Infobox types follows Zipf's law as shown in Figure 2. The x-axis represents the infobox types in numbers and y-axis represents the count of entities per infobox type.
More than half of all Wikipedia pages do not contain infobox types. Also, there are articles in Wikipedia containing more than one infobox type. However, in this work we considered only those articles having a single infobox type. The statistics of the Wikipedia articles with infobox types is provided in Table 1. It consists of the count of the Wikipedia articles containing TOC, abstract, infoboxes and the combination of these three together. Wikipedia redirect pages, disambiguation pages and the list pages are ignored in the dataset.
In this work, based on the popularity of the infoboxes, we have considered top 30 infobox types as labels with 5000 articles per label to train the classi ers. These 5000 articles per infobox type are selected by random sampling without replacement and the experiments are being carried out with three such datasets of the same size. It is important to note that the experiments are carried out to study the impact of the TOC and Named Entities separately as well as in combination with the abstract. )e tyxp 3 o b o f irenp 2 y itt (ceyn 1 enu reqF 0
Frequency
Features
TOC
Infobox(I) TOC + A + I #Wikipedia articles 9,959,830 4,935,279 2,626,841 2,575,966 For the Random Forest classi er, the experiments are carried out for both 5-Fold Cross Validation (CV) as well as split with 80% data as train set and 20% data as test set. Identically, for the CNN 80% data is considered as train set and 20% data as test set. The results are discussed in the next section. 4.3
Cross-Validation techniques are adequate, testing the proposed approach over unseen data reveals the generalization of the model and its robustness. To the best of our knowledge no benchmark exists in the eld of Wikipedia infobox type prediction. Therefore, automated Ground Truth has been generated for the purpose. The manual creation of ground truth has the advantage of yielding benchmarks incorporating human knowledge on the topic. On the other hand, it incorporates signi cant disadvantages in terms of huge manual e ort leading to a very small amount of ground truth generated. Therefore, in this work, the focus was to automatically generate ground truth with preserving the characteristics of the manually credited one. To do so, rst, all the articles without infoboxes were extracted from Wikipedia version 201611. Second, these articles were checked against the latest Wikipedia version 201812 to nd if there existed an infobox type in the new version for them. This approach leads to the generation of ground truth comprising of 32000 Wikipedia articles in total. 4.4
TF-IDF is one of the most widely used method to generate the vectors for the
text classi cation problem [
The experiments establish the fact that the text present in the TOC of a Wikipedia article is very minimal yet informative and its contribution in the classi cation process is not insigni cant. With the CNN classi er, it reaches an accuracy of 76.5% micro-F1 score and around 65% with Random Forest. On the other hand, with the TF-IDF vectorizer the accuracy is very little. This is due to the fact that TF-IDF captures how important is the word for a document whereas the word embeddings capture the semantic similarity of the words in the text. Since the TOC is generated from the headings provided by the authors at the time of the creation of the articles and no guidelines being available, the vocabulary of the headings varies from article to article. Hence TF-IDF fails to capture the semantic similarity between them. For instance, in the Wikipedia article for Bill Gates, his early life is described under the section named `Early Life' whereas for Steve Jobs, his early life is described under the section `Background' and subsections `Biological and adoptive family' and `Birth'. TF-IDF fails because it treats them as di erent words. On the other hand, `family' appears in the top-5 similar words for `life' using the Google pre-trained vectors. Hence, the semantic similarity of the words is considered. This also explains the reason behind using the pre-trained vectors from Google. Therefore, it can be inferred that using word embeddings over minimalistic yet informative key words from an article is capable of predicting Wikipedia infoboxes.
However, using the abstracts as a feature to determine infobox types improves the quality of results both for TF-IDF as well as for word embeddings. A huge improvement is noticed over the TF-IDF approach due to the fact that long texts improve the quality of the vectors by this process. Furthermore, embeddings still work better for the classi cation process capturing the semantic similarities in text.
Combining both the features, abstract and TOC together provides the best result for all the cases. The data is less sparse now for TF-IDF to work better as compared to only considering TOC. Also, there is an improvement of 2% in the micro F1 score for the random forest and 1% increase in the CNN classi er from using only abstract. It can be inferred for the articles with very less text in the abstract but a well sectioned article for the rest of the document, TOC can be considered to play a vital role in the prediction process.
Additionally, for all the combination of the feature set as explained in Table 2, CNN performed better than Random Forest proving the fact that CNN models are trained better for large datasets. We also carried our experiments with SVM for a couple of the aforementioned feature sets and obtained similar results as with the Random Forest classi er. However, multi-class SVM classi cation with one vs. all approach was computationally expensive as compared to random forest for this task.
The impact of Named Entities in the classi cation process has also been studied as shown in Table 3. It is to be noted that with only the Named Entities in the abstract, the classi cation process is not as e ective achieving a micro F1 score of around 45% with Random Forest Classi er and 62% with CNN model.
Feature Set RF(CV) RF(Split) CNN
RF(CV) RF(Split) However, the addition of abstract word vectors to the entity vectors of an article leads to an improvement of up to 86%. Moreover, the combination of the abstract, TOC and the Named Entities vectors from the word and the network embeddings improves the classi cation accuracy to 87% using random forest. This is a considerable improvement in comparison to the TF-IDF score which is 83% with all the features combined together. Furthermore, it has been noticed that classi cation with Table of Contents has a better micro F1 measure compared to the Named Entities with both Random Forest and CNN approach. Hence, it can be inferred that the word vectors for the words in the Table of Contents is capable of capturing more features relevant to the Wikipedia infobox type prediction problem compared to the entity vectors extracted from the RDFtoVec model. However, experiment with the CNN model by combining both word and entity embeddings together has not been performed because of the unequal length of the pre-trained vectors.
Last, Wikipedia articles in the ground truth i.e. the articles without infobox type, tend to be shorter and less informative due to various reasons such as, relatively new, less popular topic, lack of knowledge of the contributor on the topic etc. Therefore, the articles in the ground truth possess di erent characteristic as compared to the training data. Prediction of infobox types on the ground truth for the articles having TOC and abstract using the trained random forest model yiel.ds 53.7% micro-F1 score. 6
In this paper, a novel approach for Wikipedia infobox type prediction based on di erent types of embeddings has been analyzed. Also, the impact of using TOC and Named Entities separately as features to predict an infobox type has been studied. The achieved results strengthen the fact that adding only TOC with the abstract in the feature set improves the accuracy of the classi cation process hugely. On the other hand, entities, if used together with the abstract, also have a positive impact on the classi cation process. Wikipedia infobox type prediction is an important task as KGs such as DBpedia are constructed by automatic information extraction from Wikipedia infoboxes. Hence, cleaning and assigning the correct types of Wikipedia infoboxes indirectly leads to an improvement of DBpedia type information. Additionally, this method can be extended to any number of Wikipedia infobox type classes with articles having abstract and/or TOC. Next, we would like to train CNN model combining both word and network embeddings. Moreover, as network embedding models can be applied in this task, we would like to train a network embedding model directly to the Wikipedia articles, instead of using Google vectors and analyze its impact in the classi cation process.