This paper presents a complete information technology for collecting and analysis of a citation network of scienti c publications aimed at detecting of seminal papers in a selected domain of research. The technology consists of the seed paper selection, plain snowball sampling, probabilistic topic modeling, greedy restricted snowball sampling, and analysis of the collected citation network. The topic model is built on the base of word-word co-occurrence probability with combination of sparse symmetric nonnegative matrix factorization and principal component approximation. Experiments with the collection of High Energy Physics abstracts show that the number of topics in the model is determined in natural way and the KullbackLeibler divergence correlates with cosine similarity calculated from keywords provided by publication authors. The citation networks on \critical thinking" and \automatic pronunciation assessment" domains are collected and analyzed. The analysis shows that both networks are \small worlds" and therefore the observed saturation of the restricted snowball sampling can provide the complete set of publications in domains of interest. Multiple runs of the sampling con rm the hypothesis that the set of seminal publications is stable with respect to variations of the seed papers. The modi ed main path analysis allows to distinguish the seminal papers including new publications following main stream of research.
getting answers to both questions. The essence of the technology is the collecting and analysis of a citation network of scienti c publications aimed at detecting of seminal papers in a selected domain of research.
To our best knowledge, while the separate parts of the method are developed, the entire procedure that takes the small set of papers on some scienti c domain and produces perfect list of references is not known.
The objectives of the presented work are:
{ to present the complete technology that takes a manually selected seed
papers and produces the short list of interconnected scienti c publications that
re ects evolution of main ideas of the selected scienti c domain. The
technology contains both restricted snowball sampling method and citation network
analysis.
{ to test all initial assumptions, namely
if the proposed snowball restriction method [
The distinctive features of the presented method are application of the probabilistic topic model to perform restricted snowball sampling and collect citation network, then the main path analysis is applied to point both the most in uential publications and the main path of scienti c knowledge evolution. The main path allows detecting the newest publications that follow the mainstream and the outliers that potentially contain the completely new ideas.
The structure of the paper is following. Section 2 overviews the publications related to the presented technology, Section 3 contains description of the crucial steps of the algorithm, Section 4 states the experiment pre-conditions and Section 5 discusses the results. Conclusion summarises the main results and discusses future work. 2
Publications on a domain of knowledge can be collected from conference
proceedings [
Semantic Scholar, https://www.semanticscholar.org/,
Microsoft Academic, https://academic.microsoft.com/ a set of keywords produces the biased set of publications [19] because di erent researchers use slightly di erent terms to report their results.
The most appropriate way to collect scienti c papers is snowball sampling
[
Another point of snowball sampling is dependence on the initial queue called
a seed collection. The general advice [
The snowball sampling cannot be applied directly to publication crawling because the list of references can contain the items that are not directly related to the investigated domain. Therefore the straightforward implementation of snowball publication sampling causes in nite collection in ation and some restrictions should be introduced to accept or reject the candidate publication. It should be noted that the introduced restrictions can violate the small world property and we need to check if the restricted snowball result is scale-free citation network.
To lter out the most relevant papers while sampling Ahad et al. [
Probabilistic topic models [24] use a large collection of documents and
statistical approach to model words and documents as vectors in a high-dimensional
semantic space Rn, where n is much less than number of words and number of
documents. The base idea of PTM is to construct few topis which are groups
of tightly connected words. Then document words are represented as a result
of two-stage random sampling. The most known method of topic modelling is
Latent Dirichlet Allocations (LDA) [
However, in most of the scienti c databases, full texts are often protected by copyright. Therefore the only information we can use are paper title, paper abstract, and sometimes the database-speci c keywords and topics. So the documents that we analyse are short and common PTMs based on document-word statistics lose their precision. This shortcoming is overpassed with approaches utilizing word co-occurrence statistics in Biterm Topic Model (BTM) [25] and Word Network Topic Model (WNTM) [26] instead of counting document-word pairs. Another method of word embedding, called GloVe, is proposed in [18]. It is based on word-word co-occurrence matrix and uses global matrix factorization, so it is close to BTM [25] and WNTM [26] statistical topic modelling.
Also, the vague part of common PTMs is that number of topics cannot be
determined with document analysis. To overcome this weakness, handling the
word-word co-occurences with principal component analysis (PCA) and sparse
symmetric nonnegative matrix factorization (Sparse SNMF) was proposed [
The collected citation network [22] can be analyzed using citation count and
other simple statistics [
The most appropriate way to highlight the seminal papers of the small scienti c domain is main path analysis because the method deals only with the collected citation network and allows to increase the precision of sampled dataset. On the contrary, the citation count and other statistics, supervised machine learning applied by Valenzuela, Ha and Etzioni [23] and PageRank cannot point out the tightly interconnected subset of the citation network. Klink-2 [17] and similar algorithms aim to build the map of knowledge domain but do not seek the most in uenced publications. 3
The general work ow of the restricted snowball sampling is introduced in [
The details of the restricted snowball sampling and probabilistic topic model
construction are discussed in [
The correctly built citation network must be an acyclic directed graph. However,
the publication database errors accidentally can cause cycles. The problem with
cycles is that if there is a cycle in a network then there is also an in nite number of
paths between some vertices. Since a citation network is usually almost acyclic
to transform it into an acyclic network we use the \preprint" transformation
described by Batagelj [
Our approach is similar to Search Path Count (SPC) algorithm [
The di erence of the applied algorithm from SPC algorithm is the
preservation of the citation graph connectivity. In the known algorithm [
Path resistance allows detecting new publications in the eld, not referenced yet by any other authors but existing in a mainstream of the domain. We can separate all papers into mainstream research and probably new research elds or directions. The smallest (up to a certain threshold) path resistances correspond to the mainstream, whereas the biggest path resistances correspond to the publications that are either brand new, bad written or published in a low impact journal/conference proceedings. We assume those publications are the source of potentially new ideas and topics. 5 5.1
We took three di erent corpora: \high energy physics"(HEP), \critical thinking"(CT) and \pronunciation quality assessment"(PQA).
HEP publications [20] are available from the European Laboratory for Nuclear Research. The hep-ex partition of the HEP collection is composed of 2802 abstracts related to experimental high-energy physics that are indexed with 1093 main keywords (the categories), the hep-astroph partition contains 2716 abstracts from astrophysics section and 18114 abstracts on theoretical physics in hep-th metadata. Each publication is manually annotated with keywords.
CT corpus is gathered with our snowball sampling software. The CT domain is characterized with a large noisy publications corpus tightly entangled with publications on psychology, didactics, pedagogy and phylosophy. The size of CT corpus is 24040 publication abstracts.
PQA domain is very speci c and narrow, with a moderate-size corpus containing 8339 scienti c abstracts collected by our snowball sampling software.
The sampling was run with following parameters: percentage of stop words to exclude { 2%; percentage of rare words to exclude { 5%; number of components in PCA which is maximal number of topics { 200; threshold KL-divergence { 0.18; sparsity parameter { 0.05; number of top citation paths { 50; minimal number of citations { 3. 5.2
On the part (a) of Figure 1 we can see that the mainstream of pronunciation assessment contains the publications:
The mainstream evolution of pronunciation assessment starts from application of automatic speech recognition, pays some attention to pedagogical aspects, goes to simple machine learning approaches, then to neural networks and to deep learning. Some of the seminal publications are the reviews containing discussions of the feature selection, methods comparison and combination. The part (b) of Figure 1 shows that the more top paths we keep, the more detailed knowledge map we obtain. 5.3
sampling method provides adequate semantic distance between publications. To check the statement we took HEP collection, built PTM for it, and measure similarity using keywords annotating each publication from HEP collection. For each pair of HEP publications were calculated both symmetric KL-divergence and cosine similarity. The results are shown in Figure 2.
As we can see, the PTM-based symmetric Kullback-Leibler divergence [
The restricted snowball sampling forms the scale-free network. Derek de Solla Price showed in 1965 [22] that the number of references to a paper (node degree) in a citation network had a heavy-tailed distribution following the power law and thus that the citation network is scale-free. One of our initial assumptions was that the restricted snowball sampling results in a scale-free networks so we need a few number of snowball iterations to achieve the high recall. Figure 3 was calculated on the base of PQA corpus and shows that for the small node degrees the logarithm of node number has linear dependency on the logarithm of node degree and for the large node degrees the dependency has heavy tail. That means, the restricted snowball sampling produces scalefree citation network as well as classical snowball. So we can be sure that a few iterations of the restricted snowball sampling allow collecting most of the relevant publications.
Saturation of the restricted snowball sampling. The restricted snowball sampling can be modelled as Poisson process when the publications appear sequentially and we can either (a) accept n-th publication and add it to the snowball or (b) don't accept. So we can calculate the con dence interval of Poisson distribution of event (a) and compare its upper bound with some pre-de ned acceptance probability. Figure 4 shows 0.95 con dence interval of Poisson distribution of paper acceptance as a colored strip and acceptance probability threshold 0.05 as a straight line. The con dence interval was calculated on the base of 10 snowball runs for CT collection starting from random subsets of seed paper collection. After some number of tested abstracts the upper bound of con dence interval becomes lower than the threshold so the restricted snowball sampling guarantees the saturation. The Citations Age To study the in uence of a publication age on the probability of the publication citation we have attributed each edge of the PQA citation network with age calculated as di erence between years of referencing publication and referenced one. The number of the edges as a function of edge age is shown on Figure 5. We can see that the maximal number of the references is observed for the publications that are 2{8 years old. Such publications are still regarded as new ones but at the same time are old enough to be read and estimated by many reserchers.
The biased seed papers can produce unbiased citation network. To estimate the stability of the restricted snowball sampling with respect to the seed papers variation we run the sampling starting from the full set of the PQA seed papers and mark the relevant papers with main path analysis. Then we run the sampling again starting from 10 random subsets of the PQA seed papers and count the number of the runs where each seminal paper occurs. In our experiments the random subsets contain 50% of the seed papers and 66% of relevant papers are detected every time, 14% { in 80% of runs, 14% { in 60% of runs, 6% { at least once. So we can conclude that the PQA citation network is stable with respect to large seed paper variations being input for the restricted snowball sampling and the result of the sampling is unbiased.
The main objective of the paper was to present the complete information technology that obtains a set of publications on some scienti c topic as input and produces a list of seminal publications for that topic. It provides data for future detailed analysis and serves as a good point to begin investigation in a new domain. Additionally, we tested several initial assumptions regarding the results of the technology application and show that: { PTM as a restriction criteria for the restricted snowball sampling method provides adequate semantic distance between publications. { The restricted snowball sampling guarantees the saturation. { The maximal number of the references is observed for the publications that are 2{8 years old. Such publications are still regarded as new ones but at the same time are old enough to be read and estimated by many reserchers. { The biased seed papers produce unbiased citation network. { The collected citation network is stable with respect to large seed paper variations being input for the restricted snowball sampling and the result of the sampling is unbiased.
The presented technology is implemented as sequence of Python scripts3.
17. Osborne, F., Motta, E.: Klink-2: integrating multiple web sources to generate semantic topic networks. In: International Semantic Web Conference. pp. 408{424.
Springer (2015) 18. Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: EMNLP. vol. 14, pp. 1532{1543 (2014) 19. Petticrew, M., Gilbody, S.: Planning and conducting systematic reviews. Health psychology in practice pp. 150{179 (2004) 20. Raez, A.M., Lopez, L.A.U., Steinberger, R.: Adaptive selection of base classi ers in one-against-all learning for large multi-labeled collections. In: Advances in Natural Language Processing, pp. 1{12. Springer (2004) 21. Salganik, M.J., Heckathorn, D.D.: Sampling and estimation in hidden populations using respondent-driven sampling. Sociological methodology 34(1), 193{240 (2004) 22. de Solla Price, D.J.: Networks of scienti c papers. Science 149(3683), 510{515 (1965) 23. Valenzuela, M., Ha, V., Etzioni, O.: Identifying meaningful citations. In: AAAI
Workshop: Scholarly Big Data (2015) 24. Vorontsov, K., Potapenko, A.: Tutorial on probabilistic topic modeling: Additive regularization for stochastic matrix factorization. In: International Conference on Analysis of Images, Social Networks and Texts x000D . pp. 29{46. Springer (2014) 25. Yan, X., Guo, J., Lan, Y., Cheng, X.: A biterm topic model for short texts. In: Proceedings of the 22nd international conference on World Wide Web. pp. 1445{ 1456. ACM (2013) 26. Zuo, Y., Zhao, J., Xu, K.: Word network topic model: a simple but general solution for short and imbalanced texts. Knowledge and Information Systems 48(2), 379{ 398 (2016)