We introduce the task of detection of competing model entities from scientific documents. We define competing models as those models that solve a particular task that is investigated in the target research document. The task is challenging due to the fact that contextual information is required from the entire target document to predict the model entities. Hence, traditional sequence labelling approaches fail in such settings. Furthermore, model entities themselves are long-tailed in nature, i.e, their prevalence in scientific literature is limited, along with a scarcity of labelled data for training supervised learning techniques. To address the above bottlenecks, we combine an Unsupervised Graph Ranking algorithm with a SciBERT-CRF based sequence labeller to predict the entities. We introduce a strong baseline using the above mentioned pipeline. Also, to address the label scarcity of long-tailed model entities, we use distant supervision leveraging an external Knowledge Base (KB) to generate synthetic training data. We address the problem of overfitting in small sized datasets for supervised NER baselines using a simple entity replacement technique. We introduce this model as part of a starting point for an end-to-end automated framework to extract relevant model names and link them with their respective cited papers from research documents. We believe this task will serve as an important starting point to map the research landscape of computer science in a scalable manner, needing minimal human intervention. The code and dataset is available in the given link : https://github.com/Swayatta/Competing-Models.
model names from a research paper and links them to their respective citation. While browsing related work The number of scientific publications in the computer for a given task, a researcher has to manually visit every science domain has increased exponentially in the recent research paper that uses a competing model that is used past. Hence, it has become increasingly cumbersome for the same task. This process is time-consuming if a for researchers to keep track of the advancement of the survey of a research landscape is to be done on a large research landscape. Often, research papers introduce scale. Our motivation is to automate this process by aunew models that perform strongly in comparison with tomatically extracting model names that solve a similar the baseline or advance the state-of-the-art. In order to task and linking them to their corresponding cited paper. efectively benchmark models and compare their perfor- If executed on a large scale, this pipeline would be able mances, it is important to be able to map the research to efectively map the computer science research landlandscape for similar or related tasks. Papers with Code scape in an automatic and scalable manner with minimal (Pwc1) is a community driven corpus that serves to au- human intervention. tomatically list models that solve particular subtasks , We introduce a strong baseline for this task by comwith links to the scientific research paper that introduced bining an unsupervised document level graph ranking the model. Our aim is to build a similar but automated algorithm and a supervised BERT-based sequence tagger end-to-end pipeline which detects model names from sci- to obtain entity model names. Essentially, we treat the entific papers and benchmarks them against other similar relevant keyphrases extracted by the graph ranker as a models that solve the same task. superset of candidates for the sequence labeller.
In this paper, we introduce the task of extracting com- We introduce two datasets for this task. For training peting model names from a research paper. We establish the supervised sequence tagger, we create weakly superan end-to-end pipeline that extracts all the competing vised distant labels using an external Knowledge Base and unlabelled corpora. We also release a manually annotated dataset for the evaluation purpose of the sequence tagger. For evaluating the entire framework of competing model name extraction, we release another dataset with full paper document level annotation. Furthermore, Proceedings of the AAAI-22 Workshop on Scientific Document Understanding at the Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-22)
© 2021 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org)
1https://paperswithcode.com/
we use a simple entity citation linking technique to link the extracted model names with their respective citation in the research document. We believe this task will be a significant step forward towards mapping the research landscape of computer science.
Our contributions can be summarised as follows: using supervised training using deep learning models.
However, supervised learning techniques require a large
amount of token-level labelled data for NER tasks.
Annotating a large number of tokens can be time-consuming,
expensive and laborious. For real-life applications, the
lack of labelled data has become a bottleneck on adopting
deep learning models to NER tasks.
• We introduce a novel approach of treating ranked Most scientific named entities can be classified as
longkeyphrases as a superset of sequence labellers for tailed entities because of the rarity and domain-specificity
solving this task. To the best of our knowledge, of their occurrence. Recent work on NER in scientific
docthis approach has not been used before in prior uments has been concentrated around detecting
biomedresearch work. We believe this approach can be ical named entities [10] or scientific entities like tasks,
extended to other similar tasks that require docu- methods and datasets [
dataset for final evaluation. NER has traditionally been treated as a sequence la• We introduce strong baselines while relying only belling problem, using CRF [13] and HMM [14]. Recent on distantly supervised weak labels to train our approaches have used deep learning based models [15] sequence labeller. We evaluate the trained model to address this task, which require a large amount of on our annotated evaluation dataset. labelled data to train. The high cost of labelling remains the main challenge to train such models on rare long tailed entity types, where availability of labelled data is scarce. In order to address the label scarcity problem, several methods like Active Learning [16], Distant Supervision [17, 18, 19], Reinforcement Learning-based Distant Supervision[20, 21] have been proposed. [12] focused on detecting dataset mentions from scientific text and used data augmentation to overcome the label scarcity problem.
Unsupervised Ranking Algorithms for Keyphrase
Extraction: EmbedRank[
Sequence labelling for Named Entity Recognition: Long tailed entities are named entities which rarely occur in text documents. For these types of entities, the task of Named Entity Recognition (NER) is non-trivial. Recent approaches have aimed at solving the problem of NER
Papers with Code (PwC2) is a community driven corpus that serves to automatically list models that solve particular subtasks, with links to the scientific research paper that introduced the model. Our aim is to build a similar but automated end-to-end pipeline that detects model names from scientific papers and benchmarks them against other similar models that solve the same task. We believe the task introduced in this paper (extraction of competing model names from scientific documents) to be a significant step forward towards the whole pipeline.
Competing Competing
Other transition-based models extend TransE to additionally use projection vectors or matrices to translate head and tail embeddings into the relation vector space, such as: TransH (Wang et al., 2014), TransR (Lin et al., 2015b), TransD (Ji et al., 2015), STransE (Nguyen et al., 2016b) and TranSparse (Ji et al., 2016).
In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019).
Non-competing
TransE [
To find the hyper-parameters, we used HyperOpt (Bergstra et al., 2015), which uses Bayesian optimization.
A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network SCIBERT: A Pretrained Language Model for Scientific Text On Evaluating Embedding Models for Knowledge Base Completion KG-BERT: BERT for Knowledge Graph Completion Tabular Data: Deep Learning is Not All You Need
In this paper, we present SDP-LSTM, a novel neural network to classify the relation of two entities in a sentence.
We define competing models as model names that at- autIonesnpciroeddebr,ywtheepuronpiqouseefaeantouvreelrmepordeesle,nntaamtioendleDaerenpinAguctaopeanbciolidtyero-flikdeeeNpMF tempt to solve the same task as investigated by the target (DANMF), for community detection. research paper. For example, if a research paper investi- We introduce the Multi-View Transformation Network (MVTN) that regresses gates the task of producing knowledge base embeddings, optimal view-points for 3D shape recognition, building upon advances in TransR [22] will be a competing model name as it has differentiable rendering. been introduced by prior research work to solve the same task. If a research paper investigates the task of Ques- Figure 1: Example sentences with annotated model name tion Answering, some competing model names can be T5 entities model [23] or XL-Net [24], because these are models that have been used to solve this task in prior research work.
A non-competing model name would be a model that has extracting the model names, we link the extracted entities not been used directly to solve the same task. We pro- with their respective cited papers. vide a few examples to illustrate the diference between a competing and a non-competing model in Table 1. For the first two examples, the models highlighted in bold are 5. Annotation Process competing models because they directly solve the task investigated in the input research paper. For the third We create two datasets for training and evaluation. We example, TransE is a competing model, but Word2Vec is annotate sentences from scientific papers as per tokennot. The reason for this is that TransE produces Knowl- level BIO tagging scheme to evaluate our sequence laedge Base embeddings directly that aid in Knowledge beller, which only uses contextual information from an Base completion (which is the target task in the research input sentence for sequence tagging. To evaluate the paper). But, Word2Vec is a language model that TransE whole pipeline, we provide document-level annotations is inspired by, as denoted in the sentence. Hence, it only with full length research papers as input and competing contributes indirectly to the research task. So, it is a model names as the annotated output. We use two difnon-competing model. Similarly, HyperOpt, in the last ferent datasets for a more comprehensive evaluation, as example, is non-competing, as it is an algorithm the au- our pipeline uses two stages. The first stage involves thors used for hyperparameter search and is not a model extracting candidate keyphrases utilising the entire docthat contributes directly in solving the task investigated ument level information for keyphrase ranking. The in the input research paper. second stage is our sequence labeller that uses sentence
Our task in this paper is to detect competing model level information to find model named entities. We denames given an input research document. Also, after scribe the annotation process for the dataset creation for sequence labelling first. Considering our end goal
length research papers. We read through the introduc# sentences tion and find out the task the paper solves. Then we ## teonktietniess browse the entire paper and find all mentions of model # unique entities names that solve a similar task. The process has a low avg # tokens per sen- level of ambiguity because a majority of the model mentence tions occur in the related work section, citation contexts avg # entities per sen- 2.44 3.65 2.57 or experimental results section. It is a standard practence tice among authors to cite the relevant research paper if they mention any model names from prior research work.
TOavbelreal2lstatistics of train and evaluation dataset for sequence Hence, we only consider models that the authors cite to labeller evaluation be candidates for competing models. We make sure the labelled entities are model names by referring to Google
Scholar and Semantic Scholar. If there is any ambiguity # total papers 75 regarding whether a labelled entity is a model name or # total sentences 34656 not, we discard the full paper. To infer if a model is a # avg sentences per paper 462.08 competing model or not, we find the task or the problem # entities 622 the paper solves. This is usually mentioned clearly in the # unique entities 473 introduction and the related work section. We label the # avg entities per paper 8.29 model entities (that the authors mention as solving a simTable 3 ilar problem or task as the original paper) as competing Overall statistics of the document-level annotated dataset for models. To further verify that the claim by the authors is evaluation of the entire pipeline indeed true, we visit the cited research paper and ensure that the model is solving a similar task. Furthermore, we only consider papers where the “competing” relation among models is clear and discard any paper where there of automating a high precision framework of extracting is ambiguity regarding this relation. Hence, we ensure related model names and to minimise ambiguity, we con- ambiguity to be significantly low regarding our annotasider only named models as model entities for this task . tions. The statistical details about the annotations are Few examples are - NMN+LSTM+FT, SpERT (with overlap), provided in Table 3. As we ensure a negligible level of B-BOT + Attention and CL loss, SA-FastRCNN, DS-CNNs ambiguity, we use only one human annotator (one of (Random Walk), Sparse Transformer 59M (strided). We the authors in this paper) for our annotation process. consider model entities that have a unique name or that We believe the need for multiple annotators for an interare formed by combination of other model names, eg - annotator agreement is insignificant for our task, as a NMN+LSTM+FT. A few example sentences with model low level of ambiguity is ensured by considering only entities are displayed in Figure 1. We define and annotate named models and clearly defined tasks with competing the test corpus using the standard BIO tagging scheme. model names.
Each model entity type was defined to have maximum span length. For Acronyms, we consider the full length 6. Method entity name instead of the short form acronym if it occurs in text - eg. DeCLUTR: Deep Contrastive Learning for Un- Our entire pipeline has two components. Firstly, we exsupervised Textual Representations. On average, there are tract all citation sentences from the input research paper. 2.5 tokens per entity. We refer Google Scholar and Seman- We combine all the citation sentences to create a minitic Scholar to confirm entity types. We randomly selected document. We use a graph ranking algorithm to extract a subset of abstracts from the arxiv dataset containing all the candidate keyphrases from this mini-document. 1.7M+ paper data and metadata and randomly select sen- This graph ranking algorithm utilises document level tences from them to annotate. Also, we randomly sample information to rank keyphrases. Secondly, we use a sethe DBLP citation dataset containing 1,511,035 papers quence labeller for extracting named entities from the and obtain the full length versions from the available pa- positively labelled citation sentences. Lastly, we merge pers using DOI matching and obtained a random sample the results of the graph ranker and the sequence labeller of sentences from the full text. We use two diferent sets to output final competing model entities. In the subsecof corpus because we want our model to be evaluated tion Sequence Tagging , we provide details about the on multiple domains within computer science and difer- training process and the model for our sequence tagger. ent publication venues. All the statistics related to our In subsection Graph-Ranking Algorithm, we provide deannotated corpus and train set are provided in Table 2 tails about the unsupervised graph ranking algorithm for For evaluating the whole pipeline, we annotated full
Let be the set of all citation sentences in a docu- corpus as a Knowledge Base. We crawl PwC and obrepresentation of . A set of candidate keyphrases is ex- names. For the unlabelled corpora, we use a total of keyphrase extraction. 6.1. Graph-Ranking Algorithm We use Multipartite Rank [9] as it had proved to be the state-of-the-art among all keyphrase ranking algorithms, performing particularly well on longer scholarly documents. We briefly describe how we use this algorithm for unsupervised keyphrase extraction. ment . forms an order set of citation sentences, which is collectively treated as a document. We build a graph tracted from . The candidate keyphrases are grouped into topics based on the stem forms of the words they share using hierarchical agglomerative clustering with average linkage. The candidate keyphrases are used to build a multipartite graph, where the nodes are keyphrase candidates that are only connected if they belong to a different topic. The edges between each node is weighted as the inverse of the distance between the two keyphrases , in . Weight inverse distances between
and : is calculated as the sum of the = ∑
∑ ∈( ) ∈(
1 ) − where (
) is a set of word ofset positions of . The ifrst occurring candidates of each topic are promoted more as they capture higher relevance. Weights of the ifrst occurring candidates of each topic is modified according: = + . 1
∑ ∈ ( )\
where is a hyperparameter that controls the strength of the weight adjustment, (
) is the set of candidates belonging to the same topic as , is the ofset position of the first occurrence of candidate . After the graph is built, a ranking algorithm is then used to order each keyphrase candidate . We adopt the popular TextRank
top ranked keyphrases ̃ is obtained. 6.2. Sequence Tagging For training our sequence tagger, we only rely on distant labels created using an external Knowledge Base and an unlabelled research text corpus. We also demonstrate that for long-tailed entity types, there is a need to ensure fairer distribution among entity occurrence in order to prevent overfitting, which occurs in the form of the model memorising certain popular entity names. The details about the training set creation is provided in section
tails about the model and the results on the evaluation set is provided in section Distantly Supervised NER Model.
is shown in Figure Training pipeline for the Sequence
We utilise the publicly available Papers with Code (PwC) tain all the model names occurring in the metadata for each task and subtask. We obtain a total of 14,748 model 227,000 abstracts from arxiv and obtain all sentences (7800) containing a model name mention. We find that the occurrence of some model names is much more frequent in literature (e.g - CNN). Due to the small dataset size and the large imbalance in few entity mentions, the model is prone to overfitting. To mitigate this, we use a simple entity replacement technique, where we find all model entity mentions, and randomly replace them with other names to ensure a fairer distribution. The distribution pre-replacement is shown in Figure 4. We use all 14,748 model entities at least once and limit an entity occurrence to at most 2 in the train dataset, after replacement.
We treat NER as a sequence labelling problem. Given a sequence of tokens = [ 1, ..., ], we aim to find an ≤ ) entity which is a span of tokens = [ , ..., ](0 ≤ ≤ associated with the entity type model name. We formulate this as a sequence labelling task of assigning a sequence of labels = [ 1, ..., ]. The aim of our sequence labeller is to classify each token as a certain entity type as per the BIO tagging scheme.
We consider train sentences denoted as {( , )} with distant token level annotations. We aim to learn a =1 function ( , )
, which can correctly predict the entity labels for a train sentence . We minimise the loss: ∗ = arg min 1 =1 ∑ ( , (
, )) over {( , )} cross-entropy loss.
=1
dard for the sequence labelling process.
where is the parameter and is the • A BiLSTM + CRF model where the bidirectional contextual representations are captured by the
Sentences The optimized 4-layer BiLSTM model was then calibrated and validated for multiple prediction horizons.
Furthermore, case studies show that SIMCLDA can effectively predict candidate lncRNAs for renal cancer.
Longformer's attention mechanism is a drop-in replacement for the standard self-attention.
Bi-LSTM MODEL SIMCLDA MODEL Longformer MODEL
Unlabelled corpora Knowledge
Base We evaluate our baselines using our evaluation dataset and the results are displayed in Table 4. We demonstrate that entity replacement provides a significant boost in • A BERT + CRF model where the contextualised performance for each of these models. The reason is embeddings are captured by a pre-trained BERT that the model does not memorise entity names for the base uncased model and passed onto the CRF replaced dataset and uses the context to predict the enlayer to produce token labels. tity types. The results also prove that standard NER • A SciBERT + CRF model where the domain spe- approaches can provide decent results on the evaluation cific contextualised embeddings are captured by dataset while relying only on weakly labelled training a pre-trained SciBERT [25] model. SciBERT is data.
labelled scientific corpora using MLM objective. The output embeddings are passed to the linear CRF layer which predicts token labels from contextual representations.
BiLSTM + CRF (w/o replacement) BERT + CRF (w/o replacement) SciBERT+CRF (w/o replacement) BERT+CRF (with replacement) BiLSTM + CRF (with replacement) SciBERT+CRF (with replacement)
We used the Unsupervised Keyphrase Extraction algorithm to capture only those keyphrases that are most relevant to the document. Although the Sequence Tagger performs well on detecting model name mentions using sentences as the contextual information, we need to capture document level relevance as well to extract competing models. The reason is that not all model name mentions are relevant to the task the given target research paper aims to solve. Hence, we predict only those entities which are common to both top-ranked keyphrases and the extracted model names from our distantly supervised sequence tagger. More formally,
″ = ̃ ⋂ ̃ where ̃ is the set of predicted entities by the sequence tagger, ̃ is the set of top-ranked keyphrases and ″ is the final set of predicted entities. The entire inference pipeline is illustrated in the Figure 3.
sion, Recall and F1-Score to evaluate the performance of the diferent baselines investigated. We use the full document-level annotated dataset for this evaluation.
We report the results in Table 5. We compare
performances of 4 Unsupervised Graph-Rankers for keyphrase
extraction: TextRank [
The entity citation linker is inspired from the prior work of [27]. The aim of this algorithm is to link the entities with their corresponding citation. The first step is to obtain all the possible entities and the citations. Then, a closeness score is calculated for each entity-citation pair, which is the string distance between the entity and the citation. Then, we take all the citations and keep only the closest citations per entity. Finally, we take all the entities and keep the closest entity per citation. As demonstrated by the authors, this technique is able to accurately map most entities with their corresponding citations. We use this technique to link all the extracted model entities with their respective citations. 10. Error Analysis 12. Conclusion and Future work We conduct error analysis for the Unsupervised We have introduced the task of extraction of competing keyphrase extraction, model entity extraction using se- models from a research paper. We use a novel approach quence labelling, the two-stage framework and entity of treating relevant keyphrases extracted using an Uncitation linking. For the keyphrase extraction, the graph- supervised Graph Ranking algorithm as the superset of ranker extracts most of the relevant model candidates. a BERT-based sequence labeller. We also use distant However, precision sufers significantly as most mod- supervision to train our sequence labeller. We test our seels are not keyphrases. Their are multiple keyphrases quence labeller and the entire pipeline on two annotated extracted by the algorithm that are not model names datasets. We also utilise a simple entitiy replacement - few examples being domain names like ‘Information technique to reduce overfitting in the sequence labeller. Retrieval’, ‘Networking architecture’, dataset names like Finally, we use the entity-citation linking technique to ‘SquaD 1.1’ or other terms that are relevant to the re- link all the extracted model entities with their respective search paper. citation. We believe this work to be a significant step for
For the sequence labeller, we observe mainly two types ward to map the research landscape of Computer Science of error. First, we notice precision error being introduced in an automated and scalable manner. into the model because in the training set we consider maximum span of each entity and the occurrence of IModel ( token lying inside a named entity) is relatively References high. However, in the evaluation test set of the sequence labeller, the occurrence of singular B-Model entities is massively more. This leads to the misclassification of O as an I by the model. Also, although the model is able to detect model entities reasonably given the sentence as the context, it is unable to discern competing models from unrelated ones. This leads to a significant precision decrease when evaluated on the document-level annotated evaluation set.
Finally, after evaluating the performance of the twostage pipeline on the document-level annotated dataset, we find that the model often mistakes dataset names for model entity mentions. This can be attributed to the high relevance of datasets with respect to the research paper.
Lastly, for the entity citation linker, sometimes an entity that is associated with a citation marker occurs in the initial part of a sentence and its not the closest to the citation. This can lead to missed out or incorrect linking. 11. Implementation details
tion, we use the pre-trained SciBERT tokenizer. The embedding layer is the output from the pre-trained SciBERT model. We include a dropout layer with a dropout probability of 0.5 to reduce overfitting. Learning rate is set to 1e-5 and we train all models for a total of 10 epochs. The output from the dropout layer is passed through a linear layer with input dimension same as the hidden dimension of SciBERT (768). For all Unsupervised Graph Ranker, we use the same hyperparameter settings as specified in their respective papers https://aclanthology.org/S15-1013. doi:10.18653/ distant supervision for low-resource named v1/S15- 1013. entity recognition, CoRR abs/2102.13129 [8] C. Florescu, C. Caragea, PositionRank: An unsuper- (2021). URL: https://arxiv.org/abs/2102.13129. vised approach to keyphrase extraction from schol- arXiv:2102.13129. arly documents, in: Proceedings of the 55th An- [20] F. Nooralahzadeh, J. T. Lønning, L. Øvrelid, nual Meeting of the Association for Computational Reinforcement-based denoising of distantly superLinguistics (Volume 1: Long Papers), Association vised NER with partial annotation, in: Proceedings for Computational Linguistics, Vancouver, Canada, of the 2nd Workshop on Deep Learning Approaches 2017, pp. 1105–1115. URL: https://aclanthology.org/ for Low-Resource NLP (DeepLo 2019), Association P17-1102. doi:10.18653/v1/P17- 1102. for Computational Linguistics, Hong Kong, China, [9] F. Boudin, Unsupervised keyphrase extraction 2019, pp. 225–233. URL: https://aclanthology.org/ with multipartite graphs, in: Proceedings of the D19-6125. doi:10.18653/v1/D19- 6125. 2018 Conference of the North American Chapter [21] Y. Yang, W. Chen, Z. Li, Z. He, M. Zhang, Disof the Association for Computational Linguistics: tantly supervised NER with partial annotation Human Language Technologies, Volume 2 (Short learning and reinforcement learning, in: ProceedPapers), Association for Computational Linguistics, ings of the 27th International Conference on ComNew Orleans, Louisiana, 2018, pp. 667–672. URL: putational Linguistics, Association for Computahttps://aclanthology.org/N18-2105. doi:10.18653/ tional Linguistics, Santa Fe, New Mexico, USA, v1/N18- 2105. 2018, pp. 2159–2169. URL: https://aclanthology.org/ [10] V. Kocaman, D. Talby, Biomedical named entity C18-1183.
recognition at scale, CoRR abs/2011.06315 [22] Y. Lin, Z. Liu, M. Sun, Y. Liu, X. Zhu, Learning en(2020). URL: https://arxiv.org/abs/2011.06315. tity and relation embeddings for knowledge graph arXiv:2011.06315. completion, in: AAAI, 2015. [11] S. Mesbah, C. Lofi, M. V. Torre, A. Bozzon, G.-J. [23] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, Houben, Tse-ner: An iterative approach for long- M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the tail entity extraction in scientific publications, in: limits of transfer learning with a unified text-toInternational Semantic Web Conference, Springer, text transformer, Journal of Machine Learning Re2018, pp. 127–143. search 21 (2020) 1–67. URL: http://jmlr.org/papers/ [12] Q. Liu, P. cheng Li, W. Lu, Q. Cheng, Long-tail v21/20-074.html.
dataset entity recognition based on data augmenta- [24] Z. Yang, Z. Dai, Y. Yang, J. G. Carbonell, R. Salakhuttion, in: EEKE@JCDL, 2020. dinov, Q. V. Le, Xlnet: Generalized autoregres[13] J. Laferty, A. McCallum, F. Pereira, Conditional sive pretraining for language understanding, in: random fields: Probabilistic models for segmenting NeurIPS, 2019.
and labeling sequence data, in: ICML, 2001. [25] I. Beltagy, K. Lo, A. Cohan, Scibert: A pretrained [14] H. L. Chieu, H. Ng, Named entity recognition with language model for scientific text, arXiv preprint a maximum entropy approach, in: CoNLL, 2003. arXiv:1903.10676 (2019). URL: https://www.aclweb. [15] J. Li, A. Sun, J. Han, C. Li, A survey on deep learning org/anthology/D19-1371/.
for named entity recognition, ArXiv abs/1812.09449 [26] X. Wan, J. Xiao, Single document keyphrase ex(2018). traction using neighborhood knowledge, in: AAAI, [16] S. Goldberg, D. Z. Wang, C. Grant, A probabilisti- 2008.
cally integrated system for crowd-assisted text la- [27] S. Ganguly, V. Pudi, Competing algorithm detection beling and extraction, J. Data and Information Qual- from research papers, in: Proceedings of the 3rd ity 8 (2017). URL: https://doi.org/10.1145/3012003. IKDD Conference on Data Science, 2016, CODS ’16, doi:10.1145/3012003. Association for Computing Machinery, New York, [17] X. Wang, Y. Guan, Y. Zhang, Q. Li, J. Han, Pattern- NY, USA, 2016. doi:10.1145/2888451.2888473. enhanced named entity recognition with distant supervision, in: 2020 IEEE International Conference on Big Data (Big Data), 2020, pp. 818–827.
doi:10.1109/BigData50022.2020.9378052. [18] C. Liang, Y. Yu, H. Jiang, S. Er, R. Wang, T. Zhao,
C. Zhang, BOND: bert-assisted open-domain named entity recognition with distant supervision, CoRR abs/2006.15509 (2020). URL: https://arxiv.org/ abs/2006.15509. arXiv:2006.15509. [19] M. A. Hedderich, L. Lange, D. Klakow, ANEA: