Publication databases rely on accurate metadata extraction from diverse web sources, yet variations in web layouts and data formats present challenges for metadata providers. This paper introduces CRAWLDoc, a new method for contextual ranking of linked web documents. Starting with a publication's URL, such as a digital object identifier, CRAWLDoc retrieves the landing page and all linked web resources, including PDFs, ORCID profiles, and supplementary materials. It embeds these resources, along with anchor texts and the URLs, into a unified representation. For evaluating CRAWLDoc, we have created a new, manually labeled dataset of 600 publications from six top publishers in computer science. Our method CRAWLDoc demonstrates a robust and layout-independent ranking of relevant documents across publishers and data formats. It lays the foundation for improved metadata extraction from web documents with various layouts and formats. Our source code and dataset can be accessed at https://github.com/FKarl/CRAWLDoc.
Databases such as Web of Science, Crossref, and DBLP are crucial academic resources of bibliographic
information. Identifying high-quality metadata sources about new publications is essential for these
services. While there are methods and tools for extracting bibliographic metadata [
We consider the example of DBLP, the de facto main metadata provider in computer science. The
main strategy for integrating publisher-provided metadata is to implement publisher-specific wrappers,
an approach that is time-consuming and requires maintenance whenever the publisher changes its
website [
We propose a novel retrieval system CRAWLDoc (Contextual RAnking of Web-Linked Documents),
see Figure 1, that can automatically identify relevant data sources from diverse web sources. Input is a
Digital Object Identifier (DOI) of a publication, which is provided by publishers [
Neural Document Ranking
documents along with their associated anchor texts and URLs into a shared vector space and treat the
publication’s landing page as the query. A ranking is computed by the similarity between the landing
page embedding and the embeddings of linked documents, efectively identifying the most relevant
sources for metadata extraction. By embedding the content, we efectively address the challenge that
the web sources from which the data is extracted are highly diverse and vary in structure and format [
We evaluate CRAWLDoc on a new dataset derived from DBLP, which comprises 600 publications from the six largest computer science publishers. Our dataset is unique as it provides annotated relevancy labels for all outgoing links from publication landing pages, along with bibliographic metadata, including titles, years, authors’ names, and afiliations manually. Our experiments show that CRAWLDoc reliably identifies relevant web documents based on a single seed document.
A leave-one-out experiment shows that our system is robust w. r. t. the retrieval from websites of various layouts from publishers that were excluded from the training dataset. In summary, our contributions are: • A document-as-query approach CRAWLDoc to determine relevant documents that encodes web content of various formats, anchor text, and URIs in a single embedding space. • Evaluating 600 publications from the six largest publishers in computer science. • A robustness check by training on five publishers and testing on a held-out publisher. • A new dataset of bibliographic metadata with author afiliations, along with relevancy information for linked web documents.
Below, we summarize related work. We introduce our CRAWLDoc metadata retrieval system in Section 3. The experimental apparatus is described in Section 4. The results are described in Section 5 and discussed in Section 6, before we conclude and outline future work.
We discuss research in neural information retrieval and layout-aware language models. Neural
Information Retrieval (NIR) is a prominent research area, utilizing neural networks to improve the
retrieval process. The landscape of NIR research has been extensively surveyed [
Publish with us
Help View PDF TeX Source BibTeX Related Papers Keywords < prev | next > identifying document
Layout-infused language models consider both textual content and spatial layout.
LayoutLMv3 [
We introduce CRAWLDoc (Contextual RAnking of Web-Linked Documents), a novel system for identifying relevant bibliographic sources across web documents. Based on a seed URI, a DOI of a publication, CRAWLDoc scrapes linked resources. Subsequently, the retrieved web documents in the form of H TML or PDF are ranked using a Small Language Model (SLM) [22]. Our primary assumption is that all necessary information can be found within a one-hop crawl of the landing page associated with the DOI. This assumption is based on our observation that publishers present key bibliographic information on the landing page or pages directly linked to it e. g., the PDF of the publication.
Web Scraping from Seed URI The initial step of our system involves web scraping, starting with a DOI as the input and progressing to the scraping of the corresponding web page. After this starting point, all documents linked from the seed URI are retrieved, which may be formatted in H TML or PDF. Both PDF and H TML files undergo a series of steps to extract the relevant text and its associated bounding boxes to also capture layout information. For PDF documents, the text and its corresponding bounding box coordinates are directly extracted from the file using the PDFMiner Python library. In the case of H TML documents, the page is first rendered in a Firefox web browser (Version: 129.0.2) to accurately present the content’s formatting and layout, and then the text and bounding boxes are extracted. This information is then converted into a uniform textual JSON format and serves as the input for the neural document ranking. Figure 2 illustrates the diferent steps to create our document representation.
Neural Document Ranking In the second step, we employ a SLM to create embeddings of the documents along with their associated anchor texts and URLs. For each document, we construct a single input representation by concatenating the anchor text, URL, and document content using a special separator token ([SEP]). This representation is then embedded into a dense vector space. The document originating from the DOI is embedded utilizing a query encoder, and all documents linked from the landing page are embedded with the document encoder. A Maximum Inner Product Search (MIPS) is performed with the embedding of the landing page and the embeddings of all scraped documents to create a Contextual RAnking of Web-Linked Documents (CRAWLDoc) based on the landing page.
We use the jina-embeddings-v2 model [23] as neural retriever. It is based on a BERT [
Dataset We take a subset of bibliographies from the six largest publishers in the DBLP Computer Science Bibliography dataset [26]. The publishers represent more than 80% of all publications listed in DBLP. This ensures the dataset contains a representative set of layouts encountered in bibliographic web content. We randomly select 100 publications for each publisher and split them into training, validation, and test sets in an 80/10/10 ratio with equal per-publisher distribution.
We obtained the metadata for each publication by manually retrieving the title, publication year, and authors’ names and afiliations. We retrieved the landing page of each publication and labeled every outgoing link on the landing page with a binary relevancy label. This label indicates whether the linked website or document is about the same publication . By manually creating this dataset, we ensure a high quality of the metadata and can accurately assess the document retrieval process in our proposed setup.
To prevent artificial inflation of our performance metrics, we identified and removed any instances in our test set where the landing page contained links to itself. The trivial nature of calculating document similarity to itself would otherwise result in an unrepresentative boost in ranking performance.
Our dataset consists of 600 publications with detailed metadata and 72,483 linked documents with binary relevancy labels. Per publication, we have an average of 3.63 (SD: 2.10) authors, with an average of 1.14 (SD: 0.41) afiliations per author. Furthermore, there is an average of 120.81 (SD: 76.52) linked websites per landing page and an average of only 5.45 (SD: 2.99) relevant websites per publication. To the best of our knowledge, we are the first to release a dataset that includes author afiliations as mentioned in the publications. Additionally, we are the first to provide relevancy labels for linked documents in the context of publication web data. For legal purposes, we are only able to publish the labels and not the actual websites. However, we do ofer the source code for our procedure.
The DBLP dataset is released under CC0 1.0 Public Domain Dedication license. Our annotations have the same license.
Procedure Our experimental procedure for document ranking involves fine-tuning a neural document retriever using contrastive loss to improve document ranking. To ensure robust performance, we evaluate the ranking capabilities on both in-distribution and out-of-distribution data. We optimize the hyperparameters of the neural document retriever resulting in a learning rate of 3e-05, 32 accumulation steps, and patience of 5, resulting in 16 epochs.
Metrics To evaluate the ranking of the web documents, we employ several metrics. The Mean Reciprocal Rank (MRR) evaluates the efectiveness of a retrieval system by considering the rank position of the first relevant result. The MRR focuses on the first relevant document in the ranked list, i. e., it favors a relevant document in the highest position. In contrast, Mean Average Precision (MAP) evaluates the precision of a retrieval system by averaging the precision scores at all ranks where relevant documents are found and then averaging these scores over all queries. Normalized Discounted Cumulative Gain (nDCG) [27] measures the usefulness of a document based on its position in the result list, assuming that highly relevant documents are more useful when appearing earlier. We further calculate the precision@k, recall@k, and F1@k which measure the proportion of relevant items in the top results. .
7 0.344 0.972 0.510
2
The ranking metrics for identifying relevant linked documents are shown in Table 1. Overall, we achieve a very high average ranking performance with MRR of 0.967, MAP of 0.987, and nDCG of 0.961. The MRR, MAP, and nDCG values exhibit a consistently high level of performance for all six publishers, except for MRR and nDCG on the Springer dataset. The MRR for IEEE, Elsevier, ACM, arXiv, and MDPI all achieve the maximum score of 1.000, indicating that a relevant document is always placed at the top position. To understand the impact of layout information on ranking performance, we conducted an ablation study. The results without layout information showed slightly lower performance with a MRR of 0.950, MAP of 0.976, and nDCG of 0.952.
We have conducted a more detailed examination of the ranking performance with diferent cut-of values presented in Table 2. The recall increases with increasing values of , reaching 0.951 at = 10. Precision declines from 0.972 for = 1 to 0.416 for = 10. The F1@k score, which combines precision and recall, reaches its highest value of 0.772 for = 4 and = 5.
We have evaluated robustness using a leave-one-out strategy, training on all but one publisher and testing on the left-out publisher. The results of the robustness analysis are shown in Table 3. We obtain a high performance across all publishers, with an average MRR of 0.959, MAP of 0.968, and nDCG of 0.961. This is less than one point for MRR and nDCG and less than two points for MAP compared to using the full training dataset shown in Table 1. The results were particularly strong for IEEE and arXiv, both achieving the maximum score of 1.000 for all three metrics. However, the performance was slightly lower for Springer, consistent with the result on the full training set.
Document Ranking Our system shows impressive overall ranking performance, with documents ranked at the top being mostly relevant. This trend is seen when examining the evaluation of ranking performance at various cutof values. We notice a sharp rise in recall@ for the first few documents but only minor enhancements after around five documents. The decline in precision@ as values increase is a natural result considering that a publication has on average 5.45 relevant documents (see Section 4). This is also reflected in the F1@ score, which is peaking at = 4 and = 5. Overall, the results show that CRAWLDoc maintains a good balance between precision and recall with a cut-of value of = 5.
Upon investigating cases where the model ranked irrelevant documents higher than relevant ones, we
could not identify a general error pattern. The errors we observed were predominantly paper-specific
rather than systematic. For example, errors occurred when the model ranked links from the references
section of a paper highly or when it assigned high ranks to diferent chapters of the same book. In
particular, Springer publications presented more special cases than other publishers in our dataset.
Robustness of Document Ranking Our model demonstrates strong robustness across diferent
publishers. While previous research, such as Chen et al. [
Generalization and Threat to Validity The generalizability of our work refers to diferent publishers based on a leave-one-out test (Table 3) in which we test the system for publishers on which it has not been trained. Our robustness check demonstrates that a trained model can achieve comparably good results on out-of-distribution data within our tested scope. This finding suggests that our model has learned generalizable features of document relevance beyond the specific layouts and publishers in our training data. Our approach of transforming diferent document formats (HTML and PDF) into a uniform textual representation enhances its potential for generalization. This uniformity in representation suggests applicability to other web and document-related tasks.
While our study provides robust results, it is important to reflect on potential threats to validity. One such threat is the limited scope of our investigation, which focuses on only six publishers, primarily from the computer science field. However, the threat is reduced as these publishers represent more than 80% of computer science publications and provide various formats. Nevertheless, we acknowledge that true generalizability to the remaining 20% of publications, which may exhibit greater variability in their document layouts and metadata presentation, remains to be thoroughly tested in future work. An additional possible risk is the presence of recency bias in our dataset, given that most publications in DBLP are from more recent years. Nevertheless, we have found that older publications, including papers as far back as 1967, in our test set achieve similar performance to more recent ones, which eases this concern. This indicates that the performance of our model is not much influenced by the publication year.
Although there is no particular reason why other embedding models could not be used, our work does not focus on finding optimal embedding models for the retrieval tasks. We use Jina embeddings because they are widely used and have demonstrated strong results [23].
Our Contextual RAnking of Web-Linked Documents (CRAWLDoc) retrieval system efectively identifies relevant bibliographic documents across diverse web sources. The key scientific findings include robustly identifying pertinent web documents and the system’s consistent performance across publishers with varying web layouts. The insights presented in this study can potentially advance the management and enrichment of comprehensive bibliographic databases.
Although our model’s performance is already very strong, rerankers [
We thank Florian Reitz from DBLP for valuable feedback. The authors acknowledge support by the state of Baden-Württemberg through bwHPC. This research is co-funded by the SmartER project (No. 515537520) of the DFG, German Research Foundation.
During the preparation of this work, the authors used Writefull and Grammarly to check spelling and grammar. They also used Writefull and Claude 3.5 Sonnet to improve the writing style. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. ACL 2023, Toronto, Canada, July 9-14, 2023, Association for Computational Linguistics, 2023, pp. 13345–13360. URL: https://doi.org/10.18653/v1/2023.nfidings-acl.844. doi: 10.18653/V1/2023.
FINDINGS-ACL.844. [22] Z. Lu, X. Li, D. Cai, R. Yi, F. Liu, X. Zhang, N. D. Lane, M. Xu, Small language models: Survey, measurements, and insights, CoRR abs/2409.15790 (2024). URL: https://doi.org/10.48550/arXiv. 2409.15790. doi:10.48550/ARXIV.2409.15790. arXiv:2409.15790. [23] M. Günther, J. Ong, I. Mohr, A. Abdessalem, T. Abel, M. K. Akram, S. Guzman, G. Mastrapas, S. Sturua, B. Wang, M. Werk, N. Wang, H. Xiao, Jina embeddings 2: 8192-token general-purpose text embeddings for long documents, CoRR abs/2310.19923 (2023). URL: https://doi.org/10.48550/ arXiv.2310.19923. doi:10.48550/ARXIV.2310.19923. arXiv:2310.19923. [24] O. Press, N. A. Smith, M. Lewis, Train short, test long: Attention with linear biases enables input length extrapolation, in: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net, 2022. URL: https://openreview.net/forum? id=R8sQPpGCv0. [25] A. van den Oord, Y. Li, O. Vinyals, Representation learning with contrastive predictive coding,
CoRR abs/1807.03748 (2018). URL: http://arxiv.org/abs/1807.03748. arXiv:1807.03748. [26] dblp Team, dblp computer science bibliography – Monthly Snapshot XML Release of April 2024, 2024. URL: https://doi.org/10.4230/dblp.xml.2024-04-01. doi:10.4230/dblp.xml.2024-04-01. [27] K. Järvelin, J. Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Trans. Inf.
Syst. 20 (2002) 422–446. URL: http://doi.acm.org/10.1145/582415.582418. doi:10.1145/582415. 582418.