Logic Mill is a scalable and openly accessible software system that identifies semantically similar documents within either one domain-specific corpus or multi-domain corpora. It uses advanced Natural Language Processing (NLP) techniques to generate numerical representations of documents. It leverages a large pre-trained language model to generate these document representations. The system focuses on scientific publications and patent documents and contains more than 200 million documents. It is easily accessible via a simple Application Programming Interface (API) or via a web interface. Moreover, it is continuously being updated and can be extended to text corpora from other domains. We see this system as a general-purpose tool for future research applications in the social sciences and other domains.
navigation system, since it facilitates the tracing of
knowledge elements within and across text corpora (e.g., the
There is a growing need for tools that allow researchers corpus of all patents or all scientific publications).
Previto identify related documents within the same but also ous attempts include [5] for scientific publications and
across diferent domains. With the ever-growing vol- [6] for patent documents, both of which make use of
bagume of scientific publications and patents, scholars find of-words approaches. While such systems are sometimes
it burdensome to manage relevant documents and search referred to as recommender systems [7],
recommendafor important prior contributions eficiently. Finding tion of related documents is only one possible application
relevant documents plays a significant role in building of knowledge navigation (see below for a non-exhaustive
coherent scientific arguments but is also important in list of use cases). However, these systems often use
proassessing the use of scientific research outside academia prietary algorithms, usually focus on one domain
cor[
In ex post analyses, researchers often rely on citation trieving, storing, and processing hundreds of millions data to identify relations between documents. While ci- of documents. Moreover, it requires capabilities for fast tations are helpful in tracing citation networks and in detection of particularly similar documents within and uncovering important patterns in the production and dif- across corpora. fusion of knowledge within the same corpus, they are An important problem that needs to be addressed is typically limited when searching for relations across dif- how to implement the concept of document similarity. ferent corpora. Even within the same corpus, citations This requires representing documents in a numerical can be selective or even systematically biased (see [3, 4]). form that computers can process, with the goal of genFinally, references may not exist for texts that are in the erating similar representations for similar documents at process of being created, so authors are faced with the a large scale. In the field of natural language processing challenge of identifying relevant references in the first (NLP), this process is known as document encoding. place. Therefore, tools are needed that allow for process- There are various methods for representing documents ing and analyzing the textual contents of high-volume numerically. Traditional NLP approaches like TF-IDF [8] text corpora and establishing measures of relatedness are used widely in the literature. However, these tradi(similarity) between them. tional methods are not scalable, since an extension of We refer to the system described here as a knowledge the vocabulary requires a re-computation of all representations of the corpus. In addition, these approaches do not capture the semantics of the documents; that is, the meaning of words or the interpretation of sentences in context is lost.
Therefore, we propose Logic Mill, a software system aiming to satisfy the shortcomings of existing systems and approaches. The system creates document repre2. Document Encoding sentations using modern NLP techniques and contains large document sets with pre-calculated encodings. It is easy to use and allows users to access and compare texts Document encoding in natural language processing from diferent text corpora. Furthermore, it is scalable (NLP) is a process for representing textual data in a nuand built on non-proprietary algorithms. We regularly merical form. There are various approaches to encoding update the datasets based on their release schedule. Our documents. objective is to provide a fast system of high accuracy that is openly accessible. Bag of words Simple and fast procedures construct
In the current version, the system encodes text doc- a vector whose binary elements indicate the presence
uments using the Pat SPECTER document encoder [
At present, we provide the numerical representations Furthermore, they are not scalable since the whole model for the scientific articles of OpenAlex [ 11] and for the must be retrained if a new word is added to the vocabpatent publications provided in DocDB database, main- ulary and the length of each vector equals with the votained by the European Patent Ofice. To be precise, we cabulary size. From a computational perspective, they use the titles and abstracts of these, since BERT-based are ineficient since they generate sparse matrices where models can process only 512 tokens at a time. most elements equal 0.
An important feature is that users can also feed their own text data to Logic Mill for encoding and obtaining similarity measures, within their data and between their data and the system-provided documents. They can thus link their own curated documents to patents and scientific publications according to textual similarity.
Logic Mill can be used in a number of diferent research applications, such as: Word embeddings Word embeddings are dense, ifxed-size, and continuous-valued vector representations of words that capture the meaning of the words in the document. These word representations are learned via training over large corpora of textual data using methods such as Word2Vec [12], GloVe [13], or FastText [14]. The advent of these word embedding methods was a leap towards memory-eficient dense numerical representation for words and documents from the sparse representation of bag of words models. A baseline approach to representing a document is to average or sum the learned word embeddings of the words in a document. • Explore literature: Search for research papers, and find the best matches based on textual similarity to a paper in the database or to own text documents. • Prior art search in patent examination: Look for previously granted patents or (not yet granted) Sentence/Paragraph Embeddings Sentence embedpatent applications that are similar to the focal dings can be understood as an extension of the basic one. idea of word embeddings. Word embeddings are static • Link patents to related scientific publications: representations of words that do not change even when Search for patents that the scientific publication multiple contexts are present in the document collection. might be based on or have a strong similarity to. However, the same word can have diferent meanings in • Recommend citations and readings for new docu- diferent contexts. For example, the word "bank" can be a ments: Find documents that are very similar to ifnancial institute or can relate to a river bank. Hence, the a focal one and may be useful as a reference or neural network architecture such as Recurrent Neural reading. Networks (RNNs) and Long Short-Term Memory (LSTM) • Assess the novelty of patents and publications: [15] are used for representing the word to its true context Check if a patent or publication is new or not dynamically with a notion that words appearing either by comparing it to prior texts. Documents that before or after the focal word reveal the context around have few highly similar documents may be new the focal word. RNN and LSTM also aid the vector repor even unique. resentation of the context of the sentences or paragraph • Trace concepts across domains and over time: [16].
Identify documents across domains (e.g. publi- However, RNN and LSTM architectures cannot approcations and patents) that are highly similar and priately capture the meaning of the words at the beginpossibly related. ning of a very long sentence or paragraph in its numerical representation due to their sequential structure. The rise lar embeddings for related scientific documents without of deep neural networks and recent advancements in the knowing citation information. SPECTER was used in the ifeld of NLP introduced the transformer architecture [ 17]. initial version of Logic Mill.
Transformers look at each word of a sentence together,
unlike RNN and LSTM, and learn the degree to which Pat SPECTER Pat SPECTER is a language model to
the words reflect the context of that sentence using the encode scientific publications and patents at the same
so-called attention mechanism. The transformer architec- time [
Pat SPECTER replaces SPECTER as our workhorse
document encoder model.1
BERT Language Model Bidirectional Encoder
Representations from Transformer, BERT [
involving scientific publications [ 18]. It was trained on a large corpus of 1.14M scientific publications from computer science and the broad biomedical field. This model also outputs numerical representation like BERT.
Fine-tuning a general purpose BERT model on
scientific papers produces more accurate results in this domain
[
code scientific publications also with the help of interdocument relatedness [21]. In the scientific literature, citations signal relatedness, but this information is not used by SciBERT. SPECTER transfers the learned relatedness signal to the representation of a scientific article. During the application of this model, it generates simi
Logic Mill system is designed using a Microservice Architecture. This software design approach breaks down a large, monolithic application into smaller, independent components that can be developed, deployed, and maintained separately. Each microservice is a self-contained unit that performs a specific function and communicates with other microservices through well-defined interfaces, typically using APIs.
Figure 1 shows the multiple services of our system. From a high-level perspective, there are 6 distinct parts: a) External Data Sources: where patent documents and scientific publications are obtained from b) Document Encoder: transforms the text of the document into a numerical representation using a machine learning model c) Extract, Transform, Load (ETL): processes move documents from the external sources, process and store them d) Vector Search Database: stores the computed numerical representation along with metadata and the text e) Backend with Web API : propagate the user requests to the database f) User-Interface: provides the users with a web application where they can interact with the API via our website or using their own scripts, for example in Python or R.
on patents, namely PatentBERT [22]. Its task is to classify patents,
and therefore, it is not suitable for our purpose.
External Data Sources The system can be connected [
Initially, Logic Mill relied on publicly available patent On the one hand, we used desktop workstations with APIs from the EPO and the USPTO. For scientific publi- Nvidia graphics processing units (GPUs). For large bulk cations, we relied on Semantic Scholar [23]. sets, we made use of high-performance cloud computing
However, we decided to change the external data facilities. To allow for real-time inference, a CPU misources to streamline the processes and align with the croservice was deployed in the cloud. It is connected to preferences of the scientific community. the system and accessible to end-users via the GraphQL
Scientific publications are now obtained from the Ope- API. nAlex 2[11] database, including title, abstracts, and additional metadata such as publication date, journal name, Storage & Search For the search and storage of docuor Digital Object Identifier (DOI). ments with their numerical representation, ElasticSearch
We obtain patent documents from the DocDB3, the is used. This database is capable of full-text search and EPO’s master documentation database. This database can be used in a distributed context, which is essential has worldwide coverage and contains bibliographic data, for scalability reasons. abstracts, and citations. Furthermore, it also maintains Furthermore, ElasticSearch allows storing dense vecthe DOCDB simple patent family. tors that nearest-neighbor search algorithms can use. Exact Nearest Neighbor searchers are guaranteed to find the Extract, Transform, Load The system obtains the best solution but are ineficient and not scalable. Thereraw documents from external data sources and processes fore, we use the approximate nearest neighbor searches them through various Extract, Transform, and Load (ETL) (ANN), which trade-of precision for lower computational processes. In the first step, the textual content and the and resource burdens. ElasticSearch uses the Hierarchical metadata are extracted and stored in a global data struc- Navigable Small World graphs (HNSW) [24] algorithm ture. The structure is independent of the document type (as of version 8.0).4 HNSW organizes vectors as a graph (patent or scientific publication). In the second step, the based on their similarity to each other. Together, this document encoder encodes the text parts of the document setup finds the most similar documents with a very high and generates the numerical representation. Finally, the probability for any query document within milliseconds. search database stores the numerical representation of a In our current setting, the cluster consists of 12 nodes document along with the metadata and the text. with 8 vCPU cores, 128 GB of RAM, and 1 TB of SSD storage each. This setting is needed to allow for fast Document Encoder The document encoder is a and eficient computation, because the RAM required machine-learning model that transforms text documents by the ElasticSearch database can be distributed over into a numerical representation. We use Pat SPECTER multiple nodes. The database cluster as well as all other 2See https://openalex.org/ 3See https://www.epo.org/en/searching-for-patents/data/bulk-dat a-sets/docdb
various distance measures, HNSW performs consistently well [25]; see also http://ann-benchmarks.com. components are running on the GWDG OpenStack Cloud encoder model is already available, and the embeddings IT infrastructure5 in Göttingen, Germany. of large corpora are pre-computed and ready to be used. Users can do their projects in less time with fewer reComputation & API The back-end extends the soft- sources, since there is no need to download the raw data ware stack for more functionality and to handle user in- (time, storage), set up the machine learning pipeline (time, teractions. It is written in the language Go and provides a CPU/GPU, memory), encode the documents (time), and plug-and-play Application Programming Interface (API) search through results (time, CPU/GPU, memory, interfor end-users using GraphQL. This query language for nal storage).
APIs is a strongly typed interface that provides complete and understandable API documentation. Furthermore, it User interfaces Upon registration on the website, allows users to retrieve the data precisely that they have users can access the system either through the web appliasked for. cation on https://logic-mill.net/ (Fig. 2) or the Application
The back-end also connects the document encoder Programming Interface (API). with the client-facing run-time environment. This en- The web application aims to help users familiarize sures that end-users can upload texts, which can then themselves with the queries and data structure. It exbe encoded to numerical representations. Finally, the plains the diferent functionalities of the system and representation can be used to query similar documents interactively shows the syntactically correct GraphQL from the database. queries. Thus, users can experiment, design, and adapt
It can also be used to calculate distances between texts their queries. provided by the user using their numerical representation The web app auto-generates these queries for various and distance metrics like the Cosine Similarity (Eq. (1)), programming languages and tools. The release version the Manhattan (L1) Distance (Eq. (2)) or the Euclidean of Logic Mill provides these examples in GraphQL for (L2) Distance (Eq. (3)). Curl, Python, R, and Go. However, any programming language with the ability to make HTTP requests and retrieve and process JSON responses can interact with cos(a, b) = ab the API endpoint.
API Functionality Logic Mill provides nine API endpoints for retrieval, pairwise similarity metric computa(2) tion, and Nearest Neighbor search. Each functionality can be extended to multi-domain corpora (i.e., searching the most similar scientific publications for a patent), and they can involve available documents in the database as (3) well as user-supplied documents or texts. The endpoints
are summarized in Table 1.
rizes the end-users. This is ensured by JSON Web Tokens Table 1 (JWT), which are sent in the HTTP header of each re- Overview of Logic Mill’s API functionality. quest.
user registration, project presentation, and documentation in a Single Page web Application (SPA). It uses the Vue JavaScript framework6 and consumes the GraphQL API provided by the backend.
The general idea behind the user interface is to enable a simple and easy plug-and-play interaction that simplifies the project setup. The machinery to use the document
gen. See https://www.gwdg.de/. 6See https://vuejs.org.
Retrieval Calculation NN Search
Purely database with own documents Document, Documents encodeDocument, encodeDocuments searchDocuments similarityCalculation SimilaritySearch encodeDocumentAndSimilarityCalculation embedDocumentAndSimilaritySearch
existing patent documents or scientific publications can be retrieved via the Document and Documents endpoints. Users provide the IDs and specify the database as well as the desired information such as title, IDs, or embedding, which are detailed in the online documentation.
The returned information depends on the document type and data source, and copyright restrictions may apply. Document Search The searchDocuments endpoint encodeDocumentAndSimilarityCalculation endallows keyword-based searches. Logic Mill retrieves doc- point. The user provides, for instance, title and abstract uments matching the specified keywords and metadata of the diferent documents, identifiers for later reference, from the available corpora. The returned information and the type of distance calculation metric (cosine, l1, is consistent with that obtained through the Document l2). The system encodes the documents with the help and Documents endpoints. of the document encoder and uses provided metric to compute the distances among the encoded documents. A Encode own documents The embeddings for users’ typical use case includes the representation as a similarity curated documents can be generated and retrieved us- matrix. ing the encodeDocument and encodeDocuments endpoints of our API. Users provide a title and an abstract, 4.3. Nearest Neighbor Search and the document encoder model returns the corresponding numerical representation.
Database Document Similarity Search Given a known document, users can search for the approximate nearest neighbor within the same or other cor4.2. Calculation pora. This is based on cosine similarity using the Users interested in pairwise similarities between docu- SimilaritySearch query. For example, users can rements have two options. They can retrieve embeddings quest the five most similar scientific publications for a for a set of documents one-by-one and compute simi- specific patent. The results include the title, similarity larity metrics themselves, or the rely on functionality score, and ID of each document within its respective provided by the system. index.
Calculate Document Similarities The Own Document Similarity Search Users can also similarityCalculation endpoint retrieves the provide their own documents and search for simsimilarity matrix for multiple documents in the database, ilar ones of OpenAlex, DocDB, that have already enabling their comparison. Input consists of a list been encoded and stored in the database using the of source and target documents (identified by their embedDocumentAndSimilaritySearch query. identifiers and indices), along with the desired distance calculation metric (cosine, l1, l2).
Calculate Similarities with Own Documents To To thoroughly evaluate the performance of our system, retrieve the similarities between a set of own curated we conducted a series of experiments focusing on patentdocuments and documents in the database, users use the to-patent and patent-to-publication citations. This evaluation is analogous to the prior art search by patent ofices.
Prior art search is a crucial task in the patent examination process. It is designed to uncover existing patents, publications, and other relevant articles that might challenge the novelty of the patent application under prosecution.
Our experiments simulate the prior art search process, testing the system’s ability to identify relevant prior art accurately and ensuring its practical applicability and robustness. This approach not only validates the system’s quality but also aligns it with industry-standard practices, thereby enhancing its reliability and utility for examiners and inventors who conduct comprehensive prior art searches to make informed decisions and protect their innovations.
The first task is to identify patent-to-patent citations for 10k randomly selected patents granted by the EPO whose family consists of at least 10 documents. The goal is to predict the patent citations to prior patents on a patent family level. We then search our DocDB index for 1,000 approximated nearest neighbors present at the time of the earliest filing date and retrieve the closest patent documents from the EPO, the USPTO, and the WIPO. The approximate nearest neighbor results are then aggregated on a patent family basis and re-ranked.
Of 10,000 test cases, we were able to identify at least one correct family citation within the first 1,000 nearest neighbors in 8,641 cases (≈ 86%). In 6,912 cases, at least one family citation is among the 100 nearest neighbors (≈ 69%). Indeed , most of the first family citations are within the first 20 results (cf. Figure 3).
Table 2 shows the Mean Reciprocal Rank (MRR) and the Mean Average Precision (MAP) for various cutof values ∈ {5, 10, 20, 50, 100}. For = 5, the MRR is equal to 0.264 and the MAP equals 0.06.
cations that a patent should cite. To establish a ground
truth for these citations, we leverage the Reliance on
Science dataset [
There are instances where we are not able to obtain an embedding. A common reason is the absence of title and (English) abstract. Another reason is that IDs present in the Reliance on Science dataset have changed in more recent versions of OpenAlex.
Of initially 10,000 randomly selected samples, we found at least one cited publication among the top 100 results in 3,740 cases (≈ 37%). As can be seen in Figure 4, the distribution is not as steep as in the patent-to-patent case. Table 3 presents the MRR and MAP for various ∈ {5, 10, 20, 50, 100}.
To understand the omission of a significant share of citations, we manually examined anecdotal cases where matches were not found within our approximate nearest neighbor (ANN) results. Our analysis reveals that many of these citations were applicant-added rather than examiner-cited. These applicant-added citations often appear in the patent description, a context not considered during model training, nor encoded in our system. Additionally, a lack of semantic similarity between the cited publication and the focal patent’s abstract makes associating the two challenging even for humans.
language model utilized, but the speed and ease-of-use. The system can identify the closest 100 to 10,000 documents based on an already encoded reference document within one second. Depending on the load and search parameters, results can be retrieved even faster, within 100 milliseconds. If encoding is required, the process
navigate knowledge embedded in scientific publications, patent documents, and other text corpora. The system is scalable and openly accessible. New documents are regularly ingested into the system.
Users can leverage the system, for example, in the following contexts: • retrieve numerical representations of existing documents in Logic Mill’s database • generate numerical representations for their own documents • calculate similarities between users’ given documents, or documents in the database, or between users’ given document and the one in the database • search for similar documents present in the database given a query document that either exists in the database or users can provide the query document
Despite some present shortcomings, Logic Mill can assist many user groups. Researchers interested in studying the economics of innovation, science of science, and knowledge flows may be particularly interested in these capabilities. Its search capabilities may also be of interest to patent examiners and inventors seeking prior art related to their ongoing inventions. Researchers in many fields may want to use Logic Mill as a literature and citation recommender system.
As Logic Mill is actually agnostic about the underlying document encoder model, the system may improve with the advent of better models. For instance, the next generations of deep learning language models may provide larger context windows so that Logic Mill can be able to encode a larger portion of documents. This might extend to models that are specifically trained to identify similarities between patents and their cited publications.
The impact of the approximate nearest neighbor (ANN) search on our patent data also needs to be investigated7. 6.0.1. Acknowledgements
ticipants at the 2022 Munich Summer Institute and the 2022 Summer School on Data and Algorithms for Science, Technology & Innovation Studies at KU Leuven. We also thank the 2023 I3 NBER workshop participants for their comments. We also thank the team at the computing and IT competence center of GWDG for their continuous support. Dietmar Harhof acknowledges support from Deutsche Forschungsgemeinschaft (CRC 190).
here: https://ann-benchmarks.com/ arXiv:2208.01171 [cs]. Once a query is constructed in GraphQL, it can be im[20] S. Althammer, M. Buckley, S. Hofstätter, A. Han- plemented and executed using any modern programming bury, Linguistically Informed Masking for Repre- language. sentation Learning in the Patent Domain, 2021.
URL: http://arxiv.org/abs/2106.05768, number: Basic structure of a request A basic request written arXiv:2106.05768 arXiv:2106.05768 [cs]. in Python is displayed in source code .9 The variable [21] A. Cohan, S. Feldman, I. Beltagy, D. Downey, called query defines the GraphQL query. User-specific D. Weld, SPECTER: Document-level Representa- variables and requested information. By providing tion Learning using Citation-informed Transform- the index and the ID (in this case an EPO patent with ers, in: Proceedings of the 58th Annual Meet- the number of EP19164094B1), the title of the docuing of the Association for Computational Linguis- ment and the vector (the numerical representation) of tics, Association for Computational Linguistics, the encoded text are requested. The return variables can Online, 2020, pp. 2270–2282. URL: https://ww be customized (lines 8-11) to include other information w.aclweb.org/anthology/2020.acl- main.207. about the document. The object called response (line doi:10.18653/v1/2020.acl-main.207. 19) is a dictionary that can be processed further. As one [22] J.-S. Lee, J. Hsiang, Patent classification by fine- can see, the Python code directly includes the GraphQL tuning BERT language model, World Patent Infor- query. mation 61 (2020) 101965. URL: https://linkinghub .elsevier.com/retrieve/pii/S0172219019300742. Example code showing the structure of any interdoi:10.1016/j.wpi.2020.101965. action with the web API in Python [23] S. Fricke, Semantic Scholar, Journal of the Medical
Library Association 106 (2018). URL: http://jmla.pit import requests t.edu/ojs/jmla/article/view/280. doi:10.5195/jm import json la.2018.280. TOKEN = ’XXX’ [24] Y. A. Malkov, D. A. Yashunin, Eficient and robust ENDPOINT = ’https://lm/api/endpoint/url/here’ approximate nearest neighbor search using Hier- hea’dceornste=nt{-type’: ’application/json’, archical Navigable Small World graphs, 2018. URL: ’Authorization’: ’Bearer ’ + TOKEN, http://arxiv.org/abs/1603.09320. doi:10.48550/a } rXiv.1603.09320, arXiv:1603.09320 [cs]. queDroyc=u"m"e"n{t(index: "docdb_cos", id: "EP19164094B1") { [25] M. Aumüller, E. Bernhardsson, A. Faithfull, ANN- documentParts {
Benchmarks: A benchmarking tool for approximate title nearest neighbor algorithms, Information Systems }vector 87 (2020) 101374. URL: https://linkinghub.elsevier. } com/retrieve/pii/S0306437918303685. doi:10.101 }""" 6/j.is.2019.02.006. r = requests.post(ENDPOINT, [26] M. Marx, A. Fuegi, Reliance on science by inven- headers=headers, tors: Hybrid extraction of in-text patent-to-article json={’query’: query}) citations, Journal of Economics & Management if r.status_code == 200: Strategy 31 (2022) 369–392. URL: https://online response = r.json() library.wiley.com/doi/abs/10.1111/jems.12455. else:print(response) doi:https://doi.org/10.1111/jems.12455. print(f"Error executing\n{query}\non {url}")
these JSON responses. Recent versions of Stata can, however, embed Python code.
A. Appendix straightforward and basic implementation, it is not always the most suitable in practice. In many cases, one Logic Mill provides an API endpoint that uses GraphQL. has to make multiple requests to retrieve all the data. The GraphQL query determines what should be executed For example, it is possible to request the encoding for and what information should be returned. Users need an multiple documents in one request. However, retrieving API key to access the API. more than 10,000 documents is impossible. To do this,
Logic Mill web app provides a user-interface where the code needs to include a loop that executes a query dynamic GraphQL queries are generated for cURL, Python, with diferent parameters with each iteration. We will R and Go. 8 call this a parameterized query. The user provides two
for http requests. parameters to interact with the web API. These are the Example code showing with a query with parameGraphQL query and the query parameters. Doing the ters loop with parameters will make the code more readable.
The code example A shows the example with the query and the variables object. The code for looping is omitted, as is the base code for handling the request. # (same setup as above) # Build GraphQL query query=""" query Documents($index: String!, $keyword: String!) {
Documents(index: $index, keyword: $keyword) { id documentParts {
title # Send request r = requests.post(ENDPOINT, headers=headers, json={ ’query’: query, ’variables’: variables # Build variables variables = [ {"keyword": "EP19164094B1", "index": "docdb_cos"}, {"keyword": "20130226771", "index": "docdb_cos"}