Several specific features such as the volume of data, document size, structure of the documents, jargon, the way information needs are defined, among others, are features that justify that retrieval models should not handle all information equally when it comes to domain-specific retrieval tasks (e.g., in law and healthcare). Neural IR models can deal with such features, especially those related to contextual elements (e.g., expert knowledge). The ongoing project will consider domain-specific embeddings to contextualize within a neural ranking setup document retrieval in domain-specific tasks.
Traditional information retrieval (IR) models are designed as generic tools applicable in diferent
retrieval tasks. However, several specific features justify that retrieval models should not handle
all information equally when it comes to domain-specific (DS) retrieval tasks (e.g., in law and
healthcare) [
Neural IR is a growing field in IR; neural networks (NN) represent a tool for learning text representations, defining ranking models capable of handling diferent kinds of documents, and even introducing additional elements in the retrieval process, such as DS features.
Some of those features can be considered as contextual elements, i.e., as factors influencing
how an IR system is used and how its performance should be accordingly evaluated. The concept
of context in IR has been extensively studied, giving rise to the so called contextual IR, which
aims at optimizing the retrieval performance by defining the search context and by taking it
into account in the information selection process and in assessing the search outcome [
In last years, there have been developments in representing DS corpora; e.g., Legal-BERT
[
The main aim of the ongoing project is to define DS embeddings that can serve the purpose of contextualizing search within a neural ranking setup in DS tasks. More specifically, our aim is to evaluate the efectiveness of pre-trained embeddings with respect to traditional lexical matching-based models, and to boost the performance of neural models by fine tuning the DS embeddings in a semantic-based retrieval scenario.
A generalized Learning to Rank problem is focused on finding the optimal ranking function,
according to existing neural IR models [
This section shortly introduces two examples of DS search tasks on which we are working.
Patient- clinical trials matching: clinical trials are experiments conducted in the development
of new medical treatments, drugs, or devices. Recruiting candidates for a trial motivates the
task of matching eligible patients () to clinical trials () [
Legal case retrieval: this task involves finding precedent cases ( ) that are relevant, i.e., could
support the decision concerning a given case () in the set of candidate cases [
Compared to traditional ad-hoc text retrieval, the above tasks are relatively more challenging since the query is much longer and more complex than common keyword-based queries. Besides that, the definition of relevance of a document to a query is beyond general topical relevance, and as such its assessment requires expert knowledge.
This section presents some strategies that have the potential to address DS search tasks and the building blocks for model design.
Several works have tried to learn text representations (Φ) that incorporate contextual elements
into the model by learning embeddings from text corpora that gather information potentially
useful to better represent the information need in a given context [
Given the representations of both a query and a document, one can match diferent parts of the query with diferent parts of the document and then aggregate values as partial evidence of relevance. Interactions-based approaches model this matching using an interaction matrix formed by sweeping both the query and the document representations with a sliding window that instantiates a function () for aggregating. Each instance of the window over the query interacts with each instance of the window over the document. This process could capture local interactions based on diferent matching levels (e.g., word-level and passage-level). Denfiing the input representations, the aggregation function, and the matching level afects the model’s performance; then, all those parameters must be defined according to the specific task and its features.
Finally, there are diferent kinds of neural networks that are especially designed for some purposes that could match some of the issues described above and could be used for building a ranking model ():
Pooling layers sweep a kernel (with no weights) across the entire input, similar to the convolutional layer. The kernel applies an aggregation function, which could either selects the element with the maximum value to send to the output array (Max-pooling), or computes the average or the sum (Average-pooling and sum-pooling, respectively) within the area covered by the kernel while it moves across the input, leading to diferent architectures. Intuitively, pooling layers conduct dimensionality reduction, reducing the number of parameters in the input. This operation is useful to reduce complexity, improve eficiency, and limit the risk of overfitting.
Encoders process the input and compress the information into a context vector (also known as sentence embedding or vector) with fixed length. This representation is expected to be a good summary of the meaning of the whole input. Encoders are recurrent neural networks, i.e., LSTM or GRU units, which can model sequential data.
Attention mechanisms have become an integral part of sequence models in several tasks,
allowing the model to learn dependencies without regard to their distance in the input or output
sequences. Attention mechanisms are usually used in conjunction with recurrent networks but
also as the core of the transformer architectures [
The ongoing project hypothesizes that the building blocks described above can be exploited to deal with the challenges of DS search. Then, we can design a neural ranker for DS tasks that leverages contextual information. The contextual information can be included by analysing DS features in search tasks and text representations that pay attention to the expert knowledge contained within DS corpora. Two retrieval scenarios are considered following a re-ranking technique: first, a lexical-based retrieval phase is followed by a re-ranking phase based on semantic similarity; second, both the first phase and the re-ranking phase will be based on semantic search. These scenarios will allow us to explore diferent features of the models, for example, the value of exact matching and the efect of the vocabulary mismatch problem. On the other hand, diferent training strategies will be considered for text representation and text classification to build a more suitable semantic space for the specific task and to learn interactions between the inputs for re-ranking. This will allow us to evaluate if the tasks can benefit from fine-tuning text representations and if the learned representations are suitable for the first-stage retrieval and re-ranking.
This work is supported by the EU Horizon 2020 ITN/ETN on Domain Specific Systems for Information Extraction and Retrieval (H2020-EU.1.3.1., ID: 860721).