Multidimensional Relevance in Legal and Health Domains

Abstract

People engage in search activities to solve everyday problems that are often related to health or legal situations. In these domain-specific searches, topical relevance is insuficient, and recent studies have mentioned the need for multidimensional relevance models. Also, several users, tasks and domain factors have been identified and assessed to estimate the “utility” of an information item with respect to the situation at hand. Determining relevance in domain-specific search becomes more challenging when studied within the associated working environment. The purpose of this paper is to provide an overview of such situations, by identifying several user types, tasks and relevance factors in legal and health related searches.

Keywords

• eol>Multidimensional Relevance
• Domain-specific Search
• Professional Search
• Legal Domain
• Health Domain

1. Introduction

Relevance has been perceived as a multidimensional and dynamic concept highly influenced by user, task, and domain characteristics. Furthermore, numerous studies have examined how diferent factors afect relevance judgments under specific search tasks. Those studies have identified factors such as topicality, understandability, reliability, scope, novelty, but also, interest and habit that are user-related [ 1 ]. In addition, in [ 2 ] it has been investigated how the importance of those relevance factors changes with respect to the undertaken task. Finally, several multidimensional relevance models have been proposed to improve retrieval efectiveness, but only a few focus on domain-specific or professional search.

Our research will address this issue by incorporating domain-specific relevance factors in the retrieval decision process. Specifically, we will focus on distinct contextual situations that involve diferent types of users and tasks. In those situations, estimating relevance constitutes a complex decision that may benefit of the definition of decision-theoretic models to assess the “utility” of an information object to specific user’s needs. In addition, these models will be evaluated across diferent domain-specific and professional contexts and tasks to the aim of verifying how relevance assessment depends on specific relevance factors, in a decision theoretic setting.

2. Domain-Specific & Professional Search

Domain-specific search is defined by Lupu et al. [ 3 ] as a search focused on a specific subject area with various modalities that involve a variety of users, tasks, and technical aspects. Concerning professional search, in [ 4 ] the following definition was provided: “professional search takes place in the work context, by specialists, and using specialist sources, often with controlled vocabularies.” Adding to this, it was also reported that “professional search has the key benefit that the task to be solved is, usually, clear; at least to the person who carries out the searches.”

An initial study conducted on professionals with the aim of investigating standard search practices and goals has identified several common characteristics across four specific domains [ 5 ]. Furthermore, another recent study among professionals explored their typical search tasks [ 6 ]; these search tasks were coded with respect to task characteristics and relevance factors. At the same time, the findings showed that similar aspects of professional search apply to experienced searchers across the various studied domains.

The studies mentioned above identified several types of users in the considered domains, and associated them with specific search tasks, even within working environments. Also, several commonalities across the studied domains have been found. Both experts and non-experts may engage in domain-specific searches to fulfill similar information needs. In professional search, users are primarily experts, but their information needs are driven by diferent tasks related to their working roles. It can be concluded that considering user characteristics for assessing relevance may improve the efectiveness of domain-specific search, while taking into account additional task characteristics in professional search may also yield improvements in retrieval efectiveness. The latter was investigated in a recent study [ 7 ].

3. Assessing Domain-Specific Relevance

3.1. Health Domain

Over the years, many researchers have investigated how relevance is perceived in health-related searches by laypeople and medical experts, and a few multidimensional relevance models have been proposed. In [ 8 ], five distinct types of medical expert users have been identified, i.e., self-employed general practitioners, self-employed specialists, hospital clinicians, physicians in training, and research physicians; all of them share some level of domain expertise and perform specific tasks. Certainly, ordinary people may also search for health-related information through commercial search engines or social platforms. For ordinary people, without specific domain knowledge, the undertaken search tasks are often related to retrieving relevant information about a health condition, browsing for self-education, or search about their chronic disease [ 3 ]. Expert users engage in a variety of health-related search tasks such as literature reviews, scoping reviews, rapid evidence reviews, systematic reviews [ 5 ]. Often, both experts and laypeople utilize the same sources, such as the open web search, to fulfill their information needs.

Previous researches have identified factors that afect relevance assessments in health-related searches, and a recent systematic review analyzed the findings of 37 empirical studies, reporting consumers’ perspectives about the quality of online health information [ 9 ]. The studies analyzed concerned common people with no health expertise, while the undertaken search tasks were similar to those mentioned above and were performed in the context of general web search. In the majority of the studies, trustworthiness, expertise and objectivity related to the source and the document’s content, were indicated as core dimensions of online health information quality as perceived by consumers. Another work [ 10 ] investigated the impact of domain expertise on relevance assessment in clinical settings. Here, experts and non-experts were asked to perform clinical search tasks. The authors also explored how the levels of retrieval performance change according to the variability of the ground truth. The findings suggested that assessing relevance based only on topicality is not suficient for experts as they usually leverage from their knowledge and experience to assess a multidimensional relevance. Concluding, the authors mentioned the importance of identifying factors on which experts rely for determining relevance.

The work conducted in [ 11 ] investigated how user, context, and task characteristics can be used to predict the situational relevance of a health-related document. Several non-experts participants were asked to perform searches related to treatments to a symptom or disease. Then, diferent document, user, and task characteristics were manually collected and used to predict situational relevance. The results identified several factors that afect the perceived relevance of an information item with respect to the studied situation.

Over the years, various evaluation forums and conferences introduced tracks focused on specific health-related search tasks. For instance, the TREC Precision Medicine 1 and the Clinical Trials Track2 provided a framework in which the same information need, i.e., synthetic patient cases, can be used to retrieve either relevant biomedical articles or clinical trials. Moreover, the CLEF eHealth3 initiative has focused on consumer health searches in which relevance has been conceived as multidimensional by introducing other dimensions such as reliability and understandability.

3.2. Legal Domain

Various unique features can be found in the legal domain, such as lengthy legal documents with a specific structure that includes legal terminology and citations. Regarding the domain itself, it contains several document types that come with an associated hierarchical organization, and authority [ 5 ].

Searching for legal-related information is an activity conducted by experts, such as lawyers, law librarians, and paralegals [ 5 ]. Despite their commonalities, minor diferences related to their field of expertise and experience may exist. Similar to the health domain, non-experts may also search for legal-related information. Search activities performed by legal experts aim at gathering evidence to answer a legal question or to provide evidence to support a legal

1https://trec.nist.gov/data/clinical.html

2http://www.trec-cds.org/2021.html 3https://clefehealth.imag.fr/ argument [ 5 ]. In addition, experts search for similar prior cases and relevant statutes or acts. Recently, these search tasks have been investigated in two initiatives, COLLIE4 and AILA5. Also, laypeople may search in order to understand better the legal situation they are involved.

The wide variety of users, the complexity of the tasks undertaken by experts, and several other domain peculiarities, form a complicated framework in which the notion of relevance should be treated diferently than in other domains. An exhaustive analysis of the notion of relevance in the legal domain has been conducted by Van Opijnen and Santos [12], following the manifestations of relevance introduced in [ 1 ]. It is worth mentioning that besides the definitions presented in [ 1 ], the authors introduced bibliographic relevance which is defined as “the relationship between a request and the bibliographic closeness of the information objects.” Finally, the authors introduce domain relevance, which is described as the relevance of information objects within the legal domain itself. This new dimension derives from the “socio-cognitive relevance” measured with respect to the situation (work task or problem at hand) and the information objects within a socio-cultural context.

Following the study mentioned above, in [13] several factors that influence relevance assessment in legal professional search were identified. The most important were related to the document type, recency, level of depth, and legal hierarchy. This well-conducted study seems in the right direction, but further research is needed to identify those factors afecting relevance under a specific search activity, i.e. investigate situational relevance.

4. Discussion & Ongoing Research

Both in the legal and health domains, studies have mentioned the need for better IR systems able to assess multidimensional relevance considering factors related to the user and the undertaken task. Several studies have identified essential relevance factors following this direction, while a few studies introduced methods to measure these factors and to accordingly assess multidimensional relevance. Although multidimensional relevance has been studied in general health-related search, there is not much progress for health-related professional search. In contrast, relevance has been mainly studied within working environments and not generally in the legal domain.

The need to model domain-specific and professional search activities as a specific decision process in the given domains is supported by the previously reported observations. Moreover, within a domain, the importance of each relevance factor can be afected by particular search tasks or by the user’s role and context. As a result, the relevance factors mentioned above (domain and task-dependent), as well as their interactions, should be modeled by analysing their (possible) trade-of to estimate the “utility” of an information item with respect to the situation at hand. Multidimensional relevance models able to suitably quantify those factors (task and user-related) across domains, by considering and assessing their interactions to estimate the “utility” of an information item may be also efective when employed across diferent domains. All in all, our research will investigate the notion of relevance with respect to the undertaken tasks both inside the working environment (professional searches) but also outside (domain

4https://sites.ualberta.ca/~rabelo/COLIEE2020/

5https://sites.google.com/view/aila-2020 specific searches). In this context we aim at defining decision theoretic models that can flexibly assess relevance in domain specific contexts, by efectively accounting for the appropriate relevance factors and for their interplay in relation to specific search tasks.

Acknowledgments

This work is supported by the EU Horizon 2020 ITN/ETN on Domain Specific Systems for Information Extraction and Retrieval (H2020-EU.1.3.1., ID: 860721). users and tasks to predict the situational relevance of health web documents, Journal of Information Systems Engineering & Management 2 (2017) 25. [12] M. van Opijnen, C. Santos, On the concept of relevance in legal information retrieval,

Artif. Intell. Law 25 (2017) 65–87. [13] G. Wiggers, S. Verberne, G. Zwenne, Exploration of intrinsic relevance judgments by legal professionals in information retrieval systems, in: Proceedings of the 17th Dutch-Belgian Information Retrieval workshop, 2018, pp. 5–8.

References

[1] T. Saracevic , The Notion of Relevance in Information Science: Everybody knows what relevance is . But, what is it really?, Synthesis Lectures on Information Concepts , Retrieval, and Services, Morgan & Claypool Publishers, 2016 .

[2] Y. Xu , Relevance judgment in epistemic and hedonic information searches , J. Assoc. Inf. Sci. Technol . 58 ( 2007 ) 179 - 189 .

[3] M. Lupu , M. Salampasis , A. Hanbury , Domain specific search , in: G. Paltoglou, F. Loizides , P. Hansen (Eds.), Professional Search in the Modern World - COST Action IC1002 on Multilingual and Multifaceted Interactive Information Access , volume 8830 of Lecture Notes in Computer Science, Springer, 2014 , pp. 96 - 117 .

[4] S. Verberne , J. He , U. Kruschwitz , B. Larsen , T. Russell-Rose , A. P. de Vries, First international workshop on professional search (profs2018) , in: K. Collins-Thompson , Q. Mei , B. D. Davison , Y. Liu , E. Yilmaz (Eds.), The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , SIGIR 2018 , Ann Arbor, MI, USA, July 08 - 12 , 2018 , ACM, 2018 , pp. 1431 - 1434 .

[5] T. Russell-Rose , J. Chamberlain , L. Azzopardi , Information retrieval in the workplace: A comparison of professional search practices , Inf. Process. Manag. 54 ( 2018 ) 1042 - 1057 .

[6] S. Verberne , J. He , G. Wiggers, T. Russell-Rose , U. Kruschwitz , A. P. de Vries , Information search in a professional context - exploring a collection of professional search tasks , CoRR abs/ 1905 .04577 ( 2019 ).

[7] T. Schoegje , E. L. van den Broek , A. Pieters, Validating the importance of work tasks as context for professional search , volume 2127 , Ruzica Piskac /CEUR-WS. org, 2018 , pp. 24 - 28 .

[8] M. Kritz , M. Gschwandtner , V. Stefanov , A. Hanbury , M. Samwald , Utilization and perceived problems of online medical resources and search tools among diferent groups of european physicians , J Med Internet Res 15 ( 2013 ) e122 .

[9] Y. Sun , Y. Zhang , J. Gwizdka , C. B. Trace , Consumer evaluation of the quality of online health information: Systematic literature review of relevant criteria and indicators , J Med Internet Res 21 ( 2019 ) e12522 .

[10] L. Tamine , C. Chouquet , On the impact of domain expertise on query formulation, relevance assessment and retrieval performance in clinical settings , Inf. Process. Manag . 53 ( 2017 ) 332 - 350 .

[11] M. Oroszlányová , C. Lopes , S. Nunes , C. Ribeiro , Using the characteristics of documents,