The recommendation and assignment of reviewers and articles is an emerging topic in the academic community and is known as the Reviewer Assignment Problem (RAP). This problem can be seen as a version of the well-known Generalized Assignment Problem (GAP), while the Conference Paper Assignment Problem (CPAP) is a specific case of RAP. The thesis proposal presented is centred on CPAP. The main objective of the doctoral thesis proposal is the development and validation of multi-information recommendation systems (extracting information from the content of articles and reviewers) capable of eficiently assigning expert reviewers to scientific articles submitted to a conference.
The recommendation and assignment of reviewers to articles, generally called in the literature
a Reviewer Assignment Problem (RAP), has become an essential topic in the academic world.
RAP can be seen as a version of the Generalized Assignment Problem (GAP), and Conference
Paper Assignment Problem (CPAP) is a specific case of RAP [
In most cases, articles should be assigned to reviewers according to the following conditions: (i) Each manuscript should be assigned to a certain number of reviewers, ai, defined by the team responsible for assigning the reviewers; (ii) as far as possible, each article should be assigned to reviewers who are experts in the field. A specific limit T can be defined to identify the qualification of the reviewers; (iii) each reviewer should be assigned to, at most, a certain number of articles, bj, defined by the team responsible for assigning the reviewers; (iv) each reviewer should be assigned to approximately the same number of articles, to balance their workload.
Given a set = {1, ..., | |} of manuscripts and a set = {1, ..., ||} of reviewers, denote the matching degree of manuscript i for reviewer j, where i belongs a P and j belongs a R. A binary variable , whose value is 1 if manuscript i is assigned to reviewer j and 0 otherwise, RAP is formulated by the Following Integer Programming formulation:
Subject to : ∑︁ ∑︁
∈ ∈ ∑︁ = . ∈ ∑︁ ≤ .
∈ ∈ {︁⌊︁ ⌋︁}︁ .
= 0, 1.
The objective function (equation 1) maximizes the total matching degree of the assignment. Constraints (equation 2) and (equation 3) ensure that conditions (i) and (ii) are satisfied, respectively. Constraint (equation 4) along with (equation 5) prevent a reviewer from being assigned to a manuscript whenever is smaller than the given threshold T. Constraint (equation 6)instead of (equation 4) is brought into the mathematical model to ensure that at least one reviewer whose matching degree for manuscript i is greater than or equal to T.
∈ { } ≥ .
The peer-review system is considered the main mechanism for quality control of scientific
publications [
Second, it is common to recommend expert researchers in the same research area as the article to be reviewed. However, there are many other indicators that should be considered when selecting reviewers, for example, publications, research projects, and patents. Also, teams that select reviewers use titles, rewards, and status to assess the quality of the experts. Third, the process of manually selecting and assigning reviewers to articles ignores possible relationships between reviewers and authors. Finally, since it is necessary to manage a large amount of (1) (2) (3) (4) (5) (6) information about the reviewers and the articles submitted according to human and subjective criteria, bias may occur in the recommendation and assignment of researchers.
According to the problems presented above, there is a need to apply intelligent technologies capable of analyzing data, extracting valuable information from documents and unstructured texts, and thus automatically recommending and assigning the most appropriate researchers to scientific articles. This doctoral study aims to solve the CPAP through the development of a system capable of recommending the most appropriate and expert reviewers and eficiently assigning them to scientific articles, according to the established constraints.
The development of automatic and eficient mechanisms for recommending experts has a high economic value. These mechanisms allow the increase in productivity, which transposes into ifnding experts in a valuable search time and increasing eficiency, ensuring the highest possible level of correspondence between their profile and task.
In the scientific world, these expert recommendation mechanisms provide the added value of inferring and identifying research teams and working groups, especially between researchers from diferent institutions; discovering emerging talents with low visibility; supporting students, allowing them to identify the best supervisors and co-supervisors of master or doctoral degrees.
Also, it is essential to keep the quality standard of science high. The review of scientific articles in scientific conferences and journals, and scientific projects is one of the best-known tasks for expert researchers. The recommendation and assignment of the most suitable experts to the articles are based on human and subjective criteria, which can generate wrong decisions, such as the rejection of excellent scientific work or a potentially successful project proposal. These decisions can have significant and adverse efects on the quality of the scientific standard, namely, the quality of published studies, researchers’ careers, and the reputation of conferences and journals.
The present PhD study aims to develop a Recommendation System (RS) based on constraint programming solvers applied to the CPAP case. A more detailed description of the objectives is as follows: 1. Development of a data collection system capable of collecting relevant input information from researchers who are candidates for reviewers and from scientific articles. 2. Development of an Information Extraction system combining deep neural networks and advanced NLP techniques in extracting the semantic representation of scientific articles’ content and reviewers’ information gathered. 3. Development of an expert RS capable of extracting the expertise level of re-viewers and ifnding the ranking of expert reviewers in a given research topic (s)/area(s). 4. Search, compare and select available open-source next-generation constraint programming technology solvers and other recent optimization approaches to deliver the best trade-of between eficiency and ease of integration and customization. 5. Integration of the constraint programming-based constraint-solving module in the developed RS.
According to the previous context, motivation and objectives, some research questions arise, namely: 1. How can the CPAP be formulated as a constraint programming problem? 2. What is the efect of deep neural networks combined with natural language processing techniques in extracting the semantic representation from the content of scientific articles? 3. Do CP solvers frameworks allow handling reviewers’ constraints in the context of assigning articles more eficiently when compared to other current solving approaches? 4. Do CP solvers frameworks improve the balance between eficiency and ease of integration and customization compared to diferent methods?
RAP has two main phases – 1) finding/recommending expert reviewers and 2) assigning reviewers to submitted manuscripts. These problems are diferent and therefore require diferent approaches. In phase 1), the main objective is to compute the article-reviewer similarity factors depending on the method (implicit or explicit) chosen to describe the articles and the competencies of the reviewers. While in phase 2) the main objective is the efective assignment/allocation of expert reviewers to scientific articles.
According to the literature, diferent approaches can be applied in phase one, namely:
decision support systems [
Finding expert reviewers (phase 1) generally requires the development of relevant tasks such
as collecting data on the reviewers and, sometimes, on the authors of submitted articles; the
construction of the profiles of reviewers and articles submit-ted and, finally, the computation of
the similarity between reviewers and authors. The approaches based on DSSs and RSs have
several similarities in their processes. For example, the development of the reviewers’ profile
and articles submitted through specific measures (quality, relevance, authority, diversity, among
others) have been shown to achieve good results in recommending reviewers [
The problem of assigning/allocating reviewers to submitted articles (phase 2) presents several types of approaches tested by researchers. The approaches based on heuristic and metaheuristic algorithms have a strong presence in researchers’ studies [11], [16]. Also, greedy and genetic algorithms are the best known and selected by researchers to try to solve the attribution problem. Other approaches, such as the fuzzy approach [17], ASP [14] or integer linear programming, also aroused the interest of researchers and showed good results.
The defined approach proposes the construction of knowledge about the expertise of researchers and the recommendation and assignment of the most appropriate expert reviewers for scientific papers. The proposed system comprises four modules: 1. Data Collection Module – The main purpose of this module is to collect data from multiple data sources (open databases and Web) to create a database with information about researchers (afiliations, co-authors, citations, number of publications, awards, among others) and scientific articles (metadata). 2. Information Extraction Module – The main objective of this module is to extract relevant information from the data, transforming it into more significant representations of its semantic content and, consequently, easier to analyse. This module combines advanced NLP techniques and deep neural networks for the extraction of semantic and contextual information from the collected data. 3. RS of Experts Module – This module focuses on the recommendation of reviewers’ experts in a given research topic/area. The relevant information extracted in the previous module allows the construction of reviewers’ profiles (based on quality, reputation, and expertise) and scientific articles (based on metadata and key insights). Subsequently, the objectives of RS are to extract the behavioural model of experts, compute the expertise level of each reviewer and, finally, present the ranking of experts. 4. Assignment Module – In this module, the main objectives are the definition of the constraints of the assignment of reviewers to scientific articles and the implementation of a sophisticated optimization approach (constraint programming) to enforce the adequacy of the recommended reviewers to the input constraints and requirements.
The system validation starts with the comparison of the results obtained in this study with the results found in the literature of approaches used by other researchers to solve the RAP. Furthermore, we intend to validate the system in a real environment, namely, at a scientific conference. At this stage, we want to compare the results of the recommendation and attribution of our system with the results of the conference system. In addition, we intend to assess the satisfaction of reviewers through a survey.
The main expected contributions of this PhD study are: 1. Development of an RS to be integrated into a constraint resolution module based on sophisticated constraint programming and optimization approaches to handle constraints more eficiently when compared to other current constraint resolution approaches. 2. Study the efect of using Deep Neural Networks in extracting semantic representation of scientific papers’ content and researchers’ contents for the recommendation task.
This work was financially supported by: Base Funding -UIDB/00027/2020 of the Artificial Intelligence and Computer Science Laboratory – LIACC - funded by national funds through the FCT/MCTES (PIDDAC). The first author is supported by FCT (Portuguese Foundation for Science and Technology) under grant PD/BD/2020.04698.BD.