The problem of automatically assigning reviews is a crucial task in conference management. Not only is it a time-consuming and challenging task, but it is also one of the most important peculiarities that contribute to the good/bad organisation of the conference, not to mention the degree of satisfaction of the people involved. During review assignments, many constraints come into play: the maximum number of papers per reviewer, the minimum number of reviews per paper, conflict handling and, last but not least, the similarity between papers' topics and reviews' interests. In this paper, we propose a strategy to map topics using the ACM Computing taxonomy, and a modelization of the problem in a Constraint Satisfaction setting relying on the Answer Set Programming (ASP) logical framework. This strategy, although not scalable, showed the capability of managing many real situations that have not been captured so far. Experiments demonstrated the goodness of performances for small/medium numbers of papers.
The problem of assigning papers to reviewers is fundamental in organizing conferences, workshops or peer review processes. The challenge lies in the fact that many constraints come into play. An efective assignment should, in principle, guarantee that every paper is reviewed by (at least two) persons among the most competent in the field(s) in which the paper resides. Moreover, it is expected that reviewers are comfortable in the reviewing process by judging papers they feel are related to their line of research, ofering the possibility to create a profitable interaction between the reviewed and the reviewer. A mapping satisfying these conditions does not seem tough in principle. Yet in reality, these ideal conditions clash with constraints that make the assignment cumbersome, time-consuming, and with a generally lower satisfaction of the researchers involved, possibly causing frustration and a lower quality of the event itself. More formally, the problem of paper assignments can be viewed as finding a function (assignment) mapping papers to (a set of) reviewers. Given a finite set of papers = {1, 2, ..., }, a finite set of authors = {1, 2, ..., } with authorship function : → 2, and a set of reviewers ℛ = {1, 2, ..., }, the assignment function ℱ : → 2ℛ is a function that assigns to each paper a set of reviewers. The function is neither injective since multiple papers may be submitted to the exact set of reviewers (although rare) nor surjective given that it is not mandatory in all cases a person has to work on reviewing. Figure 1 shows the visual interpretation of an assignment function. The rest of the paper is organised as follows: Section 2 provides the theoretical background of the involved concepts, Section 3 provides an overview of the state-of-the-art for this task, Section 4 provides the modelization of the model, Section 5 lists the ASP implementation of the modelization, Section 6 shows the result of the experimentation and gives more insight about the process and, finally, Section 7 concludes the manuscript and gives rise to future extensions and integrations.
1 2 3 4 5 6 1 2 3 4
In order to understand how to relate topics of interest of reviewers with those related to the papers, a large, standard and comprehensive conceptualization needs to be considered. We refer to these conceptualizations as ontology. This concept has origins in philosophy. The Computer Science community understood the need to catalogue and classify a domain since the early beginning of the discipline. The word ontology is used with diferent meanings in diferent communities. In the early days, ontology designers sought to provide a complete classification of every concept available in the “universe” of knowledge. Not too far did this approach fail due to the overwhelming amount of objects to be classified and the variety of possible ways in which objects can be categorized according to the specific use, context, period and many other environmental factors. Last but not least, the objective of an ontology formalization is to provide a formalization that is “stable”, as much as possible. For this reason, it should not be too tight to a specific context.
In the AI field, the aim is to create a model of the reality of interest. From a formal point of view, an ontology is a triple (, ℛ, ) in which: • is a set of concepts. • ℛ is a set of relationships between concepts.
• is a set of axioms on which the universe we want to describe is based [
entity is_a
is_a object has_part is_a
is_a engine car
person repairs is_a mechanic
The problem of defining the correct function from papers to reviewers has to consider general constraints regarding (for instance) the minimum or maximum number of reviews per paper, conflicts between authors, the policy of the conference or workshop, but also some specific needs of some authors (e.g. availability of a reviewer for few papers). This setting comes under the Constraint Satisfaction Problem domain.
The classic definition of a Constraint Satisfaction Problem (CSP) is as follows. A CSP
is a triple = ⟨ , , ⟩ where is an -tuple of variables = ⟨1, 2, ..., ⟩, is a
corresponding n-tuple of domains = ⟨1, 2, ..., ⟩ such that ∈ , is a -tuple of
constraints = ⟨1, 2, ..., ⟩. A constraint is a pair ⟨ℛ , ⟩ where ℛ is a relation on
the variables in =scope( ). In other words, ℛ is a subset of the Cartesian product of the
domains of the variables in . A solution to the CSP is an -tuple = ⟨1, 2, ..., ⟩ where
∈ and each is satisfied in that ℛ holds on the projection of onto the scope .
In a given task one may be required to find the set of all solutions to determine if that set is
non-empty or just to find any solution, if one exists. If the set of solutions is empty the CSP is
unsatisfiable [
In our settings, we can map , and according to the automatic paper assignment process. In this case, is the set of papers, is the power set of the reviewers (2ℛ), and a complete assignment is a function associating papers with a set of reviewers. The set is composed of a set of constraints that guide the assignment.
We are not only interested in finding an assignment satisfying the constraints but also in
ifnding the assignment that maximises the similarity between paper topics and reviewers’
interests. This setting lies in an extension of the Constraint Satisfaction Problem which is called
the Constraint Optimization Problem. Basically, a Constraint Optimization Problem is defined
in the same way as a CSP one but the (function) assignment needs to maximise (resp. minimise)
a target numerical function. Constraint Satisfaction Problems and Constraint Optimization
Problems can be managed by diferent solvers like Gecode [
Answer Set Programming (ASP), also known as Disjunctive Logic Programming (DLP)
under stable model semantics, is a powerful framework for Knowledge Representation and
Reasoning. Originating from the work of Gelfond, Lifschitz, and Minker in the 1980s, ASP has
gained increasing interest within the scientific community. One key factor in its success is its
highly expressive language: asp programs can precisely express any property of finite structures
over a function-free first-order structure that is decidable in nondeterministic polynomial time
with an NP oracle, meaning asp encompasses the complexity class Σ 2 = . As a result,
asp enables encoding programs that cannot be converted to SAT in polynomial time. Notably,
asp is entirely declarative (the sequence of literals and rules does not afect the outcome), and its
encodings for a wide range of problems are concise, straightforward, and elegant. Unfortunately,
the high expressiveness of asp comes at the price of a high computational cost in the worst
case, making the implementation a tough task.
2.3.1. Syntax
Following a convention dating back to Prolog, strings starting with uppercase letters denote
logical variables, while strings starting with lowercase letters denote constants. A term is either
a variable or a constant. An atom is an expression (1, ..., ), where is a predicate of arity
n and 1, ..., are terms. A literal is either an atom (positive literal) or its negation ¬
(negative literal). Two literals are said to be complementary if they are of the form and ¬ for
some atom . Given a literal , ¬ denotes its complementary literal. Accordingly, given a set ℒ
of literals, ¬ℒ denotes the set {¬ | ∈ }. A set ℒ of literals is consistent if its complementary
literal is not contained in ℒ for every literal ∈ ℒ. A disjunctive rule (rule, for short) is a
construct: 1 ∨ ... ∨ ⇐ 1, ..., , ¬+1, ..., ¬ where 1, ..., , 1, ..., are literals and
≥ 0, ≥ ≥ 0. The disjunction 1 ∨ ... ∨ is called the head of , while the conjunction
1, ..., , ¬+1, ..., ¬ is referred to as the body of . A rule without head literals (i.e. = 0)
is usually referred to as an integrity constraint. A rule having precisely one head literal (i.e. n =
1) is called a normal rule. If the body is empty (i.e. = = 0), it is called a fact. If is a rule of
form (1), then ℋ() = {1, ..., } is the set of literals in the head and ℬ() = ℬ+() ∪ ℬ− ()
is the set of the body literals, where ℬ+() (the positive body) is {1, ..., } and ℬ− () (the
negative body) is {+1, ..., }. An asp program (also called Disjunctive Logic Program or DLP
program) is a finite set of rules. A not-free program (i.e., such that ∀ ∈ : ℬ− () = ∅) is
called positive or Horn, and a v-free program (i.e., such that ∀ ∈ : |ℋ()| ≤ 1) is called
normal logic program. In asp, rules in programs are usually required to be safe. A rule is
safe if each variable in also appears in at least one positive literal in the body of . An asp
program is safe if each of its rules is safe. A term (an atom, a rule, a program, etc.) is called
ground if no variable appears in it [
Many solutions to this problem rely on semi-automatic processes. Dumais et al. [
Nguyen et al. [
To the best of our knowledge, Kalmugov [
All these works explore a wide range of methods to compute the distance between papers and reviewers. Yet there is still the need to model constraints in the assignment process, which gives us reasons to explore a new approach.
Having shown the ingredients we are going to use, we describe here how we model the problem. First of all, we need a taxonomy (ontology) of disciplines in the Computer Science field. This taxonomy allows us to measure the similarity between topics. Referring to Figure 2, we have only a set of classes and the is_a relationship. The structure we will have is a (rooted) tree, in which the root is named Computer Science, and from that, all the specializations will be given. With this structure, we have a connected graph, in which it is always possible to find a path connecting two classes by traversing the is_a arcs or their inverses (indicated as is_a− ). Hence, the algorithm used is the Shortest Path Length [18]. In our case, the Shortest Path Length coincides with the minimum number of edges (is_a or is_a− ) needed to connect the two nodes in the graph. More formally, we have a tree = (, ℰ ) where is the set of nodes and ℰ : × expressing the is_a function. We simplify the notation (1, 2) ∈ ℰ with 1 – is_a – 2 and we generalise with is_a* to represent is_a or is_a− . We define a path from 0 to as 0 – is_a* – 1 – is_a* – ... – is_a* – . The minimum number of needed is_a* is the Shortest Path Length. Given the tree structure, it is necessary to assume that ̸= ∀ ̸= in order to find the minimum path. Given ℒ the length of the taxonomy, the distance between two nodes , (indicated as distance(, ) always exists and 0 ≤ distance(, ) ≤ 2ℒ. The two extreme cases (distance = 0 or distance = 2ℒ) happen, respectively, when = and when , have only the root of the tree as a common parent.
A shared conceptualization is useful for aligning papers’ topics with reviewers’ ones. To this extent, the keywords are used for the papers; for the reviewers, we use the labels available on Google Scholar. Obviously, paper keywords and researchers’ labels are not syntactically aligned. Here the shared conceptualization comes into play. In order to align words syntactically, we use the Large Language Model (LLM) GPT-4o [19]. The query asked to select, for every keyword in the paper (also taking into account the others) the closest category in the ontology. Quite similarly, for every Google Scholar label of researchers, we asked for the closest category in the ontology. As it happens in general with LLMs, hallucinations are a problem in the results. To mitigate this problem, we do not take the result provided by GPT-4o as it is but we try to link it to the correct ACM label. We make use of the Levehnstein distance [20] to retrieve the string that has the lowest distance with the label provided by the LLM. Given the interpretation of this distance (i.e. the number of characters to be added, removed or changed in order for two strings to be equal), we decided that if the value is greater than 40% of the length of the closest ACM category, then the label provided by the LLM is discarded since considered unreliable. Ultimately, we have a full syntactic mapping between papers and researchers. From now on, we use keyword both for paper keywords and researchers’ labels since they have been mapped within a common vocabulary.
The Constraint Optimization Problem is provided with the following variables (and domains): • = {1, ..., } papers, • = {1, ..., } researchers, • ℛ ⊆
reviewers, • = {1, ..., } keywords, • ℒ : ( ∪ ) → 2 labelling function on papers and authors, • : → 2 authorship function on papers, • : × 2 conflicts relationship between researchers.
Having defined the distance between two keywords, we can define the distance between a paper and a researcher. Both papers and researchers are represented by sets of keywords, and, intuitively, we look for the similarity between the two sets. For every keyword of the paper, we look for the keyword of the researcher minimizing the distance. The same keyword for the researcher may be used more than once. Formally, the distance between a paper and a researcher is computed as: distance(, ) = ∑︀∈ℒ() min∈ℒ() distance(, ).
Stepping back to Figure 2, let’s suppose a paper has keywords mechanic, car and entity, and the reviewer has labels engine, object, and entity. For the keyword mechanic, the closest reviewer’s label is one of the three indiferently since the minimum distance is 2, for car the closest are engine and object with distance 1, and for entity we have an overlap with the reviewer’s label. Hence, in this case, the distance is 0. In the end, the sum is 2 + 1 + 0 = 3.
The objective is to find the assignment : → 2ℛ that minimizes ∑︀∈ ∑︀∈ () distance(, ). Many constraints can be considered for this task. Given , ∈ N, the ones we consider are: i. ∀ ∈ ℛ , | ∈ : ∈ ()| ≤ , ii. ∀ ∈ , | ()| ≥ , iii. ∀ ∈ , () ∩ () = ∅, iv. ∀ ∈ , () ∩ { | ∃ ∈ () : ∈ ()} = ∅, v. | ∈ : ∈ ()| ≤ , < .
Constraint (i) guarantees that at most papers are assigned to each reviewer so as to not overload someone. Constraint (ii) guarantees that a paper is reviewed by at least reviewers, to guarantee a certain reliability and heterogeneity in the judgment process. Constraint (iii) guarantees that authors do not review themselves. Constraint (iv) generalizes (iii) and avoids conflicts between authors and reviewers. Constraint (v) includes a family of constraints. It guarantees that a specific reviewer (for some reason) is not available to review more than a quantity of papers that is lower than the maximum .
Finally, the solution to our problem is arg min ∑︀∈ ∑︀∈ () distance(, ) satisfying constraints from (i) to (v).
We opted for ASP to model the problem given its competitive performance, expressiveness, and optimization utilities. Clingo1 is responsible for grounding and solving and has the capability to abduce all the possible solutions. When Clingo performs optimization, intermediate results are available. The optimal solution is not guaranteed to be unique, and Clingo provides all the possible models. For the experiments, we selected the IRCDL dataset available from the 20th anniversary in which all the (315) publications of the first twenty editions were available. Authors, titles, and keywords were extracted from the dataset. As per reviewers, we opted for the Program Committee members of the 20th edition of IRCDL.
In the computer science field, in order to categorize topics we used the ACM Computing Classification System 2. In principle, this taxonomy is not rooted but we added the root Computer Science for two reasons: (i) we can compute distances between topics not having a common ancestor, instead of complicating the notation and the management with possible undefined values, and (ii) to use the root as a collector for extremely general keywords or research areas in the field. The classification includes 1915 categories and 1928 is_a relationships between categories.
In the constraints, we force every paper to be reviewed by at least two authors, and every author cannot be assigned to more than four papers. For every author we have afiliations, and by general rule we consider authors to be in conflict if and only if they belong to the same afiliation (or have one afiliation in common). All experiments have been conducted on an Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz-1.50 GHz processor with 16GB of RAM. The chosen execution is multithreaded with 4 cores. In the implementation, we tended to anticipate all the possible preprocessing phases like finding ancestors in the ACM classification and conflicts among researchers. Full ASP implementation is available here (https://zenodo.org/records/14733925).
Listing 1 shows the ASP implementation of the automatic reviews’ assignment.
Listing 1: Automatic Reviews’ Assignment in ASP 1 %sort areas 2 ancestor(X, X, 0) :- area(X). 3 ancestor(X, Y, 1) :- parent(X, Y). 4 ancestor(X, Y, N) :- parent(X, Z), ancestor(Z, Y, N-1). 5 6 has_child_ancestor(X, Y, A) :- parent(A, B), ancestor(B, X, _), ancestor(B, Y, _). 7 1https://potassco.org/clingo/ 2https://dl.acm.org/ccs 8 %find first common ancestor 9 1 { first_common_ancestor(X, Y, A) : ancestor(A, X, _), ancestor(A, Y, _) } 1 10 :- area(X), area(Y). 11 :- first_common_ancestor(X, Y, A), has_child_ancestor(X, Y, A). 12 13 %compute distance between two areas 14 distance(X, Y, D1 + D2) :15 first_common_ancestor(X, Y, A), 16 ancestor(A, X, D1), ancestor(A, Y, D2). 17 18 %conflicts between researchers of the same affiliation 19 conflict(X, Y) :- X != Y, affiliation(X, U), affiliation(Y, U). 20 21 %maximum of articles to assign to everybody 22 max_articles(4). 23 24 %number of reviews 25 n_reviews(2). 26 27 %minimum distance between a keyword and a reviewer 28 1 { distance_target(K, R, D) } 1 :29 keyword(_, K), reviewer(R), 30 D = #min { Dis, R2 : distance(K, R2, Dis), 31 research(R, R2) }. 32 33 %distance between a paper and a reviewer 34 dissimilarity_target(P, R, S) :35 paper(P), reviewer(R), 36 S = #sum { D : distance_target(K, R, D), 37 keyword(P, K) }. 38 39 %generate assignments 40 N { assign(P, R, S) : dissimilarity_target(P, R, S) } N :- paper(P), n_reviews(N). 41 42 %avoid conflicts 43 :- assign(P, R, _), author(P, A), conflict(R, A). 44 :- assign(P, R, _), author(P, R). 45 46 %number of assignments per reviewer 47 assigned_reviewer(R, N) :- reviewer(R), N = #count { P,R,S : assign(P, R, S) }. 48 49 %respect limit of reviews per reviewer 50 :- assigned_reviewer(R, N), assign_max(R, M), N > M. 51
For evaluation, we ran multiple experiments varying the number of maximum articles to be assigned to each reviewer and the total number of papers. We do not expect that the maximum number of articles will afect the general performance so much. Conversely, we want to verify that including constraints creates benefits in the process, and we also tested with and without the constrain to avoid conflicts. Times reported in Figure 3 consider only the time to reach the first result (local optimization). Many local optimizations can be output before reaching the global optimization but the time for moving from one optimization to the next one is quite constant.
Results show that times are more than acceptable, especially when the number of papers does not exceed 150. As expected, the number of maximum articles does not afect the results too much, although increasing that value creates benefits in general. More interestingly, removing the condition of conflicts is unhelpful when the number of papers increases, as shown in Figure 4. Our suspect is that removing the computation of conflicts is for sure beneficial at the beginning, but having more pruning helps in the coupling between papers and reviewers by removing lots of possible combinations. To date, it is not possible to quantitatively assess the accuracy of the match given that the dataset of IRCDL has not been labeled according to the reviewer’s interest. Compared to the state-of-the-art, to the best of our knowledge, this is the first approach capable of taking into account all the possible constraints given its symbolic nature. Although the complexity of ASP with optimization is (2 · ) where is the number of constraints, it may be worth using it instead of heuristics-based algorithms that cannot take many constraints (apart from the maximum number of assigned papers) into account. It is likely that for huge conferences this approach may be limiting, but it could be used in combination with existing approaches when, for example, allocating fewer papers that are not easy to assign, or solving specific conflicts during assignments.
The problem of assigning reviews can be generalized as a classification problem. When thinking about digital libraries in the general run of things, it is possible to categorize documents into categories. For instance, it is possible to map categories of documents with categories in the context of specific libraries. Given the physical limitations of libraries, constraints come into play also in this context. Apart from ontology mapping which is always time-consuming, the general methodology is reusable and the new constraints can be adapted without many dificulties.
In conclusion, in this paper, we showed an alternative modelization in the process of automatic paper assignments to reviewers. This strategy, given its symbolic nature, can take into account realistic constraints for the conferences, solve conflicts automatically, and introduce personalization in the decision process. Experiments demonstrated that for not big conferences the performances are more than acceptable, showing also how increasing the number of constraints afects execution in general.
We look forward to receiving the results of the assignment process by asking IRCDL reviewers to evaluate the quality of the assignment process. Extensions of this work are the evaluation of other datasets, for which accuracy can be computed, and explore also diferent strategies (e.g. pure CSP) and solvers. [18] G. Yu, J. Yang, On the robust shortest path problem, Computers & operations research 25 (1998) 457–468. [19] J. A. Baktash, M. Dawodi, Gpt-4: A review on advancements and opportunities in natural language processing, arXiv preprint arXiv:2305.03195 (2023). [20] F. P. Miller, A. F. Vandome, J. McBrewster, Levenshtein distance: Information theory, computer science, string (computer science), string metric, damerau? levenshtein distance, spell checker, hamming distance, 2009.