Conference papers are evaluated according to many criteria reflected in numerical scores, and the wording of the criteria differs among conferences. This makes the role of meta-reviewers tough when summarizing the evaluation across multiple criteria and reviewers. Based on a micro-study within semantic technology conferences, we conjecture that the criteria can, for particular fields, be mapped on generic metrics, and provide a provisional ontological representation for such a mapping and a set of metrics, as well as a manual mapping tool. Finally, we showcase an application exploiting the mappings: a graphics generator that aggregates the review data into a complex pictorial metaphor.
Introduction Conference papers are often evaluated according to multiple criteria reflected in numerical scores. In large conferences with many reviews per paper this amounts to dozens of partial figures.This sheer number, and the fact that the wording of the criteria differs from one conference to another, make the role of meta-reviewers during the discussion periods difficult, and the effort invested into the detailed scoring may partly get lost.
In the research we first explored whether the criteria can be generalized across events within a field such as Semantic Technology (ST) to a small set of review metrics. Based on the positive outcome of this study, we designed a provisional ontology for representing the mappings between specific review forms and such generic criteria, and developed a simple mapping authoring tool and a mapping execution component. Finally, we developed a tool that demonstrates one possible way of exploiting the mappings: a review visualizer that assembles the metric values, for a set of reviews of the same paper, into a compound pictogram relying on the racing cars metaphor.
Contributions of this paper are thus both the small empirical study and a multipart demo. In the demo we can demonstrate how: 1) a mapping from a form to the common set of metrics can be created and published, 2) the values for a concrete set of reviews can be manually entered, thus emulating an automatic input from a hypothetical component of a conference review system, and 3) the pictorial scene can be generated.
Review Criteria Micro-Study and Mapping to Generic Metrics We analyzed the review forms of nine ST conferences, always for the latest edition we could access as author/s or reviewer/s. We semantically clustered the field labels (referring to the reviewer guidelines where in doubts), yielding seven partial review metrics that we named as in the first column of Tab. 2, plus two global metrics, Confidence and Overall score, present in all forms. The partial criteria converged well despite the varying wording, though some forms missed certain metrics at all. ISWC and SEMANTiCS were clearly influenced by one another, having the same set of fields. ESWC had two fields that we both ranged under ‘Technical quality’. We do not list K-CAP, as it had no partial numerical field; this may be related to the ‘workshop flavor’ of this event. 3
Ontological Representation of Review Forms and Metrics
For a review of existing relevant ontologies we can refer to our up-to-date study of
research-related ontologies in general [
Demo Suite We developed a suite of four simple tools to demonstrate the whole concept. They are bundled by the web page http://pictoreview.vse.cz/. The source code for the first three tools is at https://github.com/jurs02/PictoReviewDev.
The first tool allows the user to create a mapping from the custom set of review form fields of a particular event to the proposed set of generic metrics. The mapping can be 1:1, 1:N or N:1. An example of a mapping (for the SEMANTiCS’18 Research Track) is in the first two columns of Tab. 1. The mapping can be currently stored as a JSON structure or as an RDF dataset described by our ontology from Sect. 3. 3 https://sparontologies.github.io/fr/current/fr.html 4 https://sparontologies.github.io/bido-review-measures/current/ bido-review-measures.html
The second tool is a simple mapping execution API, which transforms a set of review form fillings of a specific conference (a JSON structure) to the generic metrics (also output in JSON), using the JSON mapping (valid for that conference) authored by the first tool. For the N:1 mapping (i.e., of multiple fields to the single metric), a numerical mean of the values is computed. Note that the first and second tool together provide (a baseline of) a general review data interoperability infrastructure, usable independent of the rest of the demo; for example, the reviewing emphases of different conferences could be compared based on the mappings.
The third tool emulates the role of a hypothetical plug-in to an off-the-shelf review management system (RMS). The user manually enters both JSON data structures expected by the second tool: the (saved) mapping, and the specific review form fillings, for example, those from the last three columns of Tab. 1. The data is then transformed to generic metrics (by the second tool) and passed to the fourth tool.
The fourth tool, the pictogram generator, eventually, converts the generic metric values to components of a complex pictorial metaphor. We identified ‘car’ as a relatively close metaphor to a research paper, and car components (plus other ‘car race’ features) as visual variables expressing the metrics values. In Fig. 1 we see the visual representation of the set of reviews from our SEMANTiCS’18 example, cf. Tab. 1. The whole picture encodes 27 numerical values: 9 metrics 3 reviewers. For brevity let us only point out the ‘good’ and ‘bad’ scores. Reviewer 3 (R3) appreciated the novelty of the paper (big engine), its evaluation (solid wheels), and also presentation (smiling face). R1 valued the state of the art (shining headlamp) and technical quality (body style: cabrio as most suitable for a racing car), and also evaluation (wheels). R2, in turn, only praised the paper for its high relevance (this would be indicated by the track quality, however, the difference is too small;5 with an even lower value, the track would change to dirt or even turf), while the presentation was poor in particular (frowning face). The reviewer confidence (lower for R1) does not measure the paper quality as such; therefore we use an orthogonal visual magnitude paradigm, the color saturation/salience. Finally, the cars are positioned on the track by their overall scores. 5 For simplicity, all scores except the overall evaluation are mapped to three-valued visual variables only, thus 4 and 5 fall to the same interval. C i T ) 9 1 0 2 ( I A C J I ) 6 1 0 2 ( S I O F ) 8 1 0 2 ( C W S ) 6 1 0 2 ( I A
C iew ic
r v t e e s e c n e r e f n o c E K f o s d l e fi m r m w e o f d n a s c i r t e i v e r c i r e n e g n e e e b g n i p p a w t m d e s o p o r P
m e E I r
n A ig
N S e h t f o y t i l a r e n e g d n y a d
u y t it s l l i i c
a b t n e u
m d i ro r
e p p A e x e c n a c fi i n g ifi e n c g n i a
S c A i N S n o i t a u l a v l a c i y n ti h l c a e u n o s i s t a en t n d e n m uo le s p d m n I a l a c i y n ti h l c a e u
T q r o i t n c l fi a c y i t n i h l e i a cS tec qu d e se th o f p o o r n p o
i eh ss h t
u ac f c o is ro s d p s e d ap n n e a d t e l
e n s o o
p p it om tra ro c s p
n e nd o h
t a s em f
o
D s s en ; t n e i t r c o e i r t r o l
e u p
o o r C s p l a c i
y n ti h l c a e u l a c i
y n ti h l c a e u T q k r o w d e t a l e p i h s r a l o h c S s e c n e r e f t r a e h t f o e t a t s e h t f o n o i t a u l a v p i h s r a l o h c
S e h t f o e t ta tr
S a s s / e
n ty e li iv t a v a n i g o i r n O in t ro on y ti l a e v v o y t i l a n i g i r n o i t u l o s d e s o p o r p e h t f o y t l e v o y t l e v o y t i l a n i g i r y t l e v o e ce ap n p a v e R e th K l eR fo to f o y t l e v o e n N th co , N C A )8 W M 1 S E 20 I S (
A
R R
O C q w A o n N N in e A A
R N N R
N
E
R e N E N ) W 0 A 2 K 02 A E ( N
d i n h - n d a h n s t c l u s p e a o e e A T c s n d N t i r l A A la d a y t i t i n u f r g n i N
N N C a q o w E
R P q O
T q R m
R
N
Future Prospects
The paper presents an initial proof of concept of a review form interoperability
framework, plus a review pictogram generator on the top of it. To bring the concept closer to
real usage, we have to undertake experiments determining whether and in what setting
the pictograms provide an added value over numerical tables. Some of the visual
variables adhere to metaphors studied by psychologists [