On an abstract level, one is often confronted with some type of classification problem where we have one example instance or a textual query and we are looking for the class most appropriate for this instance or query. More concretely, we consider journals as classes and the papers published in certain journals as constituting and describing the respective class. In this scenario two information needs are conceivable: (1) We know one paper and we are looking for all journals which could potentially contain similar work. (2) We want to write a paper, have a first working title, and are looking for journals which could be potential targets for a submission of that paper. In this work, we transfer methods used in expert search and data fusion to find appropriate journals: Using a flat, title based search query for articles we examine voting models used in expertise retrieval with its different data fusion techniques to find and rank journals associated with the matching articles that potentially contain most suitable other articles. To evaluate the ranking of found journals, we remove several test articles from the applied collection and utilize them as request items with their titles. We assume that-on average-the journals where these test articles have been published should be among the top ranked journals to provide a suitable result. This fully automated evaluation provides the opportunity to execute a huge number of requests against the collection of articles and to evaluate the different voting techniques transferred from expert search.
In a first step, we don’t evaluate the discussed techniques using manually created test data or test users. Instead, we use article-titles from the collection itself to automatically send these titles as search requests. Since we have the information in which journal a single article has been published, we can measure the position of this respective journal in the result ranking and evaluate the algorithms.
This work is based on research in data fusion techniques and their application in the
field of expertise retrieval. Different approaches show, that combining multiple
retrieval results using voting models can improve retrieval effectiveness [
In the paper at hand, we investigate how such approaches perform in our setting. We present first promising experimental results and discuss potential future research directions.
It should be mentioned that existing journal recommenders from publishers—like EndNote’s manuscript matcher, Elsevier’s journal finder, or Springer’s journal suggester—are obviously related to our approach. However, these systems apply complex ranking schemes using much more information than our simple approach discussed in this short paper. The aim of our paper is to investigate the capability of rather simple voting techniques in the sketched scenario. A comparison with the existing journal recommenders will be an interesting next step but is out of scope for this short paper. 2
This section describes the utilized ranking techniques. For the flat, title-based article
search—i.e. the underlying document ranking—we use Elasticsearch’s classic TF/IDF
[
Document ranking TF/IDF. The ( , ) of a document given a query which consists of terms is computed as follows ( , ) = ∑ ( ( ) ⋅ ( )2 ⋅ ( )). The term frequency of term in document is computed as ( ) = √ . The inverse document frequency for term is computed as ( ) = 1 + log ( ), where is the number of all documents in the collection and ( ) + 1 ( ) is the number of documents containing term .
( ) =
for a matching document causes √ higher weights for short documents (documents with a lower number of terms ) in the score computation.
The score formula for Lucene’s classic similarity TF/IDF in addition contains other weighting factors (for normalization and coordination) which are not considered or not relevant in our experiments and not involved in the score-computation. BM25. For BM25, the ( , ) of a document given a query which consists of terms is computed as follows: The term frequency describes the number of occurrences of term in document , | | represents the document length, and is computed as the average document length over all documents in the collection.
Here the inverse document frequency for term is computed as ( ) = log (1 + − ( ) + 0(.5) + 0.5), with and ( ) defined as before. In our experiments, we use BM25 with standard values for (1.2) and (0.75). 2.2
Collection ranking
Based on the article ranking as search result, four approaches introduced in expert
search are adopted to derive a journal ranking—respectively, a collection ranking. In
general, the voting model can be based on different inputs: the number of items in the
search result associated with a collection, the ranks of the items associated with a
collection, and the score values calculated for the items associated with a collection [
Let ( ) be the set of articles retrieved for the query and ( , ) the computed score for journal and query , we apply four different voting models: Votes. This metric takes the number of found articles for every journal as the score: ( , ) = |{ ( , )
CombMAX. This metric takes the first result stemming from , respectively, the article
with the highest ranking, as voting candidate with its score:
( , ) =
({
(
, ) ∶
For our experiments, we take data from the dblp computer science bibliography (Digital
Bibliography & Library Project), an online reference for bibliographic information on
major computer science publications [
In the first step, we take 10,000 randomly chosen articles and remove them from the collection. All removed items serve as requests for the evaluation series with their title. The length distribution of all titles used as keyword queries is as follows: The minimum is 1 word. 25% of the queries/titles comprise 7 or fewer terms. The median is 9 and the 3rd quartile is 12. The longest title among the 10,000 chosen titles consist of 37 words.
For every item of the 10,000 articles, all combinations of the two similarity algorithms and the four voting models described in section 2 are combined and applied. Hence, every search request yields eight sets of ranked journals.
For each article and for each set of ranked journals the rank of the article’s corresponding journal is determined and saved. Ideally, an article’s corresponding journal should appear at rank 1 or at least among the top ranks of all ranked journals.
Our setup contains 1,657 different journals and nearly 1,500,000 articles. Figure 1 shows the distribution of articles over journals. Nearly 60% of all journals (965 items) contain 500 articles at most. As the maximum, the collection contains one journal in which more than 18,000 articles are published. 4
For all 10,000 article-requests, we determine the rank of the article’s corresponding journal. Table 1 shows the effectiveness results for each combination of document-similarity model and voting technique. We state the 1st quartile (value meaning that for 25% of the queries the respective journal has been among the top results), the median, the 3rd quartile, the share of queries where the expected journal is in top 10, and the mean reciprocal rank (MRR).
Aggregations calculated by the Votes algorithm for BM25 and TF/IDF have the worst—and the same—results. For only 25% of the queries the journal associated with the requested article appears among the top 30 results. CombANZ and CombSUM follow, they deliver at least 25% of the corresponding journals within the top 20 results. ) 1,000 e l a c s g o ( 100 l s l a n r u o fJo 10 r e b m u N 1 965 278 < < < < < < < < < < < < < < < < < < < < < 005 ,0010 ,0150 ,0200 ,0300 ,0040 ,0050 ,0060 ,0070 ,0080 ,0090 ,0010 ,1010 ,2010 ,3010 ,4010 ,5010 ,0610 ,0710 ,0810 ,0910 0 0 0 0 0 0 0 0 0 0
Number of Articles Fig. 1. All 1,657 journals categorized by the number of published articles Z N A b m o
C model ► voting technique (combination) ►
median 70 3rd quartile 242 share with journal 15.7% in top 10
MRR 0.06 When comparing all ranking methods, it turns out that CombMAX yields the best results regarding the journal ranking. Here, no noise based on lower ranked results— respectively, articles—that contribute as voters is generated. In certain cases, these lower ranked results considered in the aggregation distort the ranking. When applying CombANZ and CombSUM, this effect is shown. Both methods yield worse results, especially regarding the rankings above the median and first quartile.
The approach using the Votes algorithm is not well suited in this form: not even 50% of the rankings see the expected journal under the top 100. This is because the approach does not differentiate between results on higher and on lower ranks.
CombMAX as the best performing approach is also in accordance with the
experimental results presented in [
For further research, we plan to modify the applied aggregation techniques: according to current results, a CombSUM technique which considers only the upper ranking articles might deliver more accurate results. Considering only the top article result for every journal would end up in the applied CombMAX method.
Taking a closer look at the results it is a bit astonishing that CombSUM is much
closer to Votes than to CombMAX. One could hypothesize that the decline of the score
values might be well suited for ranking single documents but not for summing them up
to yield a combined score for a journal. Problems with independence assumptions as
well as ranking equivalent transformations in the formulas might be potential reasons
in this respect. We plan to consider this further and to investigate combination schemes
reflecting these observations. The sketched observations might also be the reason why
using the reciprocal ranks from documentsrespectively, articlesin combination
schemes performed surprisingly well in [