While many web-based systems recommend relevant or interesting scientific papers and authors, few tools actually recommend journals as likely outlets for publication for a specific unpublished research manuscript. In this paper we discuss one such system, Manuscript Matcher, a commercial tool developed by the authors of this paper, that uses both content and bibliometric elements in its recommendations and interface to present suggestions on likely “best fit” publications based on a user's draft title, abstract, and citations. In the current implementation, recommendations are well received with 64% positive user feedback. We briefly discuss system development and implementation, present an overview and contextualization against similar systems, and chart future directions for both product enhancements and user research. Our particular focus is on an analysis of current performance and user feedback especially as it could inform improvements to the system.
many thousands of additional data points. While we are focusing specifically on recommending scholarly journals (based at least in-part on the cumulative reputation of a journal) this is different than general journal influence as often represented in metrics like the Journal Impact Factor [7] and the Eigenfactor Article Influence scores [8].
Recommender systems are typically classified based on their filtering approach in three broad categories: content-based filtering, collaborative filtering, and hybrid recommendation systems [9]. For the discussions here, we will consider Manuscript Matcher as a content-based system augmented with bibliometric-enhanced filtering. The established characteristic strengths and weakness of these approaches are welldocumented [1, 9] so frame our definition of bibliometric-based filtering as an approach that starts with linguistic content—text in article titles and abstracts—and enhances Natural Language Processing (NLP) analysis of this content with bibliometric elements [10].
Content-only approaches have often shown to be error-prone do to the complexities of matching terms among myriad vocabularies [3, 4]. To minimize these challenges, we enhanced our text analytics and content-based classifications with bibliometrics. In particular, we leveraged the rich subject categories, journal ranking metrics, and citation network from the Clarivate Analytics Web of Science and Journal Citation Reports (JCR). More than 10 million content records from 8,500 journals with hundreds of millions of supporting bibliometric data elements from the past 5 years of indexing were used [11].
There is some recent research validating the successful use of bibliometric elements in scholarly paper recommendation tools. However, these do not specifically focus on recommending journals as likely publication outlets for unpublished research papers so findings should be viewed as tangential [12, 13]. 2
Manuscript Matcher is currently in “soft commercial release”—meaning that it is publically available but not widely promoted or advertised. The feature was launched in February of 2015 and branded as the “Match” function of EndNote online. More than 50,000 users have tried the tool and the feedback from these users is discussed and analyzed later in this paper.
While our focus in this study is not on the algorithmic details of the Manuscript Matcher system development, we include here just a brief overview of the broad underlying technical approaches. We generally took a “human in the loop machine learning” approach that enabled human expertise, spot-checking of results, and expert user feedback to supplement the learning tasks of the algorithms. To make recommendations for new, unpublished papers, we looked at millions of previously published papers in journals across many academic domains.
This data was sourced is two ways: first, full text papers were collected from various open-access repositories, and second, we used meta-data records from the Web of Science. The system architecture comprises both journal classifiers and a recommendation aggregator journal taxonomy, which has three levels and is based on an agglomerative clustering of the domain journals, and applied thousands of models on each paper in the training data. Manuscript Matcher itself uses a Support Vector Machine (SVM) classifier, implemented with LibLinear, as a global classification algorithm. A Lucene based inverted index is then used as the basis for a k-Nearest Neighbors (kNN) local clustering algorithm. Both algorithms are supervised in that they utilize the true journal a given paper was published in as training data. Both models are used concurrently and the average of their confidence score is used to calculate how well the recommended journals match the users input.
The system analyzes jargon used in manuscripts and determines citation patterns in bibliographies. Citations, specifically author name, journal and full title, are used as features, and the model learns the importance of each citation part. This way, one journal model can learn that citations coming from a specific author are important for that journal, while the model of another journal can learn to prefer papers citing a specific seminal paper. In the current implementation, key bibliometric and content elements of a draft paper are identified and used to enable the algorithms to identify the most suitable journals for a submitted manuscript and provide predictive insight to its acceptance probability.
The training data used were titles, abstracts and citations of papers that were actually published in the domain journals covered in the Web of Science corpus. Experiments with predicting acceptance probability based on an accept/reject flag and full text were carried out during the proof of concept phase, but this was not included in the current state product. The reason being that the results were inconclusive; while there was some signal for predicting acceptance probability, it was a much more difficult problem than matching a manuscript to a journal.
Manuscript Matcher also includes a specialized capability to match multidisciplinary submissions to journals of a corresponding, multi-disciplinary nature; this capability was influenced by some core applications of Bradfordizing [14, 15]. Plus, the system is capable of using a set of rejected manuscripts to determine which journals are least likely to accept the manuscript for publication. In the interface, the user is presented with supporting bibliometric evidence from the JCR for the recommended journals; these data points help the author determine the ultimate “best fit” for their paper. The user interface also includes recommendations for similar or related papers that serve to further contextualize the journal recommendations. Based on general user feedback, the similar article recommendations are among the most popular and useful features of the Manuscript Matcher tool.
We did some preliminary experiments with co-authorship, now often included in discussions of “social network analysis” [4, 16] but have not implemented these approaches in the current version of the tool as these methods did not result in significant improvements to the accuracy or quality of the recommendations. Further investigation along these lines may be explored in future phases of development and research. 3
Publishing manuscripts efficiently is essential for disseminating scientific discoveries
and for building an author’s reputation and career. But even with the use of
streamlined, web-based systems, this process can take time; a recent study of journals on a
leading online submission platform, found that the time to first decision on submitted
manuscripts averages 41 days [17]. Appropriateness of articles—matching the scope
of the journal—is overwhelmingly cited as both the primary quality editors and
reviewers look for and the main reason for rejection from journals across many
academic fields [
While any scholarly author might find Manuscript Matcher useful, it is targeted toward a few specific user personas hoping to publish in a peer-reviewed journal: researchers in the early stages of their career with minimal publishing history, nonnative speakers who may be publishing in an English language journal for the first time, and established researchers who want to publish outside their core discipline.
For the early career researcher, whose concerns often focus on establishing their reputation, Manuscript Matcher recommendations are accompanied by ancillary data to facilitate making the best choice. This data includes: an overall Match score, the Current and 5-year Journal Impact Factor, and Subject Category, Rank and Quartile information from JCR. When advising novice researchers, many experienced authors specifically recommend targeting journals from Web of Science and those with a Journal Impact Factor [22, 23].
For researchers looking to publish in an English language journal for the first time, and established researchers who want to publish outside their core discipline, their results will include links to articles similar to their submission sourced from the Web of Science, which can be added to their EndNote library and cited in a later draft.
Manuscript Matcher results are derived from greater than 10 million records across hundreds of subject areas contained within the Web of Science corpus. Purposefully excluded from the 10 million records are the contents of journals with a very low Journal Impact Factor and journals that publish infrequently. The intention of Manuscript Matcher is to use the wide, multi-disciplinary scope of content and bibliographic data from the Web of Science to recommend journals from a broad range of publishers that cover varied subject areas within the sciences, medicine, and humanities to bring distinct usefulness over the current state of the art.
While Manuscript Matcher includes novel elements, it is not the only such system available [24]. In preparing this paper we found six other similar and publically available tools: Elsevier’s Journal Finder [20], Springer’s Journal Suggester [25], the Biosemantics Group’s Jane [26], SJFinder’s Recommend Journals [27], Research Square’s Journal Guide [28] and Edanz’s Journal Selector [29].
These are all hosted by either established primary academic publishers like Elsevier and Springer, where recommendations focus on the journals they publish, or by newer organizations offering a suite of bespoke publishing and editing services. After briefly experimenting with these sites, it appears that five of the six tools use some type of bibliometric indicators that are most commonly manifest in the user interface as a single journal influence metric per recommended journal. It is unclear whether any of the services listed above leverage bibliometric data in their actual recommendation algorithms, if not, the use of bibliometric data in Manuscript Matcher seems more substantial as it is included in both the algorithms and the user interface. 4
We gather user feedback in hopes of continually refining and improving Manuscript Matcher’s recommendation output. This data is currently being collected for insights into general user satisfaction and to use in collaborative filtering approaches in future improvements to the recommendations. This user feedback data is the basis for the analysis presented below. An end user who has submitted a title, abstract, and an optional set of citations will be presented with journal recommendation results. While not every submitted combination of abstract, title, and citations meets the minimum confidence threshold to result in recommendations, when recommendations are provided, early user testing feedback indicated that the optimal number of results was between two and ten. Each recommendation provides an option for the user to respond with feedback. This is displayed in the interface as a question, “Was this helpful?” with “Yes” and “No” answers; users can include free text commentary in addition to the binary choice. It is important to note that, since leaving feedback is nonmandatory, significant non-response bias is introduced to the data. This bias has not been corrected for in this analysis.
Approximately 5.6% (2,770) of the nearly 50,000 users of Manuscript Matcher left feedback on 1,800 journal recommendations for the specific date range of February 20, 2015 to September 26, 2016. During this period, there was an overall 64% satisfaction rate—those choosing a positive “Yes” response when asked “was this helpful?”—for users supplying feedback on individual journal recommendations. Of the 36% negative, or “No” responses, about half, or 44.5%, submitted a comment indicating why the recommendation was not useful to them. In contrast only 24.5% on those choosing a positive “Yes” response left written feedback. Also interesting is the actual frequency of the words used in the comments: “good” and “helpful” were the most common in positive comments and occurred nearly twice as frequently as “nothing” which was the most prevalent word in negative comments. Looking more closely at this, the ten most frequently used words indicate some broader patterns and point toward some curious inferences. Positive feedback words—good, helpful, thanks, match, great, excellent, one, will, relevant, research—generally reflect homogenous sentiment. We interpret this as—those who had positive feedback and took the time to write a comment, indicated a straightforward approving tone. Conversely, the words in negative remarks—nothing, match, study, related, research, field, topic, subject, title, case—suggest more variety and, perhaps, that users were trying to share more details in hopes of helping to improve future recommendations.
Percentage of Users Providing Written Feedback Positive Words thanks good helpful match great excellent one will relevant research
N Negative Words 46 nothing 33 match 32 study 23 related 16 research 13 field 12 topic 12 subject 11 title 11 case
Benchmarking our user satisfaction numbers has been particularly challenging as so little research exists on comparable systems particularly focused on user feedback. We found studies that looked at: algorithmic optimization of user feedback, that show “click-through” performance numbers ranging from 5-69%, and test data performance (in matching likely publication venues) ranging from 44-94% but these are only tangentially analogous [4, 12, 30]. In certain situations, negative feedback could actually be interpreted as validation of the algorithm's accuracy, in the case of users complaining that they had already been published in a recommended journal and are looking for an alternative option.
Contrary to some other published findings [3], during development we found an overall 30% increase in accuracy in analysis of test data using a combination of NLP techniques on titles and abstract text along with key bibliometric data compared to just the use of text. While internal testing indicated that including the citations of a paper in the submission improved the quality of recommendations, user feedback for submissions with citations did not confirm this. Of the submissions that included citations, the mean number of included citations was 43.9, with a median value of 23. Submissions without citations garnered 66.3% positive feedback, while those with citations received only 52.9% positive feedback. It is difficult to draw conclusions on this data due to the many sources of bias inherent in post-hoc data analysis. 5
currently shown within the Web of Science and on the Highly Cited Researchers website [35]. Expanded sort capabilities and results filtering would help the user apply research indicators in order to support their manuscript submission decision.
The inclusion of targeted feedback related to publication outcomes is designed to support the user’s decision in which journal to publish and to serve as a more informative guide on wider journal performance using research indicators. In later iterations, the user’s profile and past publications may factor in to display personalized suggestions. 6
As previously noted, we found very little formal research published specifically on journal recommendation services like Manuscript Matcher. So, the field is wide open to future investigations into various dimensions including: their effectiveness, technical and algorithmic approaches, user perceptions and satisfaction. During initial development, we did interview users and their feedback informed technical and design decisions in the current tool. We plan to continue to collect user input through the existing interface as well as perform more in-depth interviews of users from a wide-range of academic domains.
From the growing body of research done on online product feedback trends, user sentiment and motivation, there are parallels and diversions with Manuscript Matcher that warrant further investigation. The overall Manuscript Matcher feedback skews positive with 64% selecting “Yes” next to each journal recommendation returned when asked if it was helpful, a distribution that differs from studies done on optional product feedback trends. In particular, a large scale assessment of Amazon ratings of books or CDs showed optional feedback following a U-shaped distribution, with most ratings either very good or very bad, which was in contrast to controlled experiments, where opinions on the same items are normally distributed [36]. A notable difference is that the Amazon users studied were able to provide a star rating out of five, while Manuscript Matcher users are only able to provide a Yes/ No answer. In future iterations, implementing an A/B test where users are either presented with the option of providing a rating out of five stars or Yes/ No feedback to see if the different feedback options impact 1) the percentage of users that provide feedback and 2) the ratings distribution.
We would also like to further investigate why submissions without citations garnered 66.3% positive feedback, while those with citations received only 52.9% positive feedback. This was an unexpected outcome, as the inclusion of citations increases the accuracy of the Manuscript Matcher recommendations, and research done on end user motivations for providing feedback was consulted.
Altruism is cited as one of the leading reasons for providing both positive and negative feedback [37]. In the case of Manuscript Matcher, altruism would take the form of providing positive feedback in order to give the company “something in return” for a good experience. Altruism could also motivate a user to provide negative feedback when they want to prevent others from experiencing the problems they encountered as well as help improve the tool’s accuracy so that future users benefit [38].
Another primary motivation for submitting negative feedback would be anxiety reduction, the easing of anger and frustration when faced with results the user views as either not relevant or substandard [37]. This would likely be prevalent in users who had entered their title, abstract, and had taken the extra step of providing a curated EndNote Group with their manuscript citations. More invested in the process and its outcome, these users submitting citations would likely feel greater levels of dissatisfaction when recommendations did not meet or exceed their expectations. 7
In this paper we describe Manuscript Matcher, a commercial journal recommendation tool that leverages content-based and bibliometric approaches to recommendations. We give an overview of the current implementation, briefly compare Manuscript Matcher to a few similar tools, and analyze current user satisfaction and feedback from nearly 2,800 users of the system. As well, we discuss plans for future user research and development. In informal tests Manuscript Matcher performed well compared to similar systems but more rigorous and formal study is needed to validate this. User feedback is largely favorable, with 64% overall positive sentiment, and we hope to improve recommendation acceptance with both expanded bibliometric approaches and the addition of collaborative filtering.
Further future development of the tool will utilize new Clarivate Analytics data points to aid the user. These will include user interface changes to display expanded data points derived from the JCR. User experience improvements will support the manipulation of results through the application of data filters as well as targeted and eventually personalized recommendations. The end goal of these additions is to provide as much relevant data as possible to ensure that Manuscript Matcher users are able to make the most informed decision in the journal submission process. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36.