=Paper=
{{Paper
|id=Vol-1823/paper2
|storemode=property
|title=Manuscript Matcher: A Content and Bibliometrics-based Scholarly Journal Recommendation System
|pdfUrl=https://ceur-ws.org/Vol-1823/paper2.pdf
|volume=Vol-1823
|authors=Jason Rollins,Meredith McCusker,Joel Carlson,Jon Stroll
|dblpUrl=https://dblp.org/rec/conf/ecir/RollinsMCS17
}}
==Manuscript Matcher: A Content and Bibliometrics-based Scholarly Journal Recommendation System==
https://ceur-ws.org/Vol-1823/paper2.pdf
BIR 2017 Workshop on Bibliometric-enhanced Information Retrieval
Manuscript Matcher: A Content and Bibliometrics-based
Scholarly Journal Recommendation System
Jason Rollins1, Meredith McCusker2, Joel Carlson3, and Jon Stroll4
1
Clarivate Analytics, San Francisco, CA, US
jason.rollins@clarivate.com
2
Clarivate Analytics, Philadelphia, PA, US
meredith.mccusker@clarivate.com
3
Clarivate Analytics, San Francisco, CA, US
joel.carlson@clarivate.com
4
Clarivate Analytics, Philadelphia, PA, US
jon.stroll@clarivate.com
Abstract. While many web-based systems recommend relevant or interest-
ing scientific papers and authors, few tools actually recommend journals as
likely outlets for publication for a specific unpublished research manuscript. In
this paper we discuss one such system, Manuscript Matcher, a commercial tool
developed by the authors of this paper, that uses both content and bibliometric
elements in its recommendations and interface to present suggestions on likely
“best fit” publications based on a user’s draft title, abstract, and citations. In the
current implementation, recommendations are well received with 64% positive
user feedback. We briefly discuss system development and implementation,
present an overview and contextualization against similar systems, and chart fu-
ture directions for both product enhancements and user research. Our particular
focus is on an analysis of current performance and user feedback especially as it
could inform improvements to the system.
Keywords: Recommendation Services × Bibliometrics × Algorithms × Machine
Learning × Paper Recommender System × User Feedback × Content Based Filter-
ing × Natural Language Processing (NLP)
1 Introduction & Background
Hundreds of papers and books have been written in the past decade-and-a-half study-
ing scholarly paper recommendation tools [1]. This body of literature has investigated
many facets of these systems: scope and coverage, underlying algorithmic approach-
es, and user acceptance [2]. However, relatively few studies have focused analysis on
journal recommendation tools and these have all involved relatively small data sam-
ples or single academic domains [3-6]. This paper expands on this bourgeoning work
and involves feedback from over 2,700 users for 1,800 recommended journals, and
18
BIR 2017 Workshop on Bibliometric-enhanced Information Retrieval
many thousands of additional data points. While we are focusing specifically on rec-
ommending scholarly journals (based at least in-part on the cumulative reputation of a
journal) this is different than general journal influence as often represented in metrics
like the Journal Impact Factor [7] and the Eigenfactor Article Influence scores [8].
Recommender systems are typically classified based on their filtering approach in
three broad categories: content-based filtering, collaborative filtering, and hybrid
recommendation systems [9]. For the discussions here, we will consider Manuscript
Matcher as a content-based system augmented with bibliometric-enhanced filtering.
The established characteristic strengths and weakness of these approaches are well-
documented [1, 9] so frame our definition of bibliometric-based filtering as an ap-
proach that starts with linguistic content—text in article titles and abstracts—and
enhances Natural Language Processing (NLP) analysis of this content with biblio-
metric elements [10].
Content-only approaches have often shown to be error-prone do to the complexi-
ties of matching terms among myriad vocabularies [3, 4]. To minimize these chal-
lenges, we enhanced our text analytics and content-based classifications with biblio-
metrics. In particular, we leveraged the rich subject categories, journal ranking met-
rics, and citation network from the Clarivate Analytics Web of Science and Journal
Citation Reports (JCR). More than 10 million content records from 8,500 journals
with hundreds of millions of supporting bibliometric data elements from the past 5
years of indexing were used [11].
There is some recent research validating the successful use of bibliometric ele-
ments in scholarly paper recommendation tools. However, these do not specifically
focus on recommending journals as likely publication outlets for unpublished research
papers so findings should be viewed as tangential [12, 13].
2 Overview of Current Implementation
Manuscript Matcher is currently in “soft commercial release”—meaning that it is
publically available but not widely promoted or advertised. The feature was launched
in February of 2015 and branded as the “Match” function of EndNote online. More
than 50,000 users have tried the tool and the feedback from these users is discussed
and analyzed later in this paper.
While our focus in this study is not on the algorithmic details of the Manuscript
Matcher system development, we include here just a brief overview of the broad un-
derlying technical approaches. We generally took a “human in the loop machine
learning” approach that enabled human expertise, spot-checking of results, and expert
user feedback to supplement the learning tasks of the algorithms. To make recom-
mendations for new, unpublished papers, we looked at millions of previously pub-
lished papers in journals across many academic domains.
This data was sourced is two ways: first, full text papers were collected from vari-
ous open-access repositories, and second, we used meta-data records from the Web of
Science. The system architecture comprises both journal classifiers and a recommen-
dation aggregator journal taxonomy, which has three levels and is based on an ag-
19
BIR 2017 Workshop on Bibliometric-enhanced Information Retrieval
glomerative clustering of the domain journals, and applied thousands of models on
each paper in the training data. Manuscript Matcher itself uses a Support Vector Ma-
chine (SVM) classifier, implemented with LibLinear, as a global classification algo-
rithm. A Lucene based inverted index is then used as the basis for a k-Nearest Neigh-
bors (kNN) local clustering algorithm. Both algorithms are supervised in that they
utilize the true journal a given paper was published in as training data. Both models
are used concurrently and the average of their confidence score is used to calculate
how well the recommended journals match the users input.
The system analyzes jargon used in manuscripts and determines citation patterns in
bibliographies. Citations, specifically author name, journal and full title, are used as
features, and the model learns the importance of each citation part. This way, one
journal model can learn that citations coming from a specific author are important for
that journal, while the model of another journal can learn to prefer papers citing a
specific seminal paper. In the current implementation, key bibliometric and content
elements of a draft paper are identified and used to enable the algorithms to identify
the most suitable journals for a submitted manuscript and provide predictive insight to
its acceptance probability.
The training data used were titles, abstracts and citations of papers that were actu-
ally published in the domain journals covered in the Web of Science corpus. Experi-
ments with predicting acceptance probability based on an accept/reject flag and full
text were carried out during the proof of concept phase, but this was not included in
the current state product. The reason being that the results were inconclusive; while
there was some signal for predicting acceptance probability, it was a much more diffi-
cult problem than matching a manuscript to a journal.
Manuscript Matcher also includes a specialized capability to match multi-
disciplinary submissions to journals of a corresponding, multi-disciplinary nature; this
capability was influenced by some core applications of Bradfordizing [14, 15]. Plus,
the system is capable of using a set of rejected manuscripts to determine which jour-
nals are least likely to accept the manuscript for publication. In the interface, the user
is presented with supporting bibliometric evidence from the JCR for the recommend-
ed journals; these data points help the author determine the ultimate “best fit” for their
paper. The user interface also includes recommendations for similar or related papers
that serve to further contextualize the journal recommendations. Based on general
user feedback, the similar article recommendations are among the most popular and
useful features of the Manuscript Matcher tool.
We did some preliminary experiments with co-authorship, now often included in
discussions of “social network analysis” [4, 16] but have not implemented these ap-
proaches in the current version of the tool as these methods did not result in signifi-
cant improvements to the accuracy or quality of the recommendations. Further inves-
tigation along these lines may be explored in future phases of development and re-
search.
20
BIR 2017 Workshop on Bibliometric-enhanced Information Retrieval
3 Use Cases
Publishing manuscripts efficiently is essential for disseminating scientific discoveries
and for building an author’s reputation and career. But even with the use of stream-
lined, web-based systems, this process can take time; a recent study of journals on a
leading online submission platform, found that the time to first decision on submitted
manuscripts averages 41 days [17]. Appropriateness of articles—matching the scope
of the journal—is overwhelmingly cited as both the primary quality editors and re-
viewers look for and the main reason for rejection from journals across many academ-
ic fields [18-20]. Initial rejection rates (even before peer review) are as high as 88%
based on manuscripts not meeting “…quality, relevance, and scientific interest…”
[21]. These factors were primary drivers for the development of the Manuscript
Matcher system, paired with increasing agreement that recommendation systems ca-
pable of overcoming these challenges would be a welcome aid to many researchers
[3, 4].
Fig. 1. Manuscript Matcher is accessible through EndNote online. The user enters their Title,
Abstract and optional EndNote Group of references containing their manuscript’s citations.
Fig. 2. The results page will include a list of 2 to 10 journal recommendations. Multiple data
points accompany each recommendation, Similar Articles from that journal in the Web of Sci-
ence are linked to, feedback is solicited, Journal Information provides more about the publica-
tion, and Submit takes the user directly to the journal’s submission page.
21
BIR 2017 Workshop on Bibliometric-enhanced Information Retrieval
While any scholarly author might find Manuscript Matcher useful, it is targeted
toward a few specific user personas hoping to publish in a peer-reviewed journal:
researchers in the early stages of their career with minimal publishing history, non-
native speakers who may be publishing in an English language journal for the first
time, and established researchers who want to publish outside their core discipline.
For the early career researcher, whose concerns often focus on establishing their
reputation, Manuscript Matcher recommendations are accompanied by ancillary data
to facilitate making the best choice. This data includes: an overall Match score, the
Current and 5-year Journal Impact Factor, and Subject Category, Rank and Quartile
information from JCR. When advising novice researchers, many experienced authors
specifically recommend targeting journals from Web of Science and those with a
Journal Impact Factor [22, 23].
For researchers looking to publish in an English language journal for the first time,
and established researchers who want to publish outside their core discipline, their
results will include links to articles similar to their submission sourced from the Web
of Science, which can be added to their EndNote library and cited in a later draft.
Manuscript Matcher results are derived from greater than 10 million records across
hundreds of subject areas contained within the Web of Science corpus. Purposefully
excluded from the 10 million records are the contents of journals with a very low
Journal Impact Factor and journals that publish infrequently. The intention of Manu-
script Matcher is to use the wide, multi-disciplinary scope of content and bibliograph-
ic data from the Web of Science to recommend journals from a broad range of pub-
lishers that cover varied subject areas within the sciences, medicine, and humanities
to bring distinct usefulness over the current state of the art.
While Manuscript Matcher includes novel elements, it is not the only such system
available [24]. In preparing this paper we found six other similar and publically avail-
able tools: Elsevier’s Journal Finder [20], Springer’s Journal Suggester [25], the Bi-
osemantics Group’s Jane [26], SJFinder’s Recommend Journals [27], Research
Square’s Journal Guide [28] and Edanz’s Journal Selector [29].
These are all hosted by either established primary academic publishers like Else-
vier and Springer, where recommendations focus on the journals they publish, or by
newer organizations offering a suite of bespoke publishing and editing services. After
briefly experimenting with these sites, it appears that five of the six tools use some
type of bibliometric indicators that are most commonly manifest in the user interface
as a single journal influence metric per recommended journal. It is unclear whether
any of the services listed above leverage bibliometric data in their actual recommen-
dation algorithms, if not, the use of bibliometric data in Manuscript Matcher seems
more substantial as it is included in both the algorithms and the user interface.
4 User Feedback Analysis & Methods
We gather user feedback in hopes of continually refining and improving Manuscript
Matcher’s recommendation output. This data is currently being collected for insights
into general user satisfaction and to use in collaborative filtering approaches in future
22
BIR 2017 Workshop on Bibliometric-enhanced Information Retrieval
improvements to the recommendations. This user feedback data is the basis for the
analysis presented below. An end user who has submitted a title, abstract, and an op-
tional set of citations will be presented with journal recommendation results. While
not every submitted combination of abstract, title, and citations meets the minimum
confidence threshold to result in recommendations, when recommendations are pro-
vided, early user testing feedback indicated that the optimal number of results was
between two and ten. Each recommendation provides an option for the user to re-
spond with feedback. This is displayed in the interface as a question, “Was this help-
ful?” with “Yes” and “No” answers; users can include free text commentary in addi-
tion to the binary choice. It is important to note that, since leaving feedback is non-
mandatory, significant non-response bias is introduced to the data. This bias has not
been corrected for in this analysis.
Approximately 5.6% (2,770) of the nearly 50,000 users of Manuscript Matcher left
feedback on 1,800 journal recommendations for the specific date range of February
20, 2015 to September 26, 2016. During this period, there was an overall 64% satis-
faction rate—those choosing a positive “Yes” response when asked “was this help-
ful?”—for users supplying feedback on individual journal recommendations. Of the
36% negative, or “No” responses, about half, or 44.5%, submitted a comment indicat-
ing why the recommendation was not useful to them. In contrast only 24.5% on those
choosing a positive “Yes” response left written feedback. Also interesting is the actual
frequency of the words used in the comments: “good” and “helpful” were the most
common in positive comments and occurred nearly twice as frequently as “nothing”
which was the most prevalent word in negative comments. Looking more closely at
this, the ten most frequently used words indicate some broader patterns and point
toward some curious inferences. Positive feedback words—good, helpful, thanks,
match, great, excellent, one, will, relevant, research—generally reflect homogenous
sentiment. We interpret this as—those who had positive feedback and took the time to
write a comment, indicated a straightforward approving tone. Conversely, the words
in negative remarks—nothing, match, study, related, research, field, topic, subject,
title, case—suggest more variety and, perhaps, that users were trying to share more
details in hopes of helping to improve future recommendations.
Percentage of Users Providing Written Feedback
100 Positive Words N Negative Words N
thanks 46 nothing 29
good 33 match 22
75
helpful 32 study 22
match 23 related 21
Percent (%)
Wrote Comment
50 FALSE great 16 research 15
TRUE
excellent 13 field 14
one 12 topic 13
25
will 12 subject 12
relevant 11 title 12
0 research 11 case 10
negative positive
Feedback Valence
Fig. 3. Analysis of user feedback.
23
BIR 2017 Workshop on Bibliometric-enhanced Information Retrieval
Benchmarking our user satisfaction numbers has been particularly challenging as
so little research exists on comparable systems particularly focused on user feedback.
We found studies that looked at: algorithmic optimization of user feedback, that show
“click-through” performance numbers ranging from 5-69%, and test data performance
(in matching likely publication venues) ranging from 44-94% but these are only tan-
gentially analogous [4, 12, 30]. In certain situations, negative feedback could actually
be interpreted as validation of the algorithm's accuracy, in the case of users complain-
ing that they had already been published in a recommended journal and are looking
for an alternative option.
Contrary to some other published findings [3], during development we found an
overall 30% increase in accuracy in analysis of test data using a combination of NLP
techniques on titles and abstract text along with key bibliometric data compared to
just the use of text. While internal testing indicated that including the citations of a
paper in the submission improved the quality of recommendations, user feedback for
submissions with citations did not confirm this. Of the submissions that included cita-
tions, the mean number of included citations was 43.9, with a median value of 23.
Submissions without citations garnered 66.3% positive feedback, while those with
citations received only 52.9% positive feedback. It is difficult to draw conclusions on
this data due to the many sources of bias inherent in post-hoc data analysis.
5 Future Directions for Development
We are currently working on the next generation of recommendation algorithms and
plan to use hybrid approaches leveraging complex views of relationships among vari-
ous system content and bibliometric elements. As mentioned, we hope to incorporate
memory-based collaborative filtering built on user feedback. We also plan to do fur-
ther development on co-authorship and other aspects of “social network analysis” as
limited recent research supports this approach but has not been tested on a full-scale,
production system [5, 31, 32].
There are many potential avenues to explore as we continue refining and iterating
on the core recommendation algorithms. In the future, we hope to incorporate topic
modeling of the abstract text to aid in article clustering, and to revisit social network
modeling of citation data. The current implementation loses considerable information
by relying on a bag-of-words approach to text understanding. To overcome this we
plan to explore more sophisticated approaches to natural language understanding,
such as convolutional neural networks [33] and word embedding models [34].
We also hope to include refinements to the tool’s interface in the form of additional
information, filtering options and targeted feedback. Additional da-
ta points may include the Journal and Category Expected Citation Rates displayed
alongside the JCR information currently provided and recommended journal results
that have recently published a Hot Paper will be noted with an icon. Hot Papers
are highly cited papers that have received enough citations to place it in the top 1% of
its academic field based on a highly-cited threshold for the field and publication year
calculated by Clarivate Analytics using Essential Science Indicators data. This data is
24
BIR 2017 Workshop on Bibliometric-enhanced Information Retrieval
currently shown within the Web of Science and on the Highly Cited Researchers web-
site [35]. Expanded sort capabilities and results filtering would help the user apply
research indicators in order to support their manuscript submission decision.
The inclusion of targeted feedback related to publication outcomes is designed to
support the user’s decision in which journal to publish and to serve as a more in-
formative guide on wider journal performance using research indicators. In later itera-
tions, the user’s profile and past publications may factor in to display personalized
suggestions.
6 Future Directions for Research
As previously noted, we found very little formal research published specifically on
journal recommendation services like Manuscript Matcher. So, the field is wide open
to future investigations into various dimensions including: their effectiveness, tech-
nical and algorithmic approaches, user perceptions and satisfaction. During initial
development, we did interview users and their feedback informed technical and de-
sign decisions in the current tool. We plan to continue to collect user input through
the existing interface as well as perform more in-depth interviews of users from a
wide-range of academic domains.
From the growing body of research done on online product feedback trends, user
sentiment and motivation, there are parallels and diversions with Manuscript Matcher
that warrant further investigation. The overall Manuscript Matcher feedback skews
positive with 64% selecting “Yes” next to each journal recommendation returned
when asked if it was helpful, a distribution that differs from studies done on optional
product feedback trends. In particular, a large scale assessment of Amazon ratings of
books or CDs showed optional feedback following a U-shaped distribution, with most
ratings either very good or very bad, which was in contrast to controlled experiments,
where opinions on the same items are normally distributed [36]. A notable difference
is that the Amazon users studied were able to provide a star rating out of five, while
Manuscript Matcher users are only able to provide a Yes/ No answer. In future itera-
tions, implementing an A/B test where users are either presented with the option of
providing a rating out of five stars or Yes/ No feedback to see if the different feed-
back options impact 1) the percentage of users that provide feedback and 2) the rat-
ings distribution.
We would also like to further investigate why submissions without citations gar-
nered 66.3% positive feedback, while those with citations received only 52.9% posi-
tive feedback. This was an unexpected outcome, as the inclusion of citations increases
the accuracy of the Manuscript Matcher recommendations, and research done on end
user motivations for providing feedback was consulted.
Altruism is cited as one of the leading reasons for providing both positive and neg-
ative feedback [37]. In the case of Manuscript Matcher, altruism would take the form
of providing positive feedback in order to give the company “something in return” for
a good experience. Altruism could also motivate a user to provide negative feedback
25
BIR 2017 Workshop on Bibliometric-enhanced Information Retrieval
when they want to prevent others from experiencing the problems they encountered as
well as help improve the tool’s accuracy so that future users benefit [38].
Another primary motivation for submitting negative feedback would be anxiety re-
duction, the easing of anger and frustration when faced with results the user views as
either not relevant or substandard [37]. This would likely be prevalent in users who
had entered their title, abstract, and had taken the extra step of providing a curat-
ed EndNote Group with their manuscript citations. More invested in the process and
its outcome, these users submitting citations would likely feel greater levels of dissat-
isfaction when recommendations did not meet or exceed their expectations.
7 Conclusions/Summary
In this paper we describe Manuscript Matcher, a commercial journal recommendation
tool that leverages content-based and bibliometric approaches to recommendations.
We give an overview of the current implementation, briefly compare Manuscript
Matcher to a few similar tools, and analyze current user satisfaction and feedback
from nearly 2,800 users of the system. As well, we discuss plans for future user re-
search and development. In informal tests Manuscript Matcher performed well com-
pared to similar systems but more rigorous and formal study is needed to validate this.
User feedback is largely favorable, with 64% overall positive sentiment, and we hope
to improve recommendation acceptance with both expanded bibliometric approaches
and the addition of collaborative filtering.
Further future development of the tool will utilize new Clarivate Analytics data
points to aid the user. These will include user interface changes to display expanded
data points derived from the JCR. User experience improvements will support the
manipulation of results through the application of data filters as well as targeted and
eventually personalized recommendations. The end goal of these additions is to pro-
vide as much relevant data as possible to ensure that Manuscript Matcher users are
able to make the most informed decision in the journal submission process.
References
1. Beel, J., et al. Research Paper Recommender System Evaluation: A
Quantitative Literature Survey. in Proceedings of the International
Workshop on Reproducibility and Replication in Recommender Systems
Evaluation. 2013. ACM.
2. Beel, J. and S. Langer, A Comparison of Offline Evaluations, Online
Evaluations and User Studies in the Context of Research-Paper
Recommender Systems. Research and Advanced Technology for Digital
Libraries, 2015. 9316: p. 153-168.
3. Medvet, E., A. Bartoli, and G. Piccinin. Publication Venue Recommendation
Based on Paper Abstract. in 2014 IEEE 26th International Conference on
Tools with Artificial Intelligence. 2014. IEEE.
26
BIR 2017 Workshop on Bibliometric-enhanced Information Retrieval
4. Luong, H., et al., Publication Venue Recommendation Using Author
Network’s Publication History, in Intelligent Information and Database
Systems: 4th Asian Conference, Aciids 2012, Kaohsiung, Taiwan, March 19-
21, 2012, Proceedings, Part III, J.-S. Pan, S.-M. Chen, and N.T. Nguyen,
Editors. 2012, Springer Berlin Heidelberg: Berlin, Heidelberg. p. 426-435.
5. Yang, Z. and B.D. Davison. Venue Recommendation: Submitting your Paper
with Style. in 2012 11th International Conference On Machine Learning And
Applications. 2012.
6. Pham, M.C., Y. Cao, and R. Klamma. Clustering Technique For
Collaborative Filtering And The Application To Venue Recommendation. in
Proceedings of I-KNOW 2010. 2010. Citeseer.
7. Garfield, E., The History And Meaning of the Journal Impact Factor. Journal
of the American Medical Association, 2006. 295(1): p. 90-93.
8. West, J.D., T.C. Bergstrom, and C.T. Bergstrom, The Eigenfactor Metrics
(Tm): A Network Approach to Assessing Scholarly Journals. College &
Research Libraries, 2010. 71(3): p. 236-244.
9. Shapira, B., et al., Recommender Systems Handbook. 2011, Springer.
10. Atanassova, I., M. Bertin, and P. Mayr. Mining Scientific Papers for
Bibliometrics: A (Very) Brief Survey of Methods and Tools. in 15th
International Conference of the International Society for Scientometrics and
Informetrics (ISSI) on Scientometrics and Informetrics. 2015. Bogazici Univ,
Istanbul, Turkey: International Society for Scientometrics and Informetrics
(ISSI).
11. Clarivate Analytics. Web of Science Homepage. 2017 January 31, 2017];
Available from: http://wokinfo.com/.
12. West, J.D., I. Wesley-Smith, and C.T. Bergstrom, A Recommendation
System Based on Hierarchical Clustering of an Article-Level Citation
Network. IEEE Transactions on Big Data, 2016. 2(2): p. 113-123.
13. Wesley-Smith, I., C.T. Bergstrom, and J.D. West. Static Ranking of
Scholarly Papers Using Article-Level Eigenfactor (ALEF). in 9th ACM
International Conference on Web Search and Data Mining 2015.
14. White, H.D., Bradfordizing Search Output - How It Would Help Online
Users. Online Review, 1981. 5(1): p. 47-54.
15. Mayr, P., An Evaluation of Bradfordizing Effects. Collnet Journal of
Scientometrics and Information Management, 2008. 2(2): p. 21-27.
16. Akbar, M., et al., Recommendation Based on Deduced Social Networks in an
Educational Digital Library, in 2014 IEEE/ACM Joint Conference on
Digital Libraries. 2014. p. 29-38.
17. Clarivate Analytics, Global Publishing: Changes in Submission Trends and
the Impact on Scholarly Publishers. 2012.
18. Azer, S., D. Dupras, and S. Azer, Writing For Publication In Medical
Education In High Impact Journals. Eur Rev Med Pharmacol Sci, 2014.
18(19): p. 2966-81.
19. Northam, S., et al., Nursing Journal Editor Survey Results To Help Nurses
Publish. Nurse Educator, 2014. 39(6): p. 290-297.
27
BIR 2017 Workshop on Bibliometric-enhanced Information Retrieval
20. Kang, N., M.A. Doornenbal, and R.J.A. Schijvenaars, Elsevier Journal
Finder: Recommending Journals For Your Paper, in Proceedings of the 9th
ACM Conference on Recommender Systems. 2015, ACM: Vienna, Austria.
p. 261-264.
21. Anderson, K., Editorial Rejection — Increasingly Important, Yet Often
Overlooked Or Dismissed, in The Scholarly Kitchen. 2012.
22. Bol, L. and D.J. Hacker, Publishing In High Quality Journals: Perspectives
From Overworked And Unpaid Reviewers. Journal Of Computing In Higher
Education, 2014. 26(1): p. 39-53.
23. Uysal, H.H., The Critical Role of Journal Selection in Scholarly Publishing:
A Search for Journal Options in Language-Related Research Areas and
Disciplines. Journal of Language and Linguistic Studies, 2012. 8(1): p. pp.
50-95.
24. Rollins, J., et al., Systems, Methods, and Software for Manuscript
Recommendations and Submissions. 2013.
25. Springer International Publishing. Springer Journal Suggester. January 31,
2017]; Available from: http://journalsuggester.springer.com/.
26. The Biosemantics Group. Journal / Author Name Estimator. 2017; Available
from: http://jane.biosemantics.org/.
27. SJFinder. SJFinder Recommend Journals. 2017; Available from:
http://www.sjfinder.com/journals/recommend.
28. Research Square. Journalguide. 2017; Available from:
https://www.journalguide.com/.
29. Edanz Editing. Journal Selector. 2017; Available from:
https://www.edanzediting.com/journal-selector.
30. Kim, S.C., et al., Improvement of Collaborative Filtering Using Rating
Normalization. Multimedia Tools and Applications, 2016. 75(9): p. 4957-
4968.
31. Yu, X., et al. Recommendation In Heterogeneous Information Networks With
Implicit User Feedback. in Proceedings of the 7th ACM Conference on
Recommender Systems. 2013. ACM.
32. Beierle, F., J. Tan, and K. Grunert. Analyzing Social Relations For
Recommending Academic Conferences. in Proceedings of the 8th ACM
International Workshop on Hot Topics in Planet-Scale Mobile Computing
and Online Social Networking. 2016. ACM.
33. Johnson, R. and T. Zhang, Effective Use of Word Order for Text
Categorization with Convolutional Neural Networks. Arxiv Preprint
Arxiv:1412.1058, 2014.
34. Mikolov, T., et al., Efficient Estimation of Word Representations in Vector
Space. Arxiv Preprint Arxiv:1301.3781, 2013.
35. Clarivate Analytics. Highly Cited Researchers. 2017; Available from:
http://hcr.stateofinnovation.com/.
36. Hu, N., P.A. Pavlou, and J.J. Zhang, Can Online Word-of-Mouth
Communication Reveal True Product Quality? Experimental Insights,
28
BIR 2017 Workshop on Bibliometric-enhanced Information Retrieval
Econometric Results, and Analytical Modeling. SSRN Electronic Journal,
2006.
37. Sundaram, D.S., K. Mitra, and C. Webster, Word-of-Mouth
Communications: A Motivational Analysis. NA - Advances in Consumer
Research Volume 25, 1998.
38. Hennig-Thurau, T., et al., Electronic Word-of-Mouth via Consumer-Opinion
Platforms: What Motivates Consumers to Articulate Themselves on the
Internet? Journal of Interactive Marketing, 2004. 18(1): p. 38-52.
29