This paper discusses the potential for machine learning predict the quality of scholarly documents to help rank them in information retrieval systems. Quality-based rankings may help users without the time or expertise to assess the value of the publications suggested by a system. It is argued that systems to learn the quality of documents with a degree of accuracy may be possible from the increasing availability of reviews and scores online.
A key feature of scholarly information retrieval systems is their ranking algorithms. Users
may focus on the first documents that they see, unless they are attempting a comprehensive
review. The use of citation information to rank scholarly search results is arguably appropriate
for academics because citations are an obvious, but partial, indicator of scholarly uptake or
utility. A document that has been cited a lot is very likely to have been read by many publishing
researchers and found useful enough to cite. In contrast, end users may be more interested in
applied research. For this goal, citations may be less helpful, especially if they tend to point
to basic or methodological papers rather than practical applications. Their value may also be
undermined by attempts to manipulate them (e.g., [
The main reason why academic articles are not ranked by quality in any mainstream scholarly
database may be that such quality scores are not available for most articles. Both journals and
conferences usually make binary publishing decisions (accept/reject) after reviewing and do
not publish a quality assessment or reviewers’ quality scores. Since there are increasingly many
exceptions (e.g., some open peer review conferences, F1000 post-publication ratings [
Following on from the above, is it possible and desirable to use machine learning to estimate
the quality of an academic article to support ranking in academic information retrieval systems?
One study has predicted proxy-quality scores for articles with machine learning, using journal
impact (split into thirds) as a proxy for article quality. Using this heuristic, it is possible to
generate proxy quality predictions that are substantially above the baseline in some fields, but
not others [
Preliminary unpublished experiments to predict article quality with machine learning applied to tens of thousands of human quality scores (high, medium, low) for articles in 27 Scopus broad ifelds suggest that the highest accuracy is possible for the following biomedical and physical science Scopus broad fields: Multidisciplinary; Biochemistry, Genetics and Molecular Biology; Physics and Astronomy; Chemistry. In contrast, this task is most dificult or impossible in the following Scopus broad fields: Engineering; Agricultural and Biological Sciences; Psychology; Social Sciences; Environmental Science; Energy; Arts and Humanities; Dentistry; Nursing; and Pharmacology & Toxicology. Thus, the potential for harnessing machine learning for article quality prediction may be restricted to the biomedical and physical sciences.
Based on previous studies on predicting citation counts [
• Journal impact thirds, quartiles or other groupings can be used as the target of a machine learning system in fields in which journal impact is a reasonable indicator of article quality (medicine, health, physical sciences, economics, psychology) but not in areas where citations have little value (engineering, other social sciences, arts and humanities). This could be replaced by post publication or peer review scores when they become available in suficient numbers. If this replacement is made, then a journal impact indicator could become an input. • Machine learning should be applied to data segmented into narrow coherent fields to give the algorithms the chance to learn field-specific quality patterns. • Inputs should be field and year normalised (e.g., not citation counts but normalised variants such as the Mean Normalised Citation Score (MNCS) or the Mean Normalised Log-transformed Citation Score (MNLCS)) so that related fields and years can be combined to gain suficient training data. • Valuable types of inputs include all those shown to associate with citation rates, including: (normalised) article citations, number of authors, number of institutional afiliations, article length, number of country afiliations, career publishing statistics of the authors, and abstract readability. • Text inputs, such as words and phrases used in the article title and abstract may reveal important topics, which are more relevant to citations than quality. They may also point to high quality methods (e.g., randomised control trials) and identify more subtle indicators of high-quality work, such as appropriate hedging or shared data/code. If full text can be analysed, then factors like the number of figures and tables in a paper may be useful in judging the amount of evidence supporting the article in some fields.
The above approach is clearly quite citation-dependant but at least moves one step away from a pure reliance on citations.