The CLEF 2018 Task 2 goal was to identify and rank retrieved articles relevant to conducting a systematic diagnostic test accuracy review on a given topic. The UIC/OHSU team did not attempt to rank retrieved articles by relevance directly, but rather explored the baseline value of ranking retrieved articles according to the probability that they are concerned with diagnostic test accuracy. First, a set of six publication type clusters, including a cluster of diagnostic test accuracy papers (DTAs), was built by searching PubMed from 1987-2015. We created several types of cluster similarity measures for each publication type. Similarity types included: implicit-term similarity, most important word similarity, journal similarity, and author count similarity. These similarity features were then used with weighted and un-weighted linear SVM machine learning algorithms, which were trained with a data set retrieved from PubMed searches consisting of 3481 PMIDS likely to be DTAs, and 71684 PMIDS most of which are not likely to be DTAs. The trained models produce scores predicting the probability that an individual article is a DTA. The CLEF 2018 Task 2 Test PMIDs for each topic were scored and ranked, and the cutoff probability for each of the two models determined by visual inspection of the score distribution on the test data. Cutoff probabilities chosen were 0.20 for the unweighted SVM model and 0.40 for the weighted SVM model.
We participated in Task 2 of the CLEF 2018 e-Health challenge [
Query Search #7 AND #5 NOT #6 Search "diagnostic test accuracy"[ti] OR "diagnostic accuracy"[ti] Search editorial[pt] OR letter[pt] OR "practice guideline"[pt] OR review[pt] Search #1 AND #2 AND #3 AND #4 Search humans[MeSH Terms] Search hasabstract Search ""english""[Language]) OR ""english abstract""[Publication Type]"
Search ""1987/1/1""[Date - Publication] : ""2015/12/31""[Date - Publication]
We have been extending our prior work on probability based tagging for specific
publication types [
The UIC/OHSU CLEF 2018 Task 2 submission applies a machine learning approach to ranking the PMIDs retrieved by CLEF for 20 topics. The approach assigns probabilities to individual PMIDs based the likelihood that they are DTAs. To generate positive training data, likely DTAs were retrieved using the PubMed query shown in Figure 1. No specific information about the topic queries generating the PMID list for each query was used. The system builds a predictive model in stages. First, publication type clusters, including diagnostic test accuracy papers (DTAs), were built by searching PubMed from 1987-2015. Six publication type (PT) clusters were used in this model: DTAs, Randomized Controlled Trials, Cross-sectional Studies, Cross-over Studies, Cohort Studies, and Case-Control Studies. These clusters were used as training data to create several types of cluster similarity measures for each publication type. The PT clusters are treated as consensus profiles that represent the PT as a whole, so any given article is judged to belong to it if it is sufficiently similar in its weighted sum of similarity features. While the members of each cluster are very likely to be examples of the cluster specific publication type, nothing in the method requires all the articles in a cluster to be of that publication type. Somewhat noisy training data is expected.
Similarity types used as features included: implicit term similarity, most important word
similarity, journal similarity, and author count similarity. Implicit term similarity measures
how similar a paper is to a cluster based on terms (words, bigrams, etc.) that commonly
occur with words contained in the papers within each cluster relative to the baseline
frequency across MEDLINE. A cluster “centroid-like” vector is computed as the mean vector
of the individual cluster article vectors, where each article vector consists of the 300
weighted terms most associated to the words in the article. The cluster centroid is limited
to the 300 highest total scoring terms across the cluster. See [
Most important word similarity measures the fraction of words in the paper that are in the
list of most important words computed for each cluster, as measured by the frequency of
the word occurring in that cluster versus MEDLINE as a whole.[
The trained models produce probability scores predicting whether or not an individual PMID is a DTA. The PMID predictions were then organized according to the CLEF 2018 Task 2 topics, and were ranked within a topic by probability. The cut-off probability for each of the two models was determined by visual inspection of the score distribution on the test data. Cutoff probabilities chosen were 0.20 for the unweighted SVM model and 0.40 for the weighted SVM model. This information was combined into the submission qrel files, rank ordering the topic publication PMIDs highest to lowest predicted probability, one file for each model. In this manner we produced two sets of predictions, submitted as two separate runs: OHSU_UIC_LIBLINW for the weighted model, and OHSU_UIC_LIBLINB for the unweighted model. 3
The official overall evaluation results for our systems are shown in Table 1. Across the
board, the liblinear system with inverse class frequency weighting performed slightly better
than the liblinear with bias version. These results are averages across all the topics. Based
on the similar CLEF 2017 task, these results are about median as compared to other entries.
The average precision achieved by the our liblinear weighted system was 0.180, which
would have ranked 14th out of 33 CLEF 2017 entries [
Considering that we only ranked articles according to their probability of being a DTA, and did not evaluate query topic information at all, our approach did have some significant value in identifying articles that are relevant for inclusion in topic-specific systematic reviews. We plan on continuing to work on our system, expanding the number of clusters and publication types, as well as add additional cluster similarity measures. While the current approach uses an SVM in a one-versus-rest approach for multi-classification, we are also experimenting with other classifiers which are more flexible with multiple category classification such as random forests and deep learning neural networks. 0.860 0.935 1.000 0.180 0.846 0.926 1.000 0.174