There is a high potential for data leakage in biomedical machine learning applications since biomedical data resources share, reuse and import data from each other routinely. We have investigated potential data leakage in the prediction of protein-protein interactions using the Gene Ontology knowledge graph, by comparing the performance of models trained and tested on the same versions of data versus training on archived data and predicting only for newly discovered protein interactions. Our results were not able to detect an in uence of data leakage, indicating that if this problem exists, its magnitude is not a ecting the performance of knowledge graph-based protein interaction predictions.
Machine learning methods have become a signi cant trend in several research
elds in recent years, and the semantic web is no exception. As machine learning
is increasingly being used, concerns about data leakage have been raised [
In biomedical applications, such as protein-protein interaction (PPI)
prediction, data leakage can also be an issue. It is not uncommon that multiple
databases and resources reuse the same sources of information. The majority of
PPI prediction methods that are based on knowledge graphs (KGs) [
1Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
We hypothesize that if this type of data leakage is common, then the performance of GO-based PPI prediction methods would be arti cially increased. To test this hypothesis, we compare PPI prediction models trained on older GO data and PPI interactions and tested on previously unknown interactions captured in more recent versions of STRING with same version training and testing. Furthermore, by training the models on labeled examples from the past, we more closely simulate real-world applications. 2
PPI prediction is cast as a classi cation task that takes as input the GO KG and a set of protein pairs. The rst step of our approach is using historical data to build the PPI datasets. Then we use the GO KG and the protein pairs to predict interactions using several machine learning algorithms. 2.1
The PPI datasets were obtained from the STRING Database1 which is one of the largest available PPI databases that integrates both physical interactions as well as functional associations between proteins collected from several sources. We considered the following criteria to select protein pairs from STRING: (i) each protein must be annotated with the GO; (ii) protein interactions must be experimentally determined or from curated databases (as opposed to computationally determined); (iii) interactions must have a con dence score above 950 to retain only high con dence interactions. We employed random sampling to create negative pairs composed of the human proteins present in the positive pairs but without any STRING interactions between them, building a balanced dataset.
We built several PPI datasets using three archived versions of the STRING database (v9.1, v10, and v10.5) and the current version (v11). For the current version, we created three datasets each excluding protein pairs present in each of the older versions (see Table 1). Regarding the GO KG, we obtained archived versions of the GO and GO annotations in 2015, 2017 and 2019 from the Gene Ontology Data Archive2. 2.2
We follow the setup in [
Table 1. Number of positive pairs in each version of the STRING database. of machine learning models to train classi ers using the scikit-learn library: Knearest neighbor (KNN), genetic programming (GP), decision tree (DT), XGBoost (XGB), random forest (RF), and multi-layer perceptron (MLP). The classi cation performance was evaluated using the weighted average of F-measures (WAF). 3
We conducted two types of experiments: (i) Same version, where we train the model with randomly chosen 10 000 protein interacting pairs from the archived STRING version and test it with the remaining pairs; (ii) Future version, where we train the model with randomly chosen 10 000 protein pairs from the archived STRING version and test it on data from the current STRING version (excluding interactions present in the archived version). The same randomly chosen 10 000 protein pairs are used in both settings.
Since we used three archived versions, the Future version experiments also allow us to measure the impact of using increasingly older versions of STRING and GO in training. Table 2 shows no substantial di erences between Same version and Future version experiments.
The results do not support a clear indication for data bias. While for the 2019 version, it is always slightly easier to predict future PPIs, this is reversed in the 2017 version, and varies between methods for the 2015 version, so no clear trend is discernible. The median weighted F-measure for the Same version experiments is 0.844, while it is 0.845 for the Future version (see Figure 1).
In addition to not detecting data leakage, the results also indicate that the relation between the functions of a protein and its interactions do not fundamentally change over time. Even for more recently discovered interactions that can be biologically di erent, protein functions are still a good predictor of PPIs. 4
Biomedical data resources share, reuse and import data from each other routinely. This can be a potential source of data leakage for machine learning applications. We investigated potential data leakage between the GO KG and the STRING database in the task of PPI prediction, by comparing performance on unseen interactions using archived data. Our results were not able to detect an
CP, SS, RTS are funded by the FCT through LASIGE Research Unit, ref. UIDB/00408/2020 and ref. UIDP/00408/2020. CP and RTS are funded by project SMILAX (ref. PTDC/EEI-ESS/4633/2014), SS by projects BINDER (ref. PTDC/CCIINF/29168/2017) and PREDICT (ref. PTDC/CCI-CIF/29877/2017), and RTS ML KNN GP DT XGB RF MLP measure (SSM) and machine learning (ML) algorithm for di erent GO KG version. by FCT PhD grant (ref. SFRH/BD/145377/2019). It was also partially supported by the KATY project which has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 101017453.