Chapter 6
Machine Learning Classifiers for Protein-
Protein Interaction Prediction
6.1 Introduction
In the previous chapter, I reported that the PPIs predicted by GN, GC, ES, PP and
GM methods show marginal overlap and hence they can complement each other.
Therefore, integration of these methods is likely to provide better information
than the single method alone. The integrative approach requires the tools and
methods capable of transforming the heterogeneous scores generated by these
prediction methods into meaningful combination. On the basis of rich
mathematical/statistical foundation, Machine Learning Classifiers (MLCs) allow
us to go beyond a mere linear score cutoff based predictions and provide hidden
trends of the data in the form of testable models using gold standard data. Then,
one can use these models to predict whether unknown protein pairs belong to
positive or negative category. There are a number of studies showing feasibility
of PPI prediction task using MLCs [54, 80, 88, 90, 91, 167]. It was suggested that
performance of MLCs can be improved by appropriate integration and selection
of features rather than by adding as many as features possible [80-82, 143, 168,
169]. However, previous reports lack critical comparative assessment of various
MLCs and data dependency. Furthermore, the combination of MLCs for
classification problems has been widely used in non-biological sciences but few
studies have evaluated multiple classifiers for PPI predictions [86, 88, 91, 170]. A
majority of these studies have been performed in Yeast as model organism [170-
172]. Some of the issues worth to be explored for E. coli encoded proteins is are:
1) which MLC is best suited for PPI predictions in E. coli, 2) the feasibility of
multiple MLC based PPI predictions and, 3) choice of a appropriate Gold
Standard (GS) dataset for learning.
In this chapter, I have investigated aforementioned issues by predicting
genome-wide PPI networks using seven different MLCs trained on four GS
datasets. The resulting PPI networks provide highly accurate protein-protein
functional interaction map generated to date, which can be used for system level
analysis of Escherichia coli proteome.
6.2 Material and Methods
6.2.1 Gold standard datasets
All MLCs applied to the PPI prediction task in this work are supervised learning
approaches. Therefore, these algorithms require GS dataset for training and
testing. The analysis was carried out using positive GS datasets composed of
Operon, DIP and Complex datasets. Because, PPI prediction methods were able to
strongly discriminate these datasets from the negative examples (Chapter 4,
section ). Considering the highest classification accuracy of Operon and
Complex datasets, it was decided to combine them together to understand the
discriminative power of MLCs on mixture of two different types of PPIs i.e.
functional (co-operonic PPIs) and physical (co-complex PPIs). The negative
examples were randomly chosen from the high confidence negative dataset
generated for testing PPI prediction methods (Chapter 2, section 2.2.2). Each
positive dataset was combined with a negative subset of size five times greater
than the number of positive examples. This was done to predict physiologically
meaningful PPIs whereas performance evaluation of MLCs was carried out using
positive and negative dataset of equal size to get fair judgment of prediction
accuracy.
6.2.2 Data feature encoding
The interaction scores (i.e. features) for all possible pairs of E. coli proteins
(including gold standard protein pairs) were calculated by GN, GC, ES, PP and GM
methods. There are two possible ways for encoding these scores for machine
learning. First, the scores for a particular protein pair generated by the five PPI
prediction methods is represented by a single value called as “Summary” [88].
The same information source is described by the five values called as “Detailed”
[88]. For the present analysis, “Detailed” method of feature encoding was used
due to less number of available features.
6.2.3 Machine learning classifiers
A set of seven MLCs was used in this study which includes Support Vector
Machine (SVM), Decision tress (DT), Random Forest (RF), Naïve Bayes (NB),
Bayesian Network (BN), Neural Network (NN) and Logistic Regression (LR).
LibSVM package (http://www.cs.waikato.ac.nz/ml/weka/) was used for SVM
based predictions and WEKA toolbox (http://www.cs.waikato.ac.nz/ml/weka/)
for other machine learning algorithms. The cost and gamma functions of SVM
were optimized using grid.py script provided in LibSVM package. J48 variant of
decision tree was used because it incorporates numerical attributes, allows post
pruning after induction of trees. The RF classifier is based on multiple DTs. A
total of 100 DTs were grown simultaneously where each node uses a random
subset of the features. In order to classify a new instance from an input vector,
each input vector is subjected to analyzed by each of the trees in the forest. The
output decision is based on majority vote over all the trees in the forest. For the
Naïve Bayes classifier, a kernel estimation parameter was switched on. All other
classifiers were applied with default parameters in the WEKA.
6.2.4 Cross-validation and performance assessment of MLCs
The positive and negative GS datasets were randomly divided into five equal
sized datasets. The protein pairs present in positive and negative dataset are
referred as true positives (i.e. interacting) and true negatives (i.e. non-
interacting). We performed five-fold cross-validation on all the MLCs using the
above mentioned gold standard datasets. In each round of cross-validation, we
used a random combination of four datasets for training the MLC and the
remaining dataset was used for testing the model. This process was repeated for
five times by using different combinations of “training” and “testing” datasets.
For each round, we calculated the sensitivity (or TPR), specificity, and positive
predictive value (or PPV or precision) as a measure of the quality of binary (two-
class) classifications using each MLC. The equations for sensitivity, PPV measures
have been described in the chapter 2, section
FPTN
TNySpecificit
+=
For all performance measures, TP is the number of positive dataset protein pairs
that are correctly predicted as interacting by given MLC. FN is the number of
positive dataset protein pairs that are incorrectly predicted as not interacting.
Similarly TN is the number of negative dataset protein pairs that are correctly
predicted as not interacting. FP is the number of negative dataset protein pairs
that are incorrectly predicted as interacting. The average performance of five
rounds of cross-validation was taken as a measure of prediction performance.
The TPR and FPR were calculated using the decision values for protein pairs
generated by SVM. For other MLCs, TPR and FPR values generated by WEKA
were obtained. The TPR and FPR values were used to plot ROC curves.
6.2.5 Genome-wide PPI prediction
The models generated by each of the seven MLCs using aforementioned four gold
standard datasets were used for genome-wide PPI predictions in E. coli. The
resulting 28 PPI networks were used to further analysis. The topological
properties of the networks were calculated as described in the chapter 5, section
5.2.2.
6.2.6 Comparison of predicted PPI networks with experimental and
functional PPI datasets
GSPPINet
GSPPINetJC GSPPINet
U
I*100),( =
. The gold
standard representing experimentally characterized and functional PPIs are
same as described in Table 4.1.
6.3 Results and Discussion
The 28 whole genome PPI maps of E. coli were reconstructed using models
generated by seven MLCs on five data features that are interaction scores
generated by GN, GC, ES, PP and GM. In the chapter 4, it was found that these
prediction methods were able discriminate Operon, Complex and DIP PPIs from
negatives with very high accuracy (Figure 4.1 & 4.2). Hence, these three datasets
were used as gold standard (GS) positive datasets to generate MLC models along
with a fourth GS called as Complex_Operon, which was a union of Operon and
Complex PPIs. The performance of Operon and Complex dataset was observed
relatively better than that of other datasets. It was assumed that their
combination would provide enough training examples of functional (Operon)
and physical (Complex) PPIs to generate better MLC model for predictions.
6.3.1 MLCs have predicted PPI networks with very high accuracy
A combination of five data features was used to train and test seven MLCs using
four GS datasets. Performance measures were averaged over five-fold cross-
validation for each MLC. As shown in the Figure 6.1, ROC curves for MLCs trained
on four GS have reported extremely good sensitivities/TPRs at the cost of less
than 0.05 FPR (i.e. 5%). It suggests that only 5% predictions are likely to be false
predictions. At the cost of 5% FPR, MLCs were able to discriminate 65, 80, 95 and
85 percent of PPIs (True positives) of DIP, Complex, Operon and
Complex_Operon from negative training examples (Figure 6.1). One of the
previous studies on PPI predictions in Yeast have reported that RF consistently
ranked as one of the top two classifiers [88]. Figure 6.1 depicts that the
performance of RF indeed better than other six classifiers using three out of four
GSs evaluated in this study. NB is the best performer when DIP GS was used for
training MLCs.
Figure 6.1 Performance accuracy measured as ROC curves for seven machine learning classifiers trained on four gold standard datasets. Each
solid line on plots depicts the machine learning classifier (MLC). The colors of the lines correspond to the Bayesian Network (BN), Decision tress (DT), Logistic Regression (LR), Naïve Bayes (NB), Neural Network (NN), Random Forest (RF), and Support Vector Machine (SVM) classifier. MLCs were trained on gold standard dataset from A) DIP database B) co-complex PPIs C) Co-operonic PPIs D) union of (B) and (C) PPIs. Performance of all MLCs using four gold standard dataset is elegant. RF classifier outperformed on (B), (C) and (D) gold standard datasets whereas NB on DIP. Note that y-axis starts at 0.4 TPR while x-axis ends at 0.05 FPR.
Another independent study had been performed on SVM based PPI
predictions in E. coli by Yellabiona and coworkers [54]. Authors have reported
average fivefold cross-validation accuracy of SVM trained on GN, GC and PP
features using Complex dataset of 0.89 (i.e. average of sensitivity and specificity).
The present study have performed on five data features which include scores
generated by ES and GM in addition to above mentioned three features. However,
the accuracy, which is obtained in the present study, is very similar to the
reported by these authors. It was expected that the performance accuracy of
classifiers would excel by two features additional features as compared to the
three used by them but it didn’t. One possible explanation for the similar results
could be the composition of negatives and positives in their GS dataset. They
have used 13 times higher numbers of negative examples as compared to the
positives for predictions, which raised specificity of their cross-validation
analysis to one as compared to the 0.97 achieved in this study. The sensitivity of
0.79 was achieved by them was preceded by 0.81 achieved in this study [54].
To a large extent ROCs are insensitive to absolute number of false
positives since the calculations are solely based on ranks of positives relative to
those of negatives [167]. So the ROCs alone can be misleading if absolute
numbers of false positives become relevant. The possible number of all protein
pairs among the proteins of any organism is too high as compared to the
estimated PPIs . Out of 8 million possible pairs of E. coli proteins, the
estimated number would be close to 40,000 probable interacting candidates only,
thereby the absolute number of false positives equally matter. In order to ensure
the quality of the final PPI predictions, PPV or precision and specificity of fivefold
cross-validation was calculated and has shown in Figure 6.2. The analysis of
Figure 6.2 A, reflects the outperformance of NB on all GS datasets and only two
percent of negative examples were predicted as potential interacting by NB. The
second best performance is observed for BN followed by SVM classifier.
Specificities of NB trained on four GS lead to observations similar to PPV plot
(Figure 6.2 B).
Although, the performance of RF classifier measured as ROC is better than
other classifiers, ability to detect negatives is substantially higher for the NB.
Hence, NB predictions are biologically more significant than RF by considering
the total space of possible protein pairs among proteins of an organism as
compared to the estimated numbers of actual PPI.
Figure 6.2 Performance accuracy measured as specificity and positive predictive value for seven machine learning classifiers trained on four gold standard datasets. Each bar on plots depicts the average of five-fold cross-
validation accuracy of machine learning classifier trained on corresponding gold standard dataset. The machine learning classifiers are Bayesian Network (BN), Decision tress (DT), Logistic Regression (LR), Naïve Bayes (NB), Neural Network (NN), Random Forest (RF), and Support Vector Machine (SVM) classifier. A) Accuracy measured as the specificity B) Accuracy measured as the positive predictive value. Performance of NB classifier trained on four gold standard datasets is highest as compared to the other classifiers. Note that y-axis starts at 85.
6.3.2 Numbers of PPIs in the networks predicted by various MLCs
varied greatly
A total of 28 PPI networks were predicted using model generated by seven MLCs
trained on four datasets consists of . The number of
PPIs predicted by each classifier and overlap has shown in the Table 6.1. The
numbers of PPIs predicted have shown substantial differences. A total number of
interactions among the proteins present in an organism are difficult to
determine by experimental method if not impossible. The range of 16,000 to
37,000 PPIs for Yeast proteome of ~6300 proteins was estimated by two
independent computational analyses . To best of our knowledge such
analysis has not been carried out in E. coli. Unlike Yeast, E. coli lacks sub-cellular
compartments and the number of proteins is 4132 suggests that the number of
interacting proteins may be in the same range. However, as shown in Table 6.1,
15 out of 28 predicted PPI networks by seven MLCs using four gold standard
datasets contains number of interactions more than 100 thousands whereas
numbers of interactions in remaining networks range from ~44,004 to 87,277.
Table 6.1 A statistics on the numbers of protein-protein interactions predicted by seven Machine Learning Classifiers (MLCs) trained on four different gold standard datasets DIP BN DT LR NB NN SVM RF
BN 225214 142329 85211 76773 104033 162323 112024
DT - 178111 83090 74976 102833 160398 114243
LR - - 108399 56412 93632 92254 88439
NB - - - 78956 59374 76059 61652
NN - - - - 163835 130807 134379
SVM - - - - - 190788 151070
RF - - - - - - 301174
Complex BN DT LR NB NN SVM RF
BN 102871 64661 41589 51675 57493 49669 63032
DT - 242485 52883 39048 87227 71679 152710
LR - - 80506 31997 73669 63791 59846
NB - - - 57806 43775 42900 42587
NN - - - - 156718 92134 100365
SVM - - - - - 106279 84983
RF - - - - - - 284619
Operon BN DT LR NB NN SVM RF
BN 45828 27841 19961 22449 27105 27781 26823
DT - 107462 66190 16900 67753 69531 50180
LR - - 80459 16324 56387 50732 50354
NB - - - 44004 22195 19485 27089
NN - - - - 87277 60267 53740
SVM - - - - - 85630 52094
RF - - - - - - 212703
Complex_Operon BN DT LR NB NN SVM RF
BN 69963 46460 31210 35080 32906 48146 37271
DT - 158500 51275 35510 65752 110627 68758
LR - - 63881 25291 46238 51636 52985
NB - - - 53541 28233 39489 33893
NN - - - - 75214 60430 55556
SVM - - - - - 84413 68309
RF - - - - - - 236841
Notes: The MLCs are Bayesian Network (BN), Decision tress (DT), Logistic Regression (LR), Naïve Bayes (NB). DIP, Complex, Operon and Complex_Operon are gold standard protein-protein interaction datasets. The numbers with bold face represent a total number of PPI
predicted by corresponding classifier. NB predicts least numbers of PPIs whereas RF predicts higher numbers.
Majority of MLCs predicted more than 100 thousand interactions using
Complex and DIP gold standard datasets. Though RF classifier has outperformed
using three GS datasets but the predicted number of interactions using each
model was more than 200 thousand. These numbers are far higher than the
estimated numbers and hence might be consist of many false predictions. On the
other hand, NB has predicted 78956, 57806, 44004 and 53541 PPIs using DIP,
Complex, Operon and Complex_Operon GS datasets respectively. These numbers
are quite close to the estimated PPIs in an organism and considering their small
numbers likely to have many potential interactions.
6.3.3 Functional coverage of the predicted networks is very less
The 28 PPI networks were reconstructed using classifiers that have given
fivefold cross-validation PPV of more than 85%. It means that the 85% of the
predicted PPIs using these models/classifiers are expected to be true
interactions. The coverage of various known PPIs in these networks is one of the
options to validate training accuracies of the MLCs. Plots in the Figure 6.3 show
the coverage of 12 datasets representing known experimental (six sets) and
functional (six sets) PPIs in the form of boxplots/Wisker’s plots. Boxplots
represent differences between groups without making any assumptions of the
underlying statistical distribution and hence they are non-parametric. The
spacing between the different parts of the box indicates the degree of dispersion
and skewness in the data. Here groups are various MLCs and each box on the plot
represents distribution of Jaccard coefficients (JC) between the predicted PPIs by
corresponding MLC and every known PPI datasets. JC represents the coefficient
(scaled in between 0-100) of similarity between two datasets taking into account
the size difference.
Even though MLCs have shown very high sensitivity values, coverage of
known PPIs in the predicted networks is very less (Figure 6.3). The training on
various types of PPIs is also not reasonable since the coverage of known
experimental (Figure 6.3 A) and functional (Figure 6.3 B) PPIs in the networks
predicted using DIP, Complex and Operon GS is almost similar. Every known PPI
dataset is underrepresented in the predicted networks, irrespective of the
training gold standard datasets and MLC used for prediction. JC have not risen
above 5 for a single classifier which was scaled in between zero and 100 (Figure
6.3). JC score of zero here reflects no overlap between two PPI networks while
Figure 6.3 Coverage of 12 known experimental and functional protein-protein interaction datasets in the networks, predicted by seven machine learning classifiers trained on four different gold standard datasets. Each boxplot on plots depicts the distribution of Jaccard similarity Coefficients
calculated between network predicted by corresponding machine learning classifier and 6 datasets of known physical and functional protein-protein interactions (PPIs). The machine learning classifiers are Bayesian Network (BN), Decision tress (DT), Logistic Regression (LR), Naïve Bayes (NB), Neural Network (NN), Random Forest (RF), and Support Vector Machine (SVM). A) Coverage of experimentally characterized PPI datasets which include His-tagged bait, TAP-tagged baits (2 sets), co-presence of proteins in the same protein complex, DIP database and synthetic lethality analysis. B) Coverage of functional PPI datasets which include protein pairs that belong to the same KEGG pathway, GO term, COG functional category, Operon, EcoCyc functional category and transcriptional regulatory associations. Coverage of experimental as well as functional datasets is higher in the networks predicted by NB classifier using all gold standard training datasets. Every known PPI dataset is underrepresented in the network predicted by different MLCs.
score 100 means complete overlap. It is consistent with the previous
observations that a computational PPI prediction complements the existing
knowledge of PPIs [54, 56, 86, 173, 174]. It was also observed that coverage of
known indirect interactions is better than the direct PPIs in the predicted
networks [54]. Here, indirect interaction is defined as link between two proteins
via its adjacent neighbors in the network rather than their direct interaction.
Therefore, the marginal coverage of known PPIs in the networks predicted in
present study is not an artifact but the limited available knowledge of biological
systems.
Unexpectedly, the coverage of experimental PPI datasets is relatively
higher in the networks predicted by NB classifier (Figure 3A). These
experimental PPI datasets include PPIs derived from His-tagged bait, TAP-tagged
baits (2 sets), co-presence of proteins in the same protein complex, DIP database
and synthetic lethality analysis. The coverage of these datasets in the NB
networks has observed substantially higher than that of RF classifier. RF was the
best performer in terms of ROC performance measure using three GS. It reflects
the shortcoming of routinely used ROC performance measure for PPI predictions.
As suggested by Park, Y, the ROC measure should be complimented with other
performance measure to ensure the percentage of positives among the predicted
PPIs [167]. The present study suggests the PPV and specificity plots can also be
useful for such cross validation. NB has outperformed when performance was
measured as PPV and specificity alone.
The coverage of available functional PPI datasets also led to the similar
observations (Figure 6.3 B). The coverage of functional PPI datasets is slightly
higher than the experimentally derived ones but still all JC scores are below 5
(Figure 6.3 B). Furthermore, the distributions of JCs for various MLCs have
differed comparably. Again, NB predicted PPI networks have higher coverage of
functionally linked proteins as compared to the classifiers except BN (Figure 6.3
B). BN also have relatively better coverage of the functional PPI datasets under
consideration. BN represents the probabilistic relationships between data
features/variables for the corresponding labels/evidence. Given data features,
the Bayesian network can be used to compute the probabilities of the label
associated with the data feature. NB is a special case of BN with strong
independence (naïve) assumptions. Hence these results suggest that the
classifiers based on the probabilistic models are better predictor of PPIs than the
other classifiers used in this study such as RF, SVM etc. In the literature, such
approaches have been indeed favored as compared to the other classifiers for
PPI predictions [86, 174-176].
6.3.4 Networks predicted by NB classifier are better than the
previous reports
With the very limited knowledge of the total PPIs existing in the E. coli, it is
challenging to compare these predictions with previously reported
computational predictions. Furthermore, not only methodology differs with each
report but also the GS used for evaluation. Hence, the direct comparison of
various approaches that are used so far is often difficult [87]. In addition,
performance of prediction often biased towards the GS, which is used for
validation of predictions. Considering the combination of five PPI predictions (i.e.
data features), multiple MLC models using four GS datasets, it was expected that
the predicted PPIs would be of high quality with respect to false predictions and
the coverage of proteins.
In the light of afore mentioned concerns, it was decided to use three
datasets that are not related to MLC training in the present study. At the same
time, these datasets are likely to be free from experimental noise. The first two
datasets consists of PPIs that are detected by the systematic large-scale tandem-
affinity purifications to isolate multiprotein complexes by two independent
studies [86, 109, 110]. The experimental set up, which was used by authors,
minimizes the spurious non-specific protein associations and enables recovery of
native protein complexes at near-endogenous levels. Third dataset was created
using pairs of proteins that have been associated with each other in at least two
experimental analysis. Then, percentage of these datasets was calculated in the
networks predicted in the present study and also for the four computational
genome-scale PPI networks reported in the literature [54, 81, 82, 86]. Most of the
networks predicted by various MLCs had limited coverage of these known
datasets as compared to the networks reported in previous studies. Some of
these networks had higher coverage, which may be likely due to the larger size of
the predicted networks. Some of the networks such as, NB and BN predicted
networks had much better coverage of these experimentally derived PPIs as
compared to the existing networks. The comparisons of these networks have
shown in the Figure 6.4. The size of these networks is comparatively less than
that of existing genome-scale PPI datasets but the coverage of experimentally
derived PPIs is substantially better than them (Figure 6.4).
Figure 6.4 Comparison of the predicted networks with previously published PPI networks. X-axis represents computationally predicted genome-scale
protein-protein interaction (PPI) networks. Networks with asterisk marks are predicted in the present study. Each bar represents percentage of a corresponding experimentally characterized PPIs in the predicted networks. Exp2 dataset represents PPIs that have been characterized by at least two different experimental analyses. The networks predicted in the present study have substantially high coverage of three PPI datasets used evaluation. Only network predicted by Yellabiona and coworkers is close to the predicted networks in this study. cNB, oNB and coNB stands for networks predicted by models generated by Naïve Bayes classifier using Complex, Operon and a union of complex & operon as a positive gold
standard dataset. Similaraly, coBN network is predicted by Bayesian Network using a union of complex & operon as a positive gold.
6.4 Summary
The present study describes the prediction of genome-wide PPI networks using
seven MLCs trained on four different GS datasets. It was observed that the
combination of the confidence scores generated by five PPI prediction methods
effectively increased classifiers accuracy. Majority of classifiers statistically show
higher training accuracies but the predicted networks lack coverage of known
PPIs. A probabilistic model based NB classifier, which assumes independence of
data features recovers substantially higher numbers of known PPIs. These
results led me the conclusion that probability based classifiers, which assume
feature independence, are best suited for PPI predictions in E. coli. The reason
behind their better coverage of known PPIs could be independent assumptions
of the five methods for PPI predictions. Thereby, each feature have
complemented others and enabled recovery of biologically meaningful
interactions, when NB classifier was used.
A combination of five PPI prediction methods, seven MLCs, and four GS
datasets, led to the prediction of 28 networks with size no more than 3,20,000
PPIs. Majority of the networks consist of PPIs in the range between 1,00,000-
2,00,000. These comparisons have encouraged me to speculate that a total
number of PPIs in bacterial cells could be around 1,50,000 which is much higher
than the previous estimates.