Predicting the quality of a survey questionfrom its design characteristics
Daniel OberskiDept. of Methodology & Statistics,Tilburg University, The Netherlands
U N I V E R S I T A T
P O M P E U F A B R A
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Question #1: Where is Tilburg?
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Summary
• We wanted to predict how “good” a survey question isbased on its “design characteristics”, i.e. choices such as #categories, DK present/not, use of showcards, etc.(Willem’s talk 1 & 2)
→ Trained a random forest of regression trees to predictthe ESS multitrait-multimethod estimates of reliabilityand method effect (Cf. Melanie’s talk) from these questioncharacteristics, as coded by teams of human coders.
• Prediction powers a webapp allowing users to codequestion & get predictions ± uncertainty (Diana’s talk).
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Simple example
Predicting the quality of a survey question from its design characteristics Daniel Oberski
European Social Survey, 2002
Method A:
3
ENTER START TIME: A1 TvTot CARD 1 On an average weekday, how much time, in total, do you spend watching television? Please use this card to answer.
No time at all
Less than ½ hour
½ hour to 1 hour
More than 1 hour, up to1½ hours
More than 1½ hours, up to 2 hours
More than 2 hours, up to 2½ hours
More than 2½ hours, up to 3 hours
More than 3 hours
(Don’t know)
A2 TvPol STILL CARD 1 And again on an average weekday, how much of your time watching television is spent watching news or programmes about politics and current affairs1? Still use this card. No time at all 00
Less than ½ hour 01
½ hour to 1 hour 02
More than 1 hour, up to 1½ hours 03
More than 1½ hours, up to 2 hours 04
More than 2 hours, up to 2½ hours 05
More than 2½ hours, up to 3 hours 06
More than 3 hours 07
(Don’t know) 88
1 About “politics and current affairs”: about issues to do with governance and public policy, and with the people connected with these affairs.
00 GO TO A3
01
02
03
04 ASK A2
05
06
07
88
Method B:!!""#$%&'()*%)+&#!)&,%$#!-&.# !"#$"#$%&'$(&#)&&*+$,-#./)#012.#340&-#4"#3/3$5-#+/#,/1#67&"+#)$32.4"(#
3&5&%464/"89:##
#
# # # ,$/+%#/)#;!<=>0#### ###?@A#BC@<DE>0# # # #
# # # #
-&1# #!"#$"#$%&'$(&#)&&*+$,-#./)#012.#340&-#4"#3/3$5-#+/#,/1#67&"+#5463&"4"(#3/#
3.&#'$+4/8F:##
#
# # # ,$/+%#/)#;!<=>G## ?@A#BC@<DE>G# # # #
# # # # # #
#
#
-&2# !"#$"#$%&'$(&#)&&*+$,-#./)#012.#340&-#4"#3/3$5-#+/#,/1#67&"+#'&$+4"(#3.&#
"&)67$7&'688:##
#
# # # ,$/+%#/)#;!<=>G# #?@A#BC@<DE>G# #
#
#
#
'34564#7897:5;4#;<#=>5;#4?;48;#@<A#5BC44#<C#9765BC44#=7;>#45:>#<D#;>4#D<33<=78B#6;5;4E48;6F#
# # #
-&G# H>/0&340&6#7/543426#6&&06#6/#2/07542$3&+#3.$3#C#2$"H3#'&$55,#1"+&'63$"+#).$3#
46#(/4"(#/"8IJK#
# '34564#;7:I#<84#J<?F#! A46$('&?'/"(5,# F#
# A46$('&&# 8#
# @&43.&'#+46$('&&#"/'#$('&&# I#
# ?('&&# L#
# ?('&?'/"(5,# M#
#
-&K# NC#3.4"*#C#2/15+#3$*&#$"#$234%&#'/5"#$#('/17#4"%/5%&+#)43.#7/54342$5#4661&68LJK#
# '34564#;7:I#<84#J<?F# #
! A46$('&?'/"(5,# F#
# A46$('&&# 8#
# @&43.&'#+46$('&&#"/'#$('&&# I#
# ?('&&# L#
# ?('&?'/"(5,# M#
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"#!$%%!&'!()!*+()!,-%./(0))+(1%!"'!$%%!&2!""!$%%!&3!"2!$%%!4"!
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Design characteristics for method A
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Design characteristics for method B
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Reliability and common method variance estimatesfrom an MTMM experiment
For the TV watching example:
Reliability Methodvariance
Method A: 8 categories 0.796(0.01)
0.098(0.01)
Method B: write (after outlier del’n) 0.819(0.02)
0.140(0.01)
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Goal of this research:
obtain a rough estimate of the reliability and method effects ofthese kinds of questions without needing amultitrait-multimethod experiment.
Predicting the quality of a survey question from its design characteristics Daniel Oberski
ESS multitrait-multimethod experiments
• In the European Social Survey (ESS), every round aboutsix MTMM experiments are done;
• Each experiment usually estimates the quality for 9questions (Method-Trait combinations).
• Range of topics is reasonably diverse, though factualquestions are underrepresented.
• In total about 1051 method-trait combinations available,3483 “questions” when the analysis is done separately percountry
Predicting the quality of a survey question from its design characteristics Daniel Oberski
“Old” multitrait-multimethod experiments
• In addition to the ESS, an older series of experiments alsoexists (Andrews; Koltringer; Saris; Scherpenzeel, 1990’s)
• These add another 1089 questions for which reliability andcommon method coefficients are estimated
• Combining the two datasets (ESS question qualities andOld experiment qualities, we created a database of 3483questions with their reliability and common methodestimates (1051 unique method-trait combinations).
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Reliability and common method variance estimates of3483 questions (1051 unique method-trait comb’s)
Reliability coefficient
Reliability coefficient
0.4 0.6 0.8 1.0
0
200
400
600
800
Common method variance
Common method variance
0.0 0.2 0.4 0.6 0.8
0
500
1000
1500
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Design characteristics of questions
• Social Desirability• Centrality• Reference period• Question
formulation• WH word used• Use of gradation• Balance of the
request• Encouragement• Showcards
present• Showcards have
pictures• ...
• Emphasis on subjectiveopinion in request
• Information about theopinion of other people
• Use of stimulus orstatement in the question
• Absolute or comparativejudgment
• Response scale: basicchoice
• Number of categories• Labels full, partial, or no• Labels full sentences• Knowledge provided• Survey mode• ...
• Order of the labels• Correspondence between
labels and numbers of thescale
• Theoretical range of thescale
• Neutral category• Number of fixed reference
points• Don’t know option• Interviewer instruction• Respondent instruction• Extra motivation, info or
definition available?• Agree-disagree scale• . . .
(Saris & Gallhofer 2007)
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Coding design characteristics of the 3483 questions
• For each of the 3483 questions in all countries, a team ofcoders coded 40 design characteristics of the question;
• Some codes were automatically generated by NaturalLanguage Processing software (syllables, words, etc).
• Coders were students, assistants to the local coordinatorsof the ESS, and two experts;
• For English source version, experts double-codedquestions independently, then created consensus codes;
• Non-expert codes were quality-controlled by detailedcomparison with consensus codes for the English source;
• In a meeting between the experts and each other coder,the discrepancies were discussed and either corrected orleft in as true differences.
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Domain of question # questions # unique comb’sInternatl politics 64 64Health 244 88Living conditions 433 234Other beliefs 292 292Work 509 119Family 109 7Personal relations 308 62Consumer behavior 26 26Leisure activts 131 63National gvt 169 31Institutions 412 35Political parties 30 30Trade unions 12 12Economy 266 16Other 370 89
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Concept of question # questions # unique comb’s
Evaluative belief 703 136Feeling 1044 161Importance 96 96Expectation 39 30Facts, behavior 111 60Complex concept 44 44Other simple 1338 608
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Meta-analysis dataset
• For each of the 3483 questions, we have in the database:• The estimated quality (reliability and common method
coefficients)• About 50 design characteristics (through hand- and
automatic coding)
• Predict reliability and method effect estimates from designcharacteristics
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Meta-analysis using random forests of regression trees
→
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Example of one (1) regression tree (CART).
Predicting the quality of a survey question from its design characteristics Daniel Oberski
|
domain=3,4,7,11,13,14,112
domain=3
gradation>=0.5 position< 339.5
position>=410
concept=1,2 position< 404.5
concept=1,73,78
position< 322.5
ncategories>=4.5
domain=6,101,103,120
domain=4,7,11,13,14,112
gradation< 0.5 position>=339.5
position< 410
concept=73,75,76 position>=404.5
concept=2,76
position>=322.5
ncategories< 4.5
1.955n=1988
1.724n=1303
0.9636n=108
0.4959n=36
1.198n=72
1.793n=1195
1.642n=722
2.023n=473
1.544n=108
1.28n=76
2.17n=32
2.165n=365
1.97n=217
2.45n=148
2.394n=685
1.489n=138
2.622n=547
2.384n=233
2.799n=314
2.681n=260
3.364n=54
Example regression tree for reliability coefficient
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Obtaining a prediction from a random forest
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Training the forest of regression trees
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Random forest algorithm (Breiman 2001)
1 Randomly sample (w/ replacement) cases (questions)→ some “in-bag” cases;→ Rest is “out-of-bag”.
2 Randomly sample features (characteristics);3 Grow a regression tree on the in-bag cases using the
subset of selected features;4 Calculate mean square prediction error on the out-of-bag
cases→ built-in cross-validation;5 Back to (1) until 1500 trees have been grown.
Predicting the quality of a survey question from its design characteristics Daniel Oberski
joined.data
Ordered by number of missing items
avgabs_intronabst_intro
nnouns_introshowc_boxesshowc_questshowc_startopinionothershowc_horizshowc_lettershowc_over
positionncategories
range.correspondencescal_neutral
absoluteconceptdomain
dont_knowform_basic
futureinstr_interv
instr_responmotivation
pastscale_basic
stimulususedshowcard
interviewervisual
fixrefpointslabels
scale_corresscale_trange
labels_grammcomputer.assisted
intr_requestavgabs_total
avgsy_totalavgwrd_introavgwrd_total
balancecentrality
countryencourage
gradationintropresentknowledge
labels_orderlanguage
nabst_totalnnouns_total
nsub_questnsyll_total
numsub_intronwords_intronwords_totalquestiontype
rel.estrel.est.orig
repetitionscale_urange
socdesirstudy
subjectiveopsymmetry
used_WH_wordval.est
val.est.origfrom
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Missing data• Imputed using multiple imputation with chained equations
(MICE) (van Buuren & Groothuis-Oudshoorn 2011).• A small random forest (bush?) is grown for each imputed
dataset• These bushes are joined to form the final forest• “raimforest” (RAndom IMputation forest)
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Out-of-bag crossvalidation
• R2 for logit(reliability coef.) prediction: 0.64• R2 for logit(validity coef.) prediction: 0.84
Predicting the quality of a survey question from its design characteristics Daniel Oberski
●●
●●●●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●● ●
●
●
●
●●
●
●●
●
●
● ●●●
●●
●
●●
●●●
●
●● ●
●
●●●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●●
●●
●
●
●●●
●
●●●
●
●
●
●
●●
●
●
●
●
●●
● ●
●
●
●
●
●●
●
● ●
●
● ●●
●●
●●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
● ●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●● ●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●● ●●
●
●
●●
●●
●
●●
●
●●
● ●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●●
●
● ●
●
●
●
●
● ●
●
●
● ●●
●●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●●
● ●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
● ●
●●
●
●●
●
●
●
●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●●
● ●●
●
●●
●●●
●
●●
●
●●
●
●●
●●● ●
●●
●
●●
●
●●
●●
●
●
●
●
●
●●
●
●●
●
●
● ●
●●
●●
●
●
●
●●
●
●
●
●● ●
●
● ●
●
●
● ●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●● ●
●●
●
●
●
●
●
●
● ●
●●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●●
●●
●
●●
●
●
●●
●
● ●●
●
●
●●
●●
●
●
●
●●
●●
●●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●●
●
● ●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●● ●
●
●
●
●
●
●●●
●● ●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●●●
●
●
●●
●
●
●
●
●
●
●
●●●
●●
●●
●●
●
●
●●●
●●
●
●●●●
●●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●● ●
●
●
●●●●
●●
●
●●
● ●
●●●
●
●●
●
●●
●
●
●
●●
●
●●●
●
●
●
●●
●
●
●●
● ●
●
●● ●
Quality prediction
Quality predicted from random forests
MT
MM
est
imat
e of
qua
lity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Fitted versus residuals for random forest predictions
Reliability
!
!
!
!
!
!!!!!
!
!!
!
!
!
!!!
!!
!
!!
!
!!!
!
!!!
!
!
!!
!!!
!
!!
!!!
!!
! !
!! !
! !! !
!!
!!!!
!!!! !!!!!
!! !!
!!!!!! ! !! !!!!!! ! !
!
!!!!!!
!
!
! !!
! !!! !! !!!
!!!!!
!
!!!!
!
!!!!
!
!! !!!!!
!! !!!
!!! !! !!!!!!!!
!!
!!!
!
!! !
!
!!!
!
! !!!
!
!
!!!
!
! !!
!!!!
!!!!!!
!!
!
!!
!! !!
!!!
!!
!! ! !
!
! !
!
!
!!
!!!!
!
!!!!
!!!!
!
!!
!!!!! !!!!!
!
!!!!
!!
!
!!
!! !! !!! !!!
!!
! !!
!
!
!
!
!!
!!
!
!
!
!!
!
!
!!
!
!
!!
!
!
!!!
!!
! !!
!
!!
!
!
!!!
!
!
!!!
!
!!
!
!
!!
!!
!!
!
!!! !
!!! !
! !
!!
!!
!!
!
!
!! !!!
!!!
!
!!
!
!!!
!
!!
!
!!!
!
!
!
!
!!
!!!!
!
!!
!!!
! !!!
!! !
!
!!!
!
!!
!
! !!!!
! !!!
!!
!!
!
! !!!
!! !!!
!
!!
!
!!! !
!
!!
!!
!
!!!
!!
!
!
!
!!!!
!!
!!!
!!! !
!!
!
!
!!!
! !
!!!!
!
!! !!
!
!!!!
!
!!!
!
!
!
!
!!!
!
!!!
!
!!
!
!
!!
! ! !!!!! !!!
!!!!
!
!! !!! !! !
!!
!!
!
!
! !!
!
!!
!
!!!
!
!
!
!!
!
!
!! !
!
!!
!
!
!! !! ! !! !! !
!!!!! !
!
!!! !
!!!!!
!
!!!
!!!
!
!!
!!
! !
!!! !!
! !! !!!!
!!
!!! !! !!
!
!!!! !!
!!
!! !!
!
!
!!! !!!!
!!
!! !
!
!
! !!!
!!!!!!
!! !!!!
!
!
!!!
!
!!!
!
!
! !!!
!
!!!!!
!!!!!
!
!!
!! !!
!!!!!!
!!!!
! !!
!
!!
!!!
!!
!!!
!!!!
!
!!
!! !!
!
!!!
!!! !! !
!!!
!
!!
!
!!
!
!!
!!
!!
!
!!
!!!! !
!
!!
!!
!
!!
!
!
!!!!
!
!!! !!!!
!
! !!
!
!!
!
!!
! !!
!
!
!
!
!
!
!
!
!!
!
!
!!!
!
!
!!
!
! !
! !
!
!
!!
!
!!
!
!
!
!
!
!
!!
!
!!!!
!
!!
!
!! !!!
!
!!!!!
!
!!
!
!!
!
!
!!!!
!
!!!
!!
!
!
!
!
!!
!
!!!!
!!!!!!
! !!
!
!!
!!
!!
!
!
!
!
!!!!
!
!
!
!!! ! !!
!!!
!!!!!!!
!
!
!
!!!
!
! !!
!!
!!
!
! ! !!!
!
!
!!
!
!
!!
!!!
!
!
!
!
!!!
!
!
!
!
!!!
!!
!
!!
!
! !
! !
!
!
!!
!
!!!!
!!
!!
!
!!!!!!
! !
!!!
!!! !
!!
!!!!!
!
!!!
!
!!
!
! !!
!
!
!
! !!
!
!
! !!!! !!!
!
!!
!! !
!! !!!! !
!!
!
!!!!
!
!!
!
!! !!
! !!!!
!
! !!!
!! !
!
!
!
!!!!! !!
!
! !!!
!! ! !!
! !!
!!!! !
!! !!! !!! !
!!
! !!!
!!!!! !!
!! !! !!!! !!!!! !
!!!!!
!!
! !!!!
!
!!!
!
!!
! !!!
!
!! !!!
! !!!
!!!!!
!!!!! !!!!
!! !!!
!!
!
!
!!!!! !!!!!!
!
!! !
!
!! !!
!!!
!
!
!
!
!!!!!!
!
!!!!!!!
! !! !! !
! !!!!
!!!
!!!!
!
!
!!! !!
!!!
!
!!!!!!! ! !!!
!!
! !!!
!!!!!!!!! !!
!!!!!
!!!! !!!!!!!!!
!
!!!!!!! !!
!!!!!!!!!! !! !
!!
! !!
!
!!! !!!!!
! !! !! !
! !!
!!!
!!
!
!!!
!!!
!
!!
!!!! !! !!
!!
!
!! !! !!!!
!!! !!
!!
!
!!
!
!!!
!!
!!
!!!
!
!!
!
!!
!!
!!
!
!!!!
!
!!!!!
!
!
!!!! !
!!!! !
! !!
!
!! !! !!!
!
!!
!!!!! !
!
!!!
!!!! !!!!
!!!!
!! !!!
! !!!
!!!! !
!!!!
!
!
!
!!
!!
! !!!!!!!!!!!!
!!
!!
!!!
!!
!!
!!
!!!!
!!
!!!
!!
!
!!!
!!
!
!! !!!
!!!
!!!!!!! !
!
!!!!
!! !! !!
!!!!! !!
!!
!!! !!
!
! !!!!
!
!!!
!!!!
!
! !!
!
!!!!
!
!
!!!!
!! !!!!! !! !
!!
! !!!!
!!
!!
!
! !!
!
!
!!!!!!!
!!
! !!
!!
!!
!
!! !!
!
! !! !
!!!
! !! !!
!
!! !!
!! !! !
!
!!
!!!!!
!! !!!!! !!!
!!!!
!
!
!! !!!!!!
!!! !!! !!! !
!! !!!
!!!
!
!!!!
!!! !
!!!!!!
!!!!!!!
!!!
!!!
!!!!!
! !!
!!!!!
! !!
!!
!!
!!!!
!!!!!!!
!!
! !
!
!!!!!
!!!
!!!!!!
!
!!!!
!!
!!! ! !! !!!
!!
!
!
!
!!!
!
!! !
!
!! !
!
! !!
!!
!!
!
! !!
! !!!
!
!
!!!!!! !!!!! !!!!
!!!!!
!! !!!!!!!!!!!!!!!!
!! !! !!!! !!!
!!!!
!!!!
!!! !!!
!!
!!!!
!
!!!!
!!
!
!
!
!!
!
! !!! !!
!
! !! !!
!!!!
!!!
!!!!
!! !!!!!!!!
!!!
!!
!!!!
!!!!!!
!!
!!
!
!
!!
!
!!!
!
!
!
!
!!!
!
!!
!!
!
!!!
!!!
!
!!!!!!
!! !!!
!! !
!
!! !
!!! ! !!
!
! ! !!!! !!
!
! !!!! ! !! !
!!!
!!! !!!! !! !!
!
!! !! !!!
!!!! !
!
!
!
!!
!
!!
!
!!!!! !
!
!
!
!! !!!! !
!!
!
!!
!!!
!
!
!
! ! !!
!!!!! !!!
!!!! !
!
!!!
!!
!!
!!!!
!!! !
!
!
!
!
! !!
!!!!! !!!
! !
!
!
!!
!
!!
!!
!!
!!!
!
!
!!!
!!!
! !!! ! !
!!!
! !!
!
!!
!!
!!!
!! !
!!
!!! !!! !!
!!!
!
!
!
!
!!!
!!!
!
!!
!
!
! !!
!
!!!! !
!!
!!
! !!
!
!
! ! !
!!
!
!!
!!!
!!
!!!
!!!
!!
!!
!!
!!!!
!
!
!!!
! !!!!
!
!!!!
!
!
!!
!! !
!
!
!
!!
!! !
!
!!!!
!!
!!
!!! !!!!!! ! !! !!
!!
!!
!!
!! !!!
!!!!! !
!! !
!
!!
!!
!!
!
! !! !!!
!!!
!
!!! !!! !!! !
!
!!!
!!!!
!!
!!!
!!
!
!
!!
!
!!!
!
!
!
!!!!
!
!!
!
! !
!!!
!!
!
!
!!!!
!!
!
!
!!!!! !!
!!
!
!!
!
!
!
!
! !!!!
!
!!
!
!!
!
!
!
!
!!
!
!
!
!
!!
! !!!!
!!
!! !
!
!!
!! !
!! !!!
!
!
!
!!!
!!!
!!
!
!
!!
!
!!
! !
! !
!!
!
!!
!!!!
!!
!
!
!!
!
!! !!
!
!!!!
!!
!!!
!!!
!!! !!!
!
!
!!
!! ! !
!!
!
!
!
!! !
!! !
!
!! !
!
!!
!!
!!! !
!!
!!
!!!!
!
!!
!
!
!!
!
!
!!
!!
!
!
!! !!!
!!!
!
!
!
!
!!
!
!
!
!
!
! !
!
! !!
!
!
!!
!
!!
!! !
!
!!
!
!
!
!
!!
!!
!!!
!!!
!!
!!
!!
!
!
!!
!!
!
!
!!
!
!
!
! !!!!!
!!!!
!!
!
!!!! ! !! !!
!!
!!!
! !
!
!!
!!
!
!
!!
! !!!!
!!
!!!
!! !!!!
!!!
!
!!!!
!!
!!
!
!!
!
! !!!
!
!!!
!!
!!!
!
!!
!
!
!
!! !!
!!!
!!
!
!!
!! ! ! !! ! !
!!! !!
! ! !!
!!
!
!!
!!!
!!!
!!
!
!
!
!! !!!
!!! !
!!!! !
!
!
!
!
!!!
!!!!!!!!
!! !!
!!! !!
! !!
!! !
! !!
!!
!!!
!!!!!
!!!
!!! !
!!!!!
!!!!
!! !!
!!!!
!!!! !!!! !!
!!
!!
!!!
!!! !!
!!!! !
!!! !
!!
!!! !
!!
!!
!
!!!!
!!!!!
!!! !!!
!
!
! !!!
!!!
!!
! !!!
!!
!!
!!
!!
!
1 2 3 4
−4−2
02
predict(rf)
resi
d(rf)
!
!
!
!
!
!!!!!
!
!!
!
!
!
!!
!
!!
!
! !
!
!!! !
!!
!
!
!
!!
!! !!
!!
!!!!
!!!
!! !
! !! !
!
!
!!!!
!!
!! !!!!!
!!
!!
!!!
!!!! !! !! !!! !!
!!!!!!!!!
!
!!
!!
!!! !! !!!
!!!!! !!
! !!!
!!
!!
!
!!!!
!!! !!!
!!!!!
!!!!!!!!
!! !!! !
!
!
!! !
!!!!
!
! !!!
!
!
!!!
!
!!
!!!!!!! !!!
!!!
!!
!!! !
!!!!
!
!
!! ! !
!
! !
!
!
!!
! !!!
!
!!
!!
!!!!
!
!!!
!!!! !!!!!
!
!
!!! !!
!
!!
!! !!!!!
!
!
!!
!! !!
!
!
!
!
!!
!!
!
!
!
!!
!
!
!!
!
!
!!
!
!
!!
!!!
!
!!
!
!!
!
!
!!!
!
!
! !!
!!
!!
!
!!
!!!!
!
!!! ! !!! !! !
!!!
!! !
!
!
!! !!!! !!
!
!!
!
!!
!
!
!!
!!
!! !!
!
!
!!
!!!!
!
!!
!!!
! !!
!!
! !
!
!!!
!
!!
!
! !! !!!
!!!
!!!!
!
! !!!
!!!!
!
!
!!!
! !!!
!
!!
!!
!
!!!
!!
!
!
!
!!!
!
!!
!
!!!
!! !
!
!
!
!
!!!
! !
!!!
!!
!! !!
!
!
!!!
!
!!!
!
!
!
!!
!!
!
!!!!
!!
!
!
!!
!!
!!
!!! !!
!!
!!!
!
!! !! ! !!!!!
!!!
!
!!!
!
!!
!
!!!
!
!
!
!
!!
!
!! !!
! !
!
!
!! !!
! !! !! !
!!!!!
!
!
! !!!!
!!! !
!
!!!
!!
!
!
!!
!!
! !!!
! !!
! !! !!!
!!
!
!!
! !! !!
!
!!!! !!
!!
! ! !!!
!
! !! !!!!
!!
!
! !!
!
! !!!
!!!!!! !! !!!!
!
!
!!!
!
!!!
!
!
! !!!
!
!!!!!
!!! !!
!
!!
!! !!
!! !!!!
!!!!
! !!
!
!!
!!!
!!
!!!
!!!!
!
!!
!! !!
!
!!!
!!!!
! !!!
!!
!!
!
!!
!
!!
!!
!!
!
!!
!!!! !
!
!!
!!
!!
!
!
!
!!!!
!
!! ! !! !!
!
! !!
!
!!
!
!!
! !!
!
!
!
!
!
!
!
!
!
!
!
!!!!
!
!
!!
!
! !
!!
!
!
!!
!
!!!
!
!
!
!
!
!!
!
!!!!
!!
!
!
!! ! !!
!
!!! !!
!
!!
!
! !!
!
!!!!
!
!!!
!!
!
!
!
!
!!
!
!!!!
!!!! !!
!!!
!
!!
!!
! !
!
!
!
!
!
!!!
!
!
!
!!! !!
!
!
!!
!!!!!
!!
!
!
!
!!!
!
! !!!
!!
!
!
! !!!!
!
!
!!
!
!
!!
!!!
!
!!
!
!
!!
!
!
!
!
!!!
!!
!
!!
!
!!
! !!
!
!!
!
!!
!!
!!
!!
!
!!!! !!
! !
!!!
!!! !
!!
!!! !!
!
!
!!!
!!
!
! ! !!
!
!
! !!!
!
! !!!! !!!
!!
!!
! !
!! !!!! !
!!
!
!!!!
!
!!
!
!!! !! !
!!!
!
! !!
!!! !
!
!
!
!!!
!! !!
!
! !!
!!! ! !!! !
!!!
!! !!! !!! !!
! !!!! !
!!!!!!! !!
!!!! !!!! !!! !! !
!!!!
!!!! !
! !!
!
!!!
!
!!! !!!
!
!! !!!
! !!!
!!!
!!
!!!!! !!!!
!!!!
!!
!!
!
!!!!
! !!!!!!
!
!! !
!
!! !!
! !!
!
!
!
!
!!!!!
!!
!!!!!!!
! !! !! !
!!!!
!!
!!!!!
! !
!
!!! !
!!!!
!
!!!!!! ! ! !!!!!
! !!!
!!!
!!!! !! !!!!!!!!!!! !!!!!!! !!
!!!!!!!
! !!!
!! !!!!!!! !
! !!
!!
!!
!
!!! !!!!!! !! !
!!! !!
!!!
!!
!
!!!!!
!
!
!!!
!!! !!!!
!!
!
!! !! !!!!
!!!!!
!!
!!
! !
!!!
!!
!!
!!!
!
!!
!
!!
!!
!!
!
!!!!
!
!!!!!
!
!!
!!! !
!
!!! !
! !!
!
!! !! !!!
!
!!
!!!!
!!
!
!!!
!!!!!
! !!!!!!!
!!!
!! !!!
!!!! !
!!!!
!
!
!
!!
!!
! !!!!!
!!!!
! !!
!!!!
!!!
!
!
!!
!!
!!!!
!!
!!!
!!
!
!!!
!!
!
!! !!!
!!!
!!!!!!
! !
!
!!!!!
! !!!!
!!!!!!!
!!!
!! !!
!
!!!!!
!
!!!
!!! !
!
! !!
!
!!!!!
!
!!!!
!! !!!! ! !! !!!
! !!!
!!
!
!!
!
! !!
!
!
! !
!!!!!
!!
! !!
!!
!!
!
!! !!
!
! !! !
! !!
! !!
!!
!
!! !!
!
! !! !!
!!! !!!
!!!!!!!! !
!!!!!! !
!
!!!!!!!!
!
!! !!!!
!! !!!
!!!!!
!
!
!!!!
!!! !
!!! !!!
!!! !!!!!!
!! !
!!! !!!
! !!
!!!!!
! !!
!!
!!
!!!!
!!!!!!!
!!
! !
!
!!!!!!
!!
!!!
! !!
!
!!!!
!!
!!! ! !! !!!
!!
!
!
!
!!!
!
!! !!
!! !
!
!!!
!!
!!
!
! !!
! !!!
!
!
!! !! !!
!!!
!! !!!!!!!
!!!
! !!!!!!!!!!
!!!
!!!
!! !!!!!! !!!!!!!
!!!!
! !!!
!!
!!
!!!
!
!
!!!
!!
!
!
!
!
!!
!
! !!! !!
!
! !! !!
!!!!!!
!!!!!
!!!!!!!!!!!
!!
!!
!!!!
!!!!!!
!!
!!
!
!
!!
!
!!!
!
!
!
!
! !!
!
!!
!!
!
!!!
!
! !
!
!!
!!!!!! !!!
!! !
!
!!!!
! ! ! !!!
!! !!!! !!
!
! !!
!! !!! !
!!!
!! !!!!!
!!!!
!
!! !! !
!!!!!
!!
!
!
!
!!
!
!!
!!!!!! !
!!
!
!! !!!
! !
!
!
!
!!
!!!
!
!
!
! ! !!
!!!!! ! !!
!!!! !
!
!!!
!!
!!
!!
!!
!!! !
!
!!
!
! !!
!
!!!! !!! ! !
!
!
!
!
!
!!
!
!!
!
!!!
!
!
!!!
!!!
! !!! ! !
!!!
! !!
!
!
!!
!!!
!
!! !
!
!!!! !!
! !!!
!!!
!
!
!
!!!
!
!!
!
!! !
!
!!
!!
!!!! !
!!
!!! !
!!
!
! !!
!
!
!
!!!
!!!!
!!!
!
!
!
!!
!! !
!!!!!
!
!
!!!
! !!!!
!
!!!!
!
!
!!
!! !
!
!
!
!!
!!!
!
!!!!
!!
!
!
!!!
!!! !!! !!! !!
!!
!!
!!
!! !!
!
!!!!! !
!! !!
!!
!!
!!
!
! !! !!!
! !!!
!!! !!! !!
! !!
!!!
!!!!
!!!!
!!!
!
!
!!
!
!!!
!
!
!
!!!
!
!
!
!!
! !
!!!
!!
!
!
!!!!
!!!
!
!!!!! !!
!!!
! ! !
!
!!
! !! !!
!
!!
!
!!
!
!
!
!
!!
!
!
!
!
! !! !
!!!
!!
!!
!
!
! !
!! !!! !!
!
!
!
!
!!!
!!!!!
!
!
!!
!
!!
! !
!!
!! !
!
!
!
!!!
!!!
!
!!
!
!! !
!!
! !!
!!!
!!!!!
!
!!!
!!!
!
!
!!
!! !!
!!
!
!
!
!! !!! !
!
!!!
!
!!
!!
!!!
!
!!
!!
!!! !
!
!!
!!
!!
!
!
!!
!!
!
!
!! ! !!!!
!!
!
!
!
!!
!
!
!
!
!
! !
!
! !!
!
!
!!
!
!!
!
!!
!
!!
!
!
!
!
!! !!
!!
!
!! !!
!!!
!!
!
!
!!
!!
!
!!!
!
!
!
! !!!!
!!!!!
!!
!!!!!
! !!!
!!!
!!!
! !
!
!!
!!
!
!
!
! !!
!!!
!!
! !!!! !!!
!
!!!
!
!
!!!
!!
!!
!
!!
!
! !!!
!
!!!
!!
!!
!!
!!
!!
!
!! !!
!!!
!!
!!
!!
! ! ! !! ! !!
!! !!
! ! !!!
!!
!!
!!!
!
!!!
!!
!
!
!! !
!!
!!! !
!!! ! !
!
!
!
!
!!!
!!!!
!!!! !! !!
!!! !!!
!!
!! !
! !!!!
!!!
!!!!!
!!!
!!! !
!!
!!!!!
!!!! !!
! !!!!!!!!!!! !!
!!
!
!!
!!!!! !!
!! !! !
!!! !
!!
!!! !
!!
!!
!
!!!!
!!!!!
!!! !!!
!
!
! !!!
!!!
!!
! !!!
!
!!!
!!
!!
!
1 2 3 4
−4−2
02
predict(rf)
resi
d(rf)
!
!
!
!
!
!!!!!
!
!!
!
!
!
!!
!
!!
!
!!
!
!!!!
! !!
!
!
!!
!!
!
!
!!
!!!
!!
! !
!!
!
! !!!
!!
!!!!
!!!! !!!
!!!
!!!
!!!!!! ! !
!!! ! !!! !!!
!!!
!!!
!
!
! !!
!!!! !
! !!!
!!!!!
!
!!
!!
!
!! !!
!
!!!!
!!! !! !
!!
!!! !!!!!!!!!!
!!!
!!
!
!! !
!
!!!
!
! !!!
!
!
!
!!
!
! !!!!
!!!!!!!
!! !
!!
!!! !
!
!!!
!
!
!! ! !!
! !!
!
!!
!!!!
!
!
!!!
!! !
!!
!!
!!!!! !
!!!!
!
!!!!!!
!
!!
!! !! !!! !! !!!
!!!
!
!
!
!
!!
!
!
!
!
!
!!
!
!
!!
!
!
!
!
!
!
!! !!
!
!!
!
!
!!
!
!
!!!
!
!
!!!!
!!
!
!
!
!
!!!
!!
!!!
! !
!
! !! !
!!
!!
! !
!
!
!! !!! !!!
!
!!
!
!!!
!
!!
!!
!! !!
!
!
!!
!!!!
!
!!!!
!!
!!!
!! !
!
!!!
!
!!
!
! !! !!
!!!
!!!
!!
!
! !!!
!! !!
!
!
!!!
!!!!
!
!!
!
!
!
!!!
!!
!
!
!
!!!
!
!!
!!!
!!
! !
!
!
!
!!!!
!!
!!!
! !
!!
!!
!
!!!!
!
!!!
!
!
!
!
!
!!
!
!!! !
!!
!
!
!!
!!!!!!
! !!!
!!!!
!
!! !! ! !! !
!!
!!
!
!
! !!
!
!!
!
!!!
!
!!
!
!!
!
!! !
!
! !
!
!
!! !!! !! !! !
!!!!!!
!
! !! ! !!!! !
!
!!!
! !!
!
!!
! !
! !!!!!
!! !! !
!!
!
!!
!!!!! !!
!
!!!! !!!!
! ! !
!!
!
!!
! !!!!
!!
!! !
!
!
! !!
!!
!!!
!!!! !!!!
!
!
!!!
!
!!
!!
!! !!!
!
!!!!!
!!!!!!
!!
!! !!
!!!!!!
!!!!
! !!
!
!!
!
!!!
!
!!!
!!!!
!
!!
!! !!
!
!!!
!!! !! !!!
!!
!!
!
!!
!
!!
!!
!!
!
!!
!!!! !
!
!!
!!
!
!!!
!
!!!!
!
!!! !! !!
!
!!!
!
!!
!
!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!!!
!
!
!!
!
! !
!!
!
!
!!
!
!!!
!
!
!
!
!
!!
!
!!!!
!!
!
!
!! !!!!
!!! !!
!
!!
!
! !!
!
!!!!
!
!!!
!!
!
!
!
!
!!
!
! !!!
!!!! !
!!!!
!
!! !
!
! !
!
!
!
!
!!!!
!
!
!
!!! !!!
!!
!
!!!!!!!
!
!
!
!!!
!
! !!
! !!
!
!
! ! !!!
!
!
!!
!
!
!!
!!!
!
!
!
!
!!!
!
!
!
!
!
!!
!!
!
!!
!
! !
! !
!
!
!!
!
!!!
!
!!!!!
!!!!!!
! !
!!!
!!!
!!
!!!
!!!
!
!!!
!
!!
!
! ! !!
!
!
! !!
!
!
! !!!! !!!
!
!!
!! !
!! !!!! !
!!
!
!!!!
!
! !
!
!!! !! !
! !!!
! !!
!!! !
!
!
!
!!!!! !!
!
! !!!
!! !!!
!!!
!!
!! !!! !!! !!
! !!!! !
!!!!!!! !!
!! !! !!!! !!! !!
!!
!!!
!
!!!
!!!!
!
!!!!
!!
! !!!!
!! !!!
!!!!
!!!
!!
!!!!!!!!
!!!
!!! !!
!
!
!!! !
! ! !!!
!!
!
!! !
!
!! !!! !!
!
!
!
!
!!!!!
!!
!!!!!!!
! !! !! !
!!!
!!
!!!!!!!!!
!!! !!
!!!
!
!!!!!!!! !!!!!
!!!!
!!!!!!!!! !! !! !!!
!!!! !!!!!!!
!!
!!!!!!!! !!
! !!!!!!!!! !
! !!!
! !!
!
!!!!
!!!!! !! !
!!! !!
!!!
!!
!
!!!!!
!
!
!!
!! !! !!!!
!!
!
!! !! !!!!
!!
!!!!!!!
!!
!!!
!!
!!
!!!
!!
!
!
!!
!!!!
!
!!!!
!
!!!!!
!
!
!!!! !
!
!!! !
! !!
!
!! !! !!!
!
!!
!!!!! !
!
!!!
!!!! !! !!
!!!!!
! !!!! !!
!
!!
!! !
!!!!
!
!
!
!!
!!
!!!!!!!
!!!!!!
!!!!
!!!
!!
!!
!!
!!!!
!!
!!!
!!
!!!
!!!
!
!!!!!
!!!
!!!!!!
! !
!
!!!!!
! !! !!
!!!!!!!
!
!!!!!!
!
!!!!!
!
!!!
!!! !
!! !
!!
!!!!!
!
!!!!
!! !!!!! !! !!
!
! !!!!
!!
!!
!
! !!!!
!!!! !!!
!!
! !!
!!
!!
!
!! !!
!
! !! !
! !!
! !!!!
!!! !!
!
!!! !!
!!! !
!!!
!!
!!!!! !!!!!!!!
!
!!!!!!!!
!
!!!!! !
!! !!!! !
!! !
!
!
!!!!
!!! !
!!!!!!
!!! !!!!!!!
! !!
!! !!!
! !!
!!!!!
! !!
!!
!!
!!!!
!!
!!!!
!!
!! !
!
!!!!!!
! !!!!
!!!
!
!!!!
!!
!!! ! !
! !!!!
!
!
!
!
!!!
!
!! !
!
!! !
!
! !!
!!
!!
!
! !!
! !!!
!
!
!!!!!! !!!!! !!!!!!!
!!!! !! !
!!!!
!!!
! !!!!!
!! !!!!!!!!!
!!!!
!!!!
! !!!!!
!!
!!!!
!
!!!!
!!
!
!
!
!!
!
! !!! !!!
! !!!
!!!!!!!
!!!!!!!!
!!!!!!!
!!! !
!!!!
!!
!! !!!
!!
!!
!
!
!!
!
!!
!!
!
!
!
!!!
!
!!
!!
!
!!!
!
! !
!
!!! !!!
!! !!!!!!
!
!!!
!! !! !!
!!! !!
!!!!
!
! !!
!! !!! !
!!!
!!!! !!!
!!!
!
!
!! !!
!
!!!!!
!!
!
!
!
!!
!
!!
!!!!!! !
!!
!
!!!!!
! !
!!
!
!!
!!! !
!
!
! !!
!!
!!!! !!!!
!!! !
!
!!
!
!!
!!
!!!!
!!! !
!
!!
!
! !!
!!!!! !!!
!!
!
!
!
!
!
!!
!!!
!
!!!
!
!
!!!
!!!
!!!! ! !
!!!
! !!
!
!!
!
!!!
!
!! !
!
!!!! !!
! !!!
!!!
!
!
!
!!!
!
!!
!
!!!
!
!!
!!
!!!! !
!!
!!
! !!!
!
! !!
!
!
!
!!
!!!
!!
!!!
!!
!
!!
!!
!!
!!!!
!
!
!!!
! !!!!
!
!!!!
!
!
!!
! ! !
!
!
!
!!
!! !
!
!!!!
!!
!!
!!!
!!! !!! ! !! !!
!!
!!
!!!! !!
!
!!!!!!
!! !
!
!
!!
!!!
!
! !! !!!
! !!!
!!! !!! !!! !
!
!!
!!!!!
!!!!
!!!
!
!
!!
!
!!!
!
!
!
!!!!
!
!
!!
! !
!!!
!!
!
!
!!!!
!!!
!
!!!!! !
!
!!
!
!! !
!
!
!
! !! ! !
!
!!
!
!!
!
!
!
!
!!
!
!
!
!
! !!
!!!!
!!
!!! !
!!
!!!!
!!
!!
!
!!
!!
!
!!! !
!!
!
!!
!
!!
! !
! !
!! !
!
!
!
!!!
!!
!
!
!!
!
!!!
!!
!!
!
!!!
!! !
!!!
!
!!
!!!
!
!
!!
!! ! !
!!
!
!
!
!!
!!! !
!
!!!
!
!!
!!
!!! !
!!
!!
!!!!
!
!!
!
!
!!
!
!
!!
! !!
!
!!!!!
!!!
!
!
!
!
!!
!
!
!
!
!
! !
!
! !!
!
!
!!
!
!!
!
!!
!
!!
!
!
!
!
!!
!!
!!!
!! !
!
!
!
!
!!!
!
!!
!!
!
!
!!
!
!
!
! !!!!
!!!!!
!!
!
!!!! ! !!
!!
!!
!!!
! !
!
!!
!!
!
!
!
!! !!!!
!!
! !!!! !!
!!
!!!
!
!
!!!
!!
!!
!
!!
!
! !!!
!
!!!
!!
!!
!!
!!
!!
!
!! !!
!!!
!!!
!!
!! ! ! !! ! !
!!! !!
! !!!
!! !
!!
!!!
!
!!!!
!
!
!
!! !
!!
!!
! !
!!!
! !!
!
!
!
!!!
!!!!!!
!!
!!!!
! !! !!! !
!
!! !
! !!
!!
!!!
!!!!! !
!!
!!
! !!!
!!!!!
!!!! !!
!!!! !!!!!!!
! !!!!
!!
!
!!
!!! !!
!!!! !
!!! !
!!
!!! !
!!!!
!!!!!
!!!!!
!! ! !! !!
!
! !!!
!!!
!!
! !!!
!!!!
!!
!!
!
1 2 3 4
−4−2
02
predict(rf)
resi
d(rf)
!
!
!
!
!
!!!!!
!
!!
!
!
!
!
!!
!!
!
!!
!
!!!
!
!!
!
!
!
!
!
!!!
!
!!
!!!
!
!
! !
!
! !
! !!!
!!
! !!!
!!
!! !!
!
!!!
! !!!
!!
!!!
! !! !!! !!
! !!
!
!!
!!
!!!
!
!!
!
!!!!
!! !!!
!!!!! !
!!!! !
!!
!!
!
!!!!
!!!
!! !
!!
!!!
!!
!!!!!!!
! !!
!!!
!!
!!
!
!!!
!
! !!!
!!
!!!
!! !
!!
!!!
!! !!!!
!!
!
! !!
! !!
!!!!!
!! ! !
!
! !!!
! !!!!!
!!
!!!
!!!!
!!
!!
!!!! !
!!!!
!!!!
!!!
!
!!
!! !! !!! !!
!!
!! !!
!!
!!
!!
!
! !
!
!!!
!!
!!
!
!! !
!
!
!!!!
!
! !!
!
!!!!
!!!
!
!!!!
!
! !!
!
!!
!!
!!
!
!!!
! !!! !
! !
!!
!!
!!
!
!
!! !!!
!!!
!!
!
!
!!
!
!
!
!
!
!
!!!
!
!
!
!!
!!!!
!
!
!!!
!
! !!
!!
! !
!
!!!
!
!!
!
! !!!
!!
!!
!!!
!!
!
! !
!!!
! !!
!
!
!!
!
!!! !
!
!
!
!
!
!
!
!!!
!
!
!
!
!!!!
!!
!!!
!!! !
!!
!
!!
!!
! !
!!!!
!
!! !!!
!!!!
!
!!!
!
!
!
!
!!
!
!
!!!
!
!!
!
!
!!
! ! !!!!!
!!!
!! !
!
!
!! !!
! !! !
!
!
!!
!
!!!!
!!!
!
!! !
!
!!
!!!
!!
!!!
!!
!
!
!! !! ! ! ! !! !!!
!!!
!
!
!!! !
!! !! !
!
!!!
!!
!
!
!
!
! !
! !
!!! !
!
!!! !
!!!
!
!
!!! !! !!
!
!! !!
!!
!!
! ! !
!
!
!
!!
! !!!
!
!!
!! !
!
!
! !!!
!!!!
!!
!! !!!!
!
!
!! !!! !
!!
!! !!!!
!!!!!
!!! !!
!!!!! !
!! !!
!!!!! !!! !!
!!
!
!!! !
!! !!!!!
!
!
!!!
! !!!
!!!
!!! !! !!!
!!!!
!!
! !!
!!!!!
!
!! !!!! !
!!
!!
!!
!!!
!!!
!!
!!!! !!
!!
!
! !!
!!
!!
!!
! !!
!
!!
!
!!!
!
!!
!
!!!!
!
!
!!
!
!!
! !!
!
! !
!
! !!!
!
!
!
!!!!!!!! !
! !!
! ! !!!!
!!!!!!!!
!! ! !
!!!!!!
! !!!!!
!!
!
!!
!!!!!!!
!!!
!
!!
!
!
!!
!
!
! !
!
!
!
!
!
!
!!
!
!
!
!
!! ! !
!
!
!!
! !!
!!!
!
!
!
!
!!
!
!
! !!
!!
!!
!
!! !!!
!
!
!!
!
!
!! !!!
!
! !
!
!!!!
!!
!
!!!
!!!
!!
!
! !
! !!
!
!!
!
!!!
!
!!
!!
!
!!!!!!!!
!!!
!!! !
!! !!!!! !
!!!!
!!!
! ! !!
!!
! !!!
!! !!!! !!!
!! !
!! !
!! !!!! !
!!
!
!!!!!! !
!
! !!!! !! !!!! !!
! !! !!
!!
!!!!! !!!
! !!!
!! !!!
!!!!!!! !!! !!!!!! !
!!
! !!! !!!!!!!
!! !! !!!!!!!!! !
!!!!!
!!! !!!!
!!!
!!
!!! !
!!
!
!! !
!
!! !!!
!
!!
!!
!!!!! !!!
!
!! !!
!!
!!
!
!!! !
! !!!
!!!
!
!! !
!
!! !!
!!!
!
!
!
!
!!!!!
!!
!!!!!!!
! !! !
! !! !!!!
!!!!!!!!!
!!! !! !!!
!!!!
!!!!!!!!
!!! !
!!
!!!!!!!!! !!
!!!!!
!!!! !!!!!!!!!
!
!!!!!!
!!
!
!!! !!!!!!! !! !
!!
! !!
!
!!! !!!!!
! !! !! !
! !!!
!!
!!!
!!!
!!!
!
!!
!! !! !! !!
!!
!
!! !! !!!
!
!
!!
!!
!!
!
!! !
!!!!!
! !!!!
!
!!
!!!!!!
!!!!!!!!!!!!!!!!!! !!!!!!! !!
!!!!! !!!!!!!!!!! !!!!!
!!!!
!!!!
!!!!
!
! !!
!!
!!!
!!
!! !
!!!!
!
!
!
!!
!!
! !!!!!
!!!!!!!
!!
!!
!!!!
!
!!
!!
!!!!
!!
!!!!!!
!!! !!
!
!! !! !!!!!!!!!!! !
!!
!!!!
! !! !! !!!!! !!
!!!
!!!!
!! !
!!!
!
!
!!
!
!!!
!
! !
!!
!
!!!
!
!
!!!
!
!! !!!! ! !! !!
!
! !!!!
!!
!!
!
! !! !!
!!!!!!!
! !
! !!!!
!!!!! ! !!! !! !
!!!!!!!!!
!!!!!
! !! !!!!!!!!
!!
!!!!!! !
!!!
! !!!
!
!! !!!!!!
!!!!!! !!! !
!! !!! !!!
!
!!!!!!! !
!!!!!!
!!!!!!!
!!
!
!!! !!!!
!! ! !
!!!!!
! !!
!!
!!
!!!!
!!!!!!!
!!
! !
!
!!!!!
!!!
!!!!!!
!
!!!!
!!
!!! ! !
! !!!
!!
!
!
!
!!
!!
!! !
!
!! !
!
! !!
!!
!!
!
! !!! !! !
!
!
!!!!!!
!!!
!! !!!
!!!!!!
!! !!!!!!!!!
!!!!!!
!!! !! !!!! !!!
!!!!
!!!!
! !!!
!!
!!
!!!
!
!
!!!!
!!
!
!
!!!
!
! !!!
!!!
!!! !
!
!!!!
!!!
!!!!
!! !!!
!!!!!!
!! !
!!!!
!!!!!!!
!!
!
!!
!
!!
!
!!!
!!
!
!
!!!
!
!
!!
!
!
!!!!
!!
!
!!!!!!
!! !!!
!! !
!!
! !!! ! !!!
!
! ! !!!! !!
!
! !!!!! !! !!
!!
!!!!!!! !! !! !!
!!! !
!!!!!
!!
!
!
!!
!
!
!!
!
!
!!
!
! !
!
!
!
!!!
! !
!!
!
!
!
!!
!!!
!!
!
! !!!
!!!!! !!!
!!!! !
!!!
!!!
!!
!!!!
!!! !
!!
!
!! !!
!!!!! !!!
! !!
!!
!!
!!
!!!!!!!
!
!!!!
!!!
!!
!! !
!!!!
! !!
!
!!!
!!!
!!
! !!!!!! !!
! !!
!
!!
!
!
!
!
!!!
!!!
!
!!
!
!
!!!
!
!!!! !
!
!
!!! !
!
!!
! !!!
!
!!!!
! ! !!
!!!
!
!!
! !
!
!
!!
!!!!
!
!
!!!
! !!!!
!
!!!!
!!
!!
!!
!
! !
!! !!
! !
!!!
!!!
!!!!!!!!! !!! ! !! !!
!!
!!
!
!!! !!
!
!!!!! !!! !
!!
!!
!!!
!
! !! !!! !!!
!
!!! !!! !!! !!
!!
!!!!!!!
!!!
!!
!
!
!!
!
!!!
!
!
!
!!!!
!
!
!
!
! !
!!!
!!
!
!
!
!
!!
!!
!
!
!
!!!! !
!
!!!!!!
!!
!
! !!!!!!!
!
! !!!!!!!
!
!!!!!!!
!!!
! !
!! !
!
!!!
! !!!
!!!!
!!
!!!!!!!!!
!!
!
!!
!
! !!
!!!
!
!!!
!!!
!!!!
!!!!! !!!
!!!!!!!!! !!!
!!! !!!!!
!!!! !!!!!
! !!
! !!! !!!
!!!!!
!! !! ! !!
! !!!!! ! !
!
!!
!!
!
!
!!
!
!!!
!!!! !!
!!!
!!
!
!
!!
!
!
!!
!
! !
!! !
!
!
!
!!
!
!
!
!! !
!
!!
!
!!
!
!! !!
!
!
!
!!!
!
!!!
!!
!!
!!!
!
!
!
!
!
!
!
!
! !!!!
!
!!!
!!
!
!
!!!! ! !! !!
!! !
!!
! !
!
!!!
!
!
!
!
!! !
!!!
!!
!!!
!!!!!!
!!!
!
!! !!
!
!
!!
!
!!
!
!!!!
!
!!!
!
!!!
!!
!!!!
!!! !!
!!!
!!
!
!!
!! ! ! !! ! !
!!! !!
!! !
!!
! !
!!
!!!
!
!!!
!!
!
!!! !
!!
!!! !
!!!! !
!!
!!
!! !!!!!
!!!!
!! !!
!!! !!
!!
!
!
! !! !
!!!
!!!
!!!!
!
!!
!
!!
!!
!!!!!
!!
! !!! !!
!!!!
!!!
! !!!! !!!!!!!!!
!!! !!
!!!! !!!! !
!!
!
!! ! !!!!
!
!!!
!!
!!!!
!
! ! !!!!
!
! !!!
!!!
!
!
! !!!
!!
!!
!
!!!
!
0.6 0.7 0.8 0.9 1.0−0
.20.
20.
6
invlogit(predict(rf))
resi
d(rf,
inv
= TR
UE)
!
!
!
!
!
!
!!!!
!
!
!
!
!
!
!!
!
!!
!
!!
!
!!! !
!!
!
!
!
!
!
!! !
!
!!
!!!!
!
! !
!
! !
! !!
!
!!
! !!!
!!
!! !!!
!!!
!!!
!
!!
! !!! !! !!
! !! !!
!!
!!!!
!!!
!
!!
!
!!!!
!! !!!
!!!!
! !!
! !! !
!!
!!
!
! ! !!
!!! !!
!
!!
!!!
!!!!!!!!
!! !!! !
!
!!
! !
!!!!
!
! !!!
!!
!!!
!!!
!!!!!!! !!!
!!!
!
! !!! !
!!!!
!!
!! ! !
!
!!!!
! !! !!!
!
!!
!! !!!!!
!!!
!!!! !!!!!
!!
!!! !!
!
!!
!! !
! !!! !
!
!!
!! !!
!!
!
!!!
!
!!
!
!!
!
!!
!!
!
!
! !
!
!
!!!!
! !
!!
!
!!!
!
!!!
!
!
! !!
!! !
!
!
!!!
!!!
!
!!! ! !!! !
! !
!!!
!
!!
!
!
!! !!!! !!
!!
!
!
!!
!
!
!!
!
!
!! !
!
!
!
!!
!!!!
!!!
!!
!!
!
!
!!
! !
!
!!!
!
!!
!
! !! !!
!!!
!
!!
!!
!
! !
!!!
!!!
!
!
!!
!
! !!!
!
!
!
!
!
!
!
!!!
!
!
!
!
!!!
!!!
!!
!!
!!!
!
!
!
!!!!
! !
!!!
!
!
!! !!!
!
!!!
!
!!!
!
!
!
!!
!!
!
!!
!!!!
!
!
!!
!!
!!
!!!
!!!
!!!
!
!
!!!!! !
!!
!!
!!
!
!! !
!!
!!
!
!! !
!
!
!
!!!
!!! !!
!!
!
!
!! !!
! ! ! !! !!!!!!
!
!
! !!!!
! !! !
!
!!!
!
!!
!
!
!
! !
!!
!!
! !
!
! !! !!
!!
!
!
!!
! !! !!
!
!!!
!!
!!
!
! ! !
!!
!
! !
! !!!
!
!!
!! !
!
!
! !
!!
!! !!!
!!
! !!!!!
!
!! !!
! !!!
!! !!!
!!!!!!
!!! !!
!!!
!! !!
!! !!!!!!!!
! !!
!!
!
!!! !
!! !!!
!!!
!
!!!
! !!!
!!!
!!!!! !!!
! !!!
!
!! !
!!!!!!
!
!!
!!!! !
!!
!!
!!
! !!
!!!
!!
!
!! ! !!!
!!
! !!
!
!!
!
!!
! !!
!
!!
!
!!
!
!
!!
!
!!!!
!
!!!
!
!!
!!
!
!
!!
!
!!! !
!
!
!
!!!!!!!! !
! !!
! !!!!!!
!!!!!!!
!! ! !!
!!!!!! !!!!
!!
!
!
!!
!!!!
!!!!! !
!
!!!
!
!
!!
!
! !
!
!
!
!
!
!
!!
!
!
!
!
!! !
!
!
!
!
!
!!
!!
!!
!
!
!
!
!!
!
!
! !!
!!
!!
!
! !!!!
!
!
!
!!
!
!!!
!!
!
! !
!
!!!
!
!!
!
!!!
!!!
!!
!
! !
! !!
!
!!
!
! !!
!
!!
!!
!
!!!!!!
! !!
!!!!! !
!! !!! !!!
!!!!
!!!
! ! !!
!!
! !!!!
! !!!! !
!!!
!! !
! !!! !!!! !
!!
!
!!!!!! !!
! !!!! !! !!!! !!
! !! !!!
!!!!!
! !!!
!!!!
!! !!!!!!!!!! !!! !!! !!
! !!!! !!! !!!!! !!!!!! !!!!!!!!! !
!!!!!
!!! !
! !!!
!!!
!
!!! !!!
!
!! !!
!!
!!!
!
!!
!!
!!!!!!!!
!!
! !!
!!
! !
!
!!! !
!!!!!
!!
!
!
! !!
!! !!
!!!
!
!!
!
!!!
!!!
!
!!!!!!!
! !!
!! !
!!!!! !!!!!!!!!!!!!! !!
!!
!!!!!!!!!!!!!
! !!!
!!!
!!!! !! !!!!!!!
!!!! !!!!!!! !!
!
!!!!!!
!!!
!!! !!!!!
!!!
! !!
!! !!
!
!!! !!!!!! !! !
!!! !!!
!!
!!!
!!!!!
!
!
!!
!!!!!!!
!
!!
!
!! !! !!!!
!
!!
!!
!!
!
!! !
!!!!!
! !!!!
!
!!
!! !!!!
!!!!!!!!!!!!!!!!!! !!!!!!! !!
!!!!! !!!!!!!!
!!!!!!!!!!!
!!
! !!!
!!!
!!!
!
!! !!!
!!
!! !
!!!!
!!
!
!
!
!!
! !!!!!
!!!!
! !!
!!!!
!!!!
!
!
!!!
!!!!
!!
!!!!! !
!!! !!
!
!!!! !!!!!!!!!!
! !!
!!!!!
! !!!! !!!!!!!
!!!!! !!
!!!!!!
!
!
!!
!
!! !
!
! !
!!
!
!!
!!
!
!!!
!
!! !!!! ! !! !!
!
! !!!
!!
!
!!
!
! !! !
!
! !
!!!!!! !
! !!!!
!!!!! ! !!! !!!
! !!! !!!!!
!!!!!
!!! !!!!!!!!
!!
!!!!!! !
! !!!!
!!
!
!!!!!!!!
!
!! !!! !
!! !!! !!!!!
!!
!!!!
!!!!!
!! !!!
!!!
!!!!!
! !! !
!!! !!!
! !!
!!!!!
! !!
!!
!!
!!!!
!!!!!!!
!!! !
!
!!!!!!
! !!!!
! !!
!
!!!!
!!
!!! ! !! !
!!!
!
!
!
!
!!
!!
!! !
!
!! !
!
! !
!
!!
!!
!
! !!! !!
!
!
!
!! !! !!
!!!
!! !!!
!!!!
!!!
! !!!!!!!!!!
!!!!!!!! !
!!!!! !!!!!!!
!!!!
! !!!
!!
!!
!!!
!
!
!!!!
!!
!
!
!
!!
!
! !!! ! !
!
!!! !
!
!!!!!!
!!!!
!!!!
!!!!!! !
!!
! !!
!!!!
!!!!!!
!!!
!
!
!
!!
!
!!
!
!!
!
!
!!!
!
!
!!
!
!
!!!!
! !
!
!!
!!!!!! !!!
!! ! !
!!!
!! ! !!!
!
!! !!! ! !!
!
! !!!!
!!! !!!
!!! !!!!
! !!!! !!! !! !
!!!! !
!!
!
!
!!
!
!!
!
!
!
!!!
! !
!
!
!
!!
!
!!
! !
!
!
!
!!
!!
!!
!
!
! ! !!
!
!!!! !!!
!
!!! !!!!
!!!!
!!!!
!
!!! !
!!!
!!!!
!
!!!! !!!! !
!!
!!
!!
!!
!!
!
!!!
!
!!!!
!!!
!!
!! !
!!!!
! !!!
!!!
!!!
!!
! !!!!!! !!
!!!
!
!!
!
!
!
!
!!!
!!!
!
!! !
!
!!!
!
!!!! !
!
!
!!! !
!!
!
! !!!
!
!!!!
!! !!
!!!
!
!!
!!
!
!!
!!!!!
!
!
!!
!! !
!!!
!
!!!!
!!
!!
!!
!!!
!! !!
!!
!!!
!!!
!!
!!!
!!!! !!! !!! !!
!!
!!
!
!!! !!
!
!!!!! !!! !
!!
!!
!!!
!
! !! !!! !!!
!
!!! !!! !!
! !!!!!
!!!!!!!!!
!!
!
!
!!
!
!!!
!
!
!
!!
!
!
!
!
!!
! !
!!!
!
!
!
!
!
!
!!
!!
!
!
!
!!!! !!
!!!!!!!!
!! !!!!!!!
!
! !!!!!!!
!
!!!! !!!
!!!
! !
!! !
!
!!
!! !!! !!
!
!
!!
!!!
!!! !!!
!!
!
!!
!
! !!
!!! !
!
!
!
!!!
!!!!
!!!!! !!!
!!!!!!
!!! !!!
!!! !!!!!
!!!! !!! !!
! !!
! !!! !!!
!!!!
!!
! !!! !
!!
!!!
!! !!
!
!!
!!
!
!
!!
!!
!!!
!!! !!!!
!!
!
!
!
!!
!
!
!!
!
! !
!! !
!
!
!
!!
!
!
!
!
!!
!
!!
!
!
!
!
!! !!
!!
!
!! !!
!!!
!!!!
!!
!
!
!
!
!
!
!
!
!
! !!!!
!!!!!
!
!
!
!!!!
! !! !!!! !
!!
! !
!
!!
!!
!
!
!
!!
!!!!
!!
! !!
!!!!
!!
!!!
!
!!!!
!
!
!!
!
!!
!
! !!!
!
!!!
!
!!!
!!
!!
!!
!!!!!
!!!
!!!
!!
!! ! ! !! ! !
!!! !!
!! !
!!! !!
!!
!!
!
!!
!
!!
!
!!
!!
!!
!!! !
!
!! ! !!
!!
!
!! !!!!!
!!
!!
!! !!
! !! !!
!!
!
!
! !! !!!!
!!!
!!!
!!
!!
!
!
!
!!
!!
!!!
!!
!!!! !
!
! !!!!!!!!!!! !!!
!!
!!!!
!!! !!
!! !! !!!! !
!!
!
!! ! !!!!
!
!!!!
!!!!!
!! ! !! !
!
!
!!!!
!!
!
!
!
! !!
!
!!
!!
!
!!!
!
0.6 0.7 0.8 0.9 1.0
−0.2
0.2
0.6
invlogit(predict(rf))
resi
d(rf,
inv
= TR
UE)
!
!
!
!
!
!
!!!!
!
!!
!
!
!
!
!!
! !
!
!!
!
!!!
!
! !
!
!
!
!
!
!!
!
!
! !
!!!
!
!
! !
!
! !
! !!
!!!
!!!
!
!!
!! !!!
!!!
!!!
!
!!
!!!! !
! !!
! !!! !!!
!!
!!
!!!
!
!!
!
!!!!
!! !!!
!!!!! !
!!
!! !
!!
!!
!
!!!!
!!! !!!
!!
!!!
!!!!!!!!!! !!!
!!
!!
! !
!
!!!
!
! !!!
!
!
!!!
!!!
!!!!!
!!!!!!
! !!
! !!! !
!
!!!
!!!! ! !!
! !!!!
!!!!!
!
!!!!
!! !!
!
!!
!!!!! !
!!!!
!!!!
!!!
!
!!
!! !! !!! !
! !!!
!!!
!!
!!
!!
!
! !!
!!!
!
!
!!
!
!!!
!
!
!! !!
!
! ! !
!
!!!!
!!!
!
!!!!!
! !!
!
!!
!!
!!
!
!!
!! !
!
! !! !
!!
!!
! !
!
!
!! !!! !!!
!!
!
!
!!
!
!
!
!
!
!
!!!
!
!
!
!!
!!!!
!
!
!!!
!!
!!
!!
! !!
!!!
!
!!
!
! !! !!
!!!
!
!!
!!
!
! !
!!!
!!!
!
!
!!
!
!!!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!! !
!!
!!!
!!
! !
!
!
!
!!!!
!!
!!!
!!
!!
!!!
!!!
!
!
!!!
!
!
!
!
!
!!
!
!!! !
!!
!
!
!!
!!!
!!!
!!!
!!
!!
!
!
!!!!! !! !
!
!
!!
!
!! !
!!
!!
!
!! !
!
! !
!!!
!!
!!!
!!
!
!
!! !!! ! ! !! !!!!!
!!
!
!!!!
!! !! !
!
!!!
!!
!
!
!
!
! !
!!
!!!!
!
!!! !!!
!
!
!
!!! !! !!
!
!!!
!!
!
!!
! ! !
!
!
!
!!
!!!
!
!
!!
!! !
!
!
! !
!!
!!
!!
!!!! !!!!!
!
!! !!
! !!
!
!! !!!!
!!!!!
!!!!!!!!
!! !!
!!!!!!!! !!
! !!
!!
!
!!! !
!! !!!!
!!
!
!!!
! !!!
!!!
!!!!! !!!
! !!!
!!
! !!
!!!!!
!
!!
!!!! !
!!
!!
!!
!!!
!!!
!!
!
!!! !!
!!
!
!!!
!
!!
!
!!
! !!
!
!!
!
!!
!
!
!!
!
!!!
!
!
!
!!
!
!!
!!
!
!
!!
!
!!!!
!
!
!
!!!!!!!! !! !
!! ! !!! !!
!!!!!!!
!! ! !!
!!!!!! !!!!
!!!
!
!!
!!!!
!!!!! !
!
!!
!
!
!
!!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!! !
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!!
!
!
! !!
!!
!!
!
!! !!!
!
!
!!
!
!
!!
!!!
!
! !
!
!!!
!
!
!
!
!!!
!!!
!!
!
! !
! !!
!
!!
!
!!!
!
!!!!!
!!!!!!
! !!
!!!!! !!
! !!!!! !
!!!
!!!
!! ! !!
!!
! !!!
!! !!!! !
!!!
! !!
! !!! !!!! !
!!
!
!!!!!! !
!
!!!!! !! !!!! !!! !! !
!!
!!!!!
! !!!
! !!!
!! !!!!!
!! !
!! !!! !!! !!! !
!!! !!! !!!!! !!
!! !! !!!! !!! !!!!
!!!!
!!! !!!!
!!!
!!
!!
! !!!
!
!! !!
!!
!!!
!
!!
!!
!!!!!!!!
!
!! !!
!!
! !
!
!!! !
!! !!!
!!
!
!! !
!
!! !!
! !!
!
!
!
!
!!!!!
!!
!!!!!!!
! !! !!
!!!!!! !!!!!!!!!!!! !! !!
!!
!!!!!!!! !!!!!
!!!!!!!!!!!!! !!
!! !!!
!!!! !!!!!!!
!!
!
!!!!!!! !!
!!! !!!!!!! !
! !!!
! !!
!
!!!!! !!!! !! !
!!! !!!
!!
!!!!!
!!!!
!
!
!
!! !! !!!!
!!
!
!! !! !!!!
!
!!!!!!
!
!! !
!!!
!!! !!!!
!!
!!
!!!!!!
!!!!!!!!!!!!!!!!! !!!!!!! !!
!!!!!!!!!!!!!!!! !!!!!
!!!!
!! !!
!!!!!
! !
!
!! !!
!
!!
!! !
!!!!
!!
!
!
!
!!
! !!!!!!
!!!! !!
!!!!
!!!!
!
!!
!!
!!!!
!!
!!!!! !!!! !!
!
!!!! !!!!!!!!
!!! !
!!
!!!!
! !! !!
!!!!! !!!!!
!!!!
!
!!!!!
!
!
!!
!
!!!
!
!!
!!
!
!!
!!
!
!!!
!
!! !!!!! !! !!
!
!!!!!
!!
!!
!
! !!!!
!!!!!!!
! !
! !!!!
!! !!! ! !!
!!! !! !!!!!!!
!!!!!!
! !! !!!!!!!!
!
!
!!!!!! !
! ! !! !
!!
!
!! !!!!!
!
!
!! !!! !
!! !!!!!!
! !!
!
!!!!
!!!!!
!!!!!
!!! !!!!!!
!! !
!!! !!!
! ! !!!!!!
! !!
!!
!!
!!!!
!!!!!!!
!!
! !
!
!!!!!
!! !!!!!!!
!
!!!!
!
!
!!
! ! !! !!!
!!
!
!
!
!!
!!
!! !
!
!! !
!
! !
!
!!
!!
!
! !!
! !!!
!
!
!!!!!! !!!
!! !!!
!!
!!!!
!! !! !!!!!!!
!! !!!!!!! !!!!!! !!!
!!!!
!!!!
! !! !
!!
!!
!!!
!
!
!!!!
!!
!
!
!!!
!
! !!! !!!
!!! !
!!!!!!!
!!!!
!!! !!!!
!!!!!
!! !!
!!!!
!!! !
!!
!!
!!
!
!
!!
!
!
!!
!!
!
!
!
!!
!
!
!!
!
!
!!!
!! !
!
!!! !!!
!! !!!!!!
!!
! !!
! !! !!
!
!! !!!! !!
!
! !!!! !!! !!
!!
!!!!!!!
!! !!!!
! !!!
!!!!!
!!
!
!
!!
!
!!
!
!
!
!!!
! !
!
!
!
!! !
!!
! !
!
!
!
!!
!!! !
!
!
!! !
!
!
!!!! !!!
!
!!! !!!!
!!!
!!
!!!!
!!! !
!!!
!!!!
!!!!! !!!
!!
!!
!!
!!
!!
!!
!!!!
!
!!!!
!!!
!!!
! !!!!
!
! !!
!
!!!
!
! !!
!! !!
! !!! !!
!!!
!
!!
!
!
!
!
!!!
!!!
!
!! !
!
!!!
!
!!!! !
!
!
!!! !
!!
!
! !!!
!
!!!
!! ! !!
!!!
!
!!
!!
!
!
!!
!!!!
!
!
!!
!
! !!!!
!
!!!!
!!
!!
!!
!
! !
!! !!
! !
!!!
!!!
!!
!!!
!!!! !!! ! !! !!
!!
!!
!
!!! !!
!
!!!!!!
!! !!
!!!
!!!
!
! !! !!! !!!
!
!!! !!! !!! !!!!
!!!
!!!!!!!
!!
!
!
!!
!
!!!
!
!
!
!
!
!!
!
!
!!
! !
!!!
!
!
!
!
!
!
!!
!!
!
!
!
!!!! !
!
!!!
!!!!!
!
! !!!!!!!
!
! !!!!!!!
!
!!!
! !!!!!!
! !
!! !!
!!
!! !!!!
!!!
!!
!!
!!!! !
!!!
!!
!!
!
! !!
!!! !
!!
!
!!!
!!!
!!! !!! !!
!!!!!!!
!! !!!!
!!!!!!!!
!!!! !!! !!
!!
!! !!! !
!!!!
!!!
!! !! ! !!
! !!!!!
!!
!
!!
!!
!
!
!!
!
!!!
!!!! !!
!!!
!!
!
!
!!
!
!
!!
!! !!
! !!
!
!
!!
!
!
!
!
!!
!
!!
!
!!
!
!! !!
!
!
!
!!
!
!
!!
!
!!!!
!!
!
!
!
!
!
!
!
!
!
! !!!!
!
!!!
!!
!
!
!!!! !
!! !!
!! !
!!! !
!
!!!
!
!
!
!
!! !!!!
!!
!!!
!!!!
!!
!!!
!
!
!!!
!
!
! !
!
!!
!
!!
!!
!
!!!
!
!!!
!!
!!
!!
!!!!!
!!!
!!!
!!
!!
! ! !! ! !!!! !!
! !!
!!
! !
!!!
!!
!
!!!
!!
!
!!
! !
!!
!!
! !
!
!!! !
!!
!
!
!! !!!!!
!!
!
!
!!!!
! !! ! !
!!
!
!
! !! !
!!!
!!
!!! !
!!
!!
!
!
!
!!
!!
!!!!!
!!!! !
!
!!!! !!!!!!!
! !!!!!!
!!!
!!! !!
!! !! !!!! !
!!
!
!! ! !!!!
!
!!!
!!
!!!!
!! ! !!
!!
!
! !!!
!!!
!
!
! !!!
!!
!!
!
!
!!
!
0.6 0.7 0.8 0.9 1.0
−0.2
0.2
0.6
invlogit(predict(rf))
resi
d(rf,
inv
= TR
UE)
!
!
!
!
!
!
!
!!!
!
!
!
!
!
!
!
!!
!!
!
!!
!
!!!
!
!
!!
!
!
!
!
!!
!
!
! !
!!
!!
!
!!
!
!!
!!
!
!
!!
!!
!
!
!!
!
!!!
!
!!
!
! !!
!
!
!
!!
!
!!
!!
!
! !!
!
!
!
!
!!
!
!
!!
!
!
!
!
!
!
!!!
!!
!!!
!!
!!
! !
!!
!!
!
!!
!!
!
!
! !!
!
!!!
!!
!!
!
!
!
!
!
!!!
!!!
!!!
!
!
!
!
!
!
! !
!
!
!!
!
! !
!
!!
!
!
!!
!
!
!
!
!
!
!
!
!
!!!
!!
!
!
!
! !
!
!
!
!
!!!
!!
!! !
!
!
! !
!
!
!
!
!!
!!
!
!
!
!
!
!!
!!
!
!
!
!
!
!!! !
!
!!!
!
!!
!
!
!
!!
!
!
!
!
!
! !!!
!!
!
!
!
! !!
!
!
!
!
!!
!
!
!
!
!! !
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!!
!
!
!
!!
!
!
!!
!
!
! !
!
!
!
!
!
!
!
!
!
!
!!!
!
!
!!
!!
!!
!
!
!
!
!
!
!
! !!
!
!!!
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!!
!
!
!
!!
!
!!
!
!
!
! !
!
!
!!
!
!!
!
!!
!
!!
!
!!
!
!
!
!!
!
! !
!!
!! !
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!
!!
!
!
!
!
!
!!
!
!
!!!!
!
!!
!!
!
!!!!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!!!!
!
!
!
!
!
!!
!
!
!!!
!
!!
! !
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!! !
!
!!
!
!
!!
!!
! !
! !!
!
!!
!
!
!!
!
!
!!
!
!
!!!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!!!
!
!
!!
! !
!
!
!
!!
!
!
! !
! !
!
!
!!!
!
!
!
!
!
! ! !
!
!
!
!
!
!
!!!
!
!!
!
!!
!
!
!!
!!
!!!!
!!
!
!!
!
!!
!
!
!!
!
!
!
!
!
!
!
! !!!
!
!
!
!!
!
!
!
!
!
!
!
!!
!
!!
!
!
!!
!
!
!
!
!
!!
!!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
! !!
!
!
!
!
!
!!!
!!
!!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!!! !!
!
!
!
!!
!
!!
!
!
!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
! !
!
!
!
!
!
! !!
!
!
!
!
!
!
!
!
!
!!! !
!
!
!
!
!!!!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!!!
!
!
!!!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
! !!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!
!
!
!
!!
!!!
!!
!
! !
! !
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
! !!
!!
!
!!
!
!
!
!
!!
!
!
!
!!!
!
!
!
!
!!
!!
!
!
!
!
!
!!!
! !
!
!!
!
!
!!
!
!!
!
!
!
!
!
!!
!
! !
!
!
!!
!
!
!!
!
!
!
!!
!
!
!
!
!!
!
!
!
!!!
!
!
!
!
!
! !
!!
!!
!!!!
!
!
!
!! !!!!
!!
!
!
!
!
!
!!!
!
!
!
!!
!
!!
!
!!
!
!
!!
! !
!
!
!
!
! !
!
!
!!
!!
!
!
!
!
!
!!!!!!!
!
!
!
! !!
!
!
!!
!
!
!! !
!
!
!!
!
!
!
!
!
! !
!
!
!!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
! !
!
!!
!
!
!
!!
!
!!
!
!
!
!!! !
!
!!
!!
!!
!
!!
!!
!!!
!
!!
! !
!
!
!
!
!
!!
!!!
!
!!
!
!
!
!
!
!
!!
! !
!!!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!!
!!
!
!!
!! !
!!
!
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!!
!
!!
!
!
!
!
!
! !!
!! !
!
!!
!
!! !
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!
!
!
!!!!
!
!
!
!
!! !
!
!
!!
!
! !!
!
!
!!
!
!!
!
!
!!
!
!
!
!
!!
!
!
!!
!!!!
!
!
!!
!
!!
!
!
! !
!
!
!!
!
!
!
!
!! !
!
!
!
!
!
!
!
!
!
!
!
!!!
!
!!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!!
!
!
!
!!!
!
!
!
!!
!!
!
!
!!
!!
!
!!
!
!!
!
!
!!
!!
!
!
!!
!
!
!
!! !
!
!
!!!!!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!
!!
!
!
!!
!
!
!
!
!!
!!
!! !!!
!! !
!
!
!
!
!
!
!!
!
!
!
!!
!
!!
! !
!
!!
!!
!!!
!!
! !!
!!
!
!
!
!!
!
!!
! !
!!
!!
!
!!
!
!!
!
!!!
!
!
!
!!
!
!
!
!
!
!!
!
!
!
!
!!!
!! !
!!
!
! !
!
!
!
!
! !!!
!!
!
!
!!
!
!
!
!
!!
!
!! !!
!
!!
!
!
!
!
!
!
!
!!
!
!
!!!
!!
!!
!!!
!!
!
!
!
!!
!
!!
!!
!
!!
!
!!
!!
!
! !!
!
!
!
!
!
!
!!
!!!
!!!
!
!
!
!!
!
!
!!
!!
!!
!
!
!!
!!
!
!
!
!
!
!
!
!
!!
! !!
!
!
!!
!
!
!
!
!
!!
!
!
!
! !
!
!! !
!
! !
!
!
!
!
!
!
!!
!
!
!!
!
!
!
!
!!!!
!
!!!
!
! !
!!
!
!
!!
!!
!
! !
!
!
!!!!
!!
!
!
!!!!
!
!!!
!
!
!!!
!!!
!
!!
!
!
!
!!
!
!!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
! !!
!
!!
!
!
!!
!
!
!
!
!!
!!
!
!!
!
!
!!
!
!
!
!
!
!!!
!
!
!
!
!
!!!
!
!
!!!
!!
!!
!
!!
!
!!
!
!!!
!
!
!
!
!!!
!
!
!
!
!
!
!!
!!
!!
!
!
!
!!
!!
!!
!
!!
!
!!
!
!! !
!
! !!!!
!
!!
!!
!
!!!
!
! !
!
!
!!
!
!
!
!!
!
!!!
!
!!
!!
!
!!
!
!
!!!
!
!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!!!
! !
!!
!
!
!
!!
!
!!
!
!
!
! !!
!
!
!!!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!
!
!
!
!
!
!
!
!
!!!
!!!
!
! !
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!
!!
!
!
!!
! !
!
!
!
!
! !
!
!
!
!
!
!
!
!
!
!
!!
!
!
!!
!!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
!!
!
!!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
! !
!
!
!
!
!
!!!
!
!
!
!
!
! !
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!!!
!
!!
! !
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!!
!
!
!
!
!
!
!
!
!
!!
! !
!
!
!!
!
!
!!
!
!!
!!
!
!!
!
!
!
!
!!
!
!
!
!
!!
!
!!
!
!
!!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
! !
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!!!!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!!
!!
!
!
!
!
! !
!
! !
!
!
!!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
! !
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!
! !
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
! !
!
!!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!
!
!
!
!!
!
!
!
!
!!
!! ! !!
!!
!
!
!
!
!
! !
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!!
!!
!!
!!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!!
!
!
!
!
!
!
! !!
!!!
!
!
!
!
!!
!
!!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!! !
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!!!
!
!
!
!
!!!!
!
!!
!
!!
! !
!
!!
!
!
!
!
!!
!
!!
!!
!
!
!!
!
!
!
!
!
!
!
! !
!
!
!!!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!!
!
! !!
!
!
!
!
!
!!
!
!!
!!
!!!
! !
!
!!
!
!!
!
!
! !
!
!
!
!
!
!
!!
!
! !
!!!
!
!!
!!!
!
!
!!!
!
!!!
!
!
! !!
!
!
!
!
!
!
!!
!
!
0.6 0.7 0.8 0.9 1.0
−3−1
12
3
invlogit(predict(rf))
resi
d(rf)
/resi
d.se
(rf)
!
!
!
!
!
!
!!!!
!
!
!
!
!
!
!!
!
!!
!
!!
!
!!! !
!!!
!
!
!
!
!!!
!
! !
!!!
!
!
! !
!
! !
! !
!
!
!
!
! !!
!
!!
!! !!!
!!
!
!!
!
!
!
!
! !!! !
! !!! !
! !!
!!
!!
!!
!!!
!
!
!
!
!!
!!!!
!!!
!!
!!! !
!!
!!
!
!
!
!!
!
!! !
!!
!!!
!!
!!
!!
!
!!!!
!
!!
!
!!!
!!
!
!
!
!
!!
!!
!!
!
! !
!!
!
!
!
!!
!!
!
!
!
!
!!
!! !!!
!
!
!
!
!!
!
!!
!!!!
!!
!! !!
!
! !
!
!
!
!
! !!!
!
!
!
!
!!
!!!
!
!
!
!
!
!!! !
!
!!!
!
!
!
!
!
!!
!
!
!
!
!
!
! !!!
!
!
!
!
!
! !
!
!!
!
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!!
!!!!
!
!
!!
!
!!
!
!
!!!
!
!
!!
!!
!!
!
!
!
!
!!
!
!
!
!!! !
!!!
!
!!
!!
!
!
!!
!
!
!! !!!
! !!
!
!
!
!
!!
!
!
!
!
!
!
!! !
!
!
!
!
!
!
!!!
!
!
!
!!
!!
!
!
!!
! !
!
!
!!
!
!!
!
! !! !!
!!!
!
!!
!!
!
! !
!!
!!!!
!
!
!!
!
! !!!
!
!!
!
!
!
!!!
!
!
!
!
!
!!
!!
!!
!
!!!
!
!!
!
!
!
!!
!!
!!
!!!
!!
!!!
!
!
!
!!!
!
!!!
!
!
!
!
!!
!
!
!!
!!
!!
!
!
!!
!
!
!!
!!
!
!!
!
!
!!
!
!
!!!
!!
!!!
!
!
!
!
!
!
!!!
!
!!
!
!!
!
!
!
!
!
!!
!
!!!!
!!
!
!
!! !
!
! !! !
!!
!!
!
!
!!
!
!!!!
!! !!
!
!
!!
!
!
!
!
!
!
!! !
!
!
!!! !
!
! !!
!
!
!!
!!
!!
! !
! !!
!
!!
!!
!!
!
!
! !!
!
!
!
! !
!!!!
!
!!
!
! !
!
!
!!
!!
!!
!!!
!!
! !!!
!
!
!
!!
!
!
!
!
!
!
!
! !!!
!!
!!!
!
!
!
! !!
!
!!
!!
!
!
!! !
!!!
!!
!!
!!
!
!
!!
!
!!
!
!
!
!!
!
!
!
!
!
!!
!
! !!
!
!
!
!
!
!!!
! !
!
!
!
!
!!
!
!
!!
!
!
!!
!
!
!
!
!
!!
!! !
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!!
!
!
!
!!
!!
!
!!
!
!
!!
!!!
!
!!
!
!
!
!
!
!
!
!
!
!!!
!
!
!
!
!
!!
!!
!
!
!
!
!
!!!!
!
!
!
!
!
!
!
!
!!! !
!
!
!
!
!!!
!
!
!
!!!!
!
!!
!
!!
!
!
!
!
!!
!
!
!!
!!
!
!
!
!
!
!
!
!!!
!
!!!
!!
!
!!!
!
!
!!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!! !
!
!
!
!!
!!
!!
!
!
!
!
!
!
!!
!
!
! !!
!
!!
!
!
! !!!
!
!
!
!
!
!
!
!
!
!
!!
!
! !
!
!
!!
!
!
!
!
!
!!
!!!
!!
!
!!
! !
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!!!
!
! !
!
!!
!
!
!!
!
!
!!
!
!!
!
!
!!
!
!
!
!
!
!!
!
!
!
! !!
!
!
! !!
!! !
!
!!
!
!
!
!!
!
!!
!!!
!
!
!
!
!!
!!
!
!!
!
!
!!!
! !
!
!!
!
!!
!
!
!!!
!
!
!
!
!!
!! !
!
!
!!
!
!
!!!!
!
!!
!
!!
!!!!
! !!! !!
!!
!
!
! !
!!!
!
!!! !!!!!!
!!!!!!
!!
! !
!
!!!
!
!!! !
!!!
!
!!
!
!
!!
! !!
!
!
!!
!!
!
!!
!!
!
!
!
!
!
!!!!!!!
!
!
!!
!!
!
!
! !
!
!
!! !
!
!!!
!
!
!
!
!
!!
!
!
!!!
!!!
!
!
!
!
!!
!
!!
!
!
!!!!
!
!!
!
!
!
!
! !
!
!!!
!
!
!!
!!!
!
!
!
!!!!
!
!!
!!
!!
!!!!!
!!!
!
!!
!!
!
!
!!
!
!!
!!
!!!
!
!
!!
!
!
!!!
! !
!!!
!
!!
!!
!
!!!
!!!
!
!!
!
!! !!
!!!
!!!
! !
!
!
!!
!
!
!!!!
!!
!
!
! !!
!
!!
! !
!
!
!!
!
!
!
!
!
!!!
!
!
!
!
!!!
!!!!
!
!!
!
!! !
!!!!
!
!
!
!!
!
!
!
!
!
!!
!
!!
!!
!
!!!
!
!
!
!
!
! !
!
!!
!
!
!!
!!
!
!
!!!!
!
!
!
!!!
!
!
!
!!!
! !!
!
!
!!
!!
!!
!
!!
!!
!
!
!
!
!
!
!!
!!!!
!!
!!
!!!!
!
!!
!
!!
!!
!
!
!
!!!
!!
!!
!
!
!
!
!
!
!
!!!
!
!!
!
!!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!!
!
!
!
!!!
!!
!
!!
!!
!
!
!!!!
!
!!!
!!!!
!!
!!
!
!
!!!
!
! !!!
!!!!!!
!!
!
!!!
! !!
!
!!
!!!
!
!
!!
!
!!
!
!
!!
!
!
!
!!
!
!
!
!!
!!
!! !!!! ! !!
!!
!
!!
!!!
!
!
!!
!
! !! !
!
! !
!!!!!
!!
! !!
!!
!!
!
!!!
!!
! !!!
! !!
! !!!!
!
!!!
!
!
!!!
!!
!
!
!!!
!
!
!
!
!!!!
! !
! !!!!
!
!
!
!
!!!
!
!!!
!
!!!
!!
!
!!
!
!!!
!!!!
!
!
!!
!!
!!!!
!
!! !
!!
!
!!
!!
!!
!
!!
! !
!
!!!
!
!
!!
!
!!
!!
!
!!!
!
!
!
!
!
!
!!
!!
!!!!
!
!
!
!!
!
!!!!
!
!
!!
!!!
! !
!
!
!
!!
!
!
!
!!! !!
!!
!!
!
!
!
!
!
!!
!!
!! !
!
!! !
!
! !
!
!
!
!!
!
!!!
! !!
!
!
!
!
!!
! !!
!!!
!! !
!!
!
!
!!
!!
!
!!!!
!!
!!
!!
!
!
!!!!
!
!!
!
!!!
!! !!!
!
!!
!
!!!!
!
!!
!
!
!
!
!
!!
!
!
!
!!!
!
!
!
!
!
!
!!
!
! !!
!! !!
!
!!!
!
!!
!!
!!
!
!!!
!
!!!
!!!!
!! !
!
!
!
!
!
!!!
!
!
!!!
!!
!!
!
!
!
!
!!
!
!!
!
!
!
!
!
!!
!
!
!
!
!!
!
!!!
!
! !
!
!
!
!!!!
!!!!
!
!
!!
!
!!!
!
! ! !!!
!
!!
!!
!!
!!
!
! !
!
!!
!!!
!
!!
!
!! !!!!
! !
!
!!
!
!
! !!!
!!
!! !
!!
!
!
!
!
!
!
!
!
!
!
!!
!
! !
!
!
!
!!
!
!!
! !
!
!
!
!!
!
!!
!
!
!
! !!
!
!
!!!! !
!!
!
!!
!!
!
!!
!
!
!
!
!
!
!
!
!
!
!!!
!
!
!
!
!!
!
!
!!!
!!!
!! !
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!!
!
!
!!
! !!
!
!!
!!
!!
!
!
!
!
!!
!
!
!!
!
!
!!! !
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!!
!
!
!!
!
!
!!
!!
!!
!
!
!
!
!
!
!
!!!
!
!
!
!!
!
!!
!!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
!
!
! !
!
!
!
!
!
!!!
!
!
!
!
!! !
!
!
!
!
!
!
!!
!
!!
!!
!
!
!
!
!
!!
!!!!
!!
!!!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!!
!!
!! !
!
!
!
!
!
!
!
!
!!
! !
!!
!!
!
!
!!
!!!
! !!
!!
!
!
!
!
!!
!!
!!
!!
!
!!
!
!
!!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
! !
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!!
!!
!
!!
!!!
!
!!!
!!!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!!
!!
!
!!
!
!
!!
!
!
!!
!
! !
!! !!
!
!
!
!
!!
!
!!!
!!
!
!
!!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!! !
!
!
!
!
!
!
!!
!!!
!
!
!
!
!
!
!!
!
!
!
!
!
!
! !
!
!!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!!
!!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!!!
!
!
!!
!
!
!
!
!!
!
!!
!
!
!
!
!!
!!
!!
!
!!!
!
!!!
!
!
!
!
!!
!
!
!
!!!
!
!
!
!
!!!
!
!
!!
!
!
!
!
!
!
!
!!
! !!!
!
!
!
!!
!
! !
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!
!!!
!!
!!
!!
!!
!
!
!
!!!
!!
!
!
!
!
!
!
! !
!!
!
!
!!
!!
!!
!
!
!
!
!!
!
!!
!
!
!
!!
!
!
!
!!
!
! !!
!! ! !
!
!
!!
!
!! !
!!
!!
!
!
!
!!
!
!!
!
!
!
!
!
!! !
!
!
!!!
!
!
!!!
!
!
!
!
!
!!!
!
!
!!
!! !!
!
!!
!
! !!
!!
!!
!
!
!!
! !!!!
!!
!
!!!
!
!
!
!
!
!
!
!!
!
!
!!!
!!
!!
!!!
!
!
!!
!
!
!
!
!!!
!
! !!
!
!
!
!
!
!!
!
!! !!
!! !
! !!
!! !
!!
!
!!
!
!
!
!
!
!
!
!!!
!!
!!!
!
! !!!
!!
!
!!!
!
!!!
!
!
! !!
!
!
!!
!
!
!
!!
!
0.6 0.7 0.8 0.9 1.0
−4−2
02
invlogit(predict(rf))
resi
d(rf)
/resi
d.se
(rf)
!
!
!
!
!
!
!!!!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!!!
!
!!
!
!
!
!
!
!
!
!
!
! !
!!
!
!
!
!!
!
! !
!!
!!!
!
!!!
!
!!
!! !!
!
!!
!
!
!!
!
!
!
!!
!
!!
! !!
!!!
! !!
!
!
!
!
!
!!
!
!
!!
!
!
!!!
!!
!!!
!!
!!
! !
!
!
!
!
!
!!
!!
!
!!!
!!
!! !!
!
!!
!
!
!
!!!!!
!
!!
!!
!!
!
!
!
!
!
! !
!
!
!!
!
!!
!
!!
!
!
!!
!
!
!
!
!
!
!
!
!
!!!
!!
!!
!
! !
!
!!
!
!
!!
!!
!!
!
!!
! !
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!! !
!
!!!
!
!!!
!
!
!!
!
!
!
!
!
! !!!
!! !
!
!
!!!
!!
!
!
!!
!
!
!
!
!
! !
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!!
!
!
!
!!
!
!
!!
!!!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!!
!!
!!
!!
!!
!
!
!!
!!!
!!!
!
!
!
!
!!
!
!
!
!
!
!
!! !
!
!
!
!
!
!
!!!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!!
! !!
!
!!
!
!
!
!!
!
! !
!!
!!
!!
!
!
!
!
!
!!!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!!
!!
!
!!
!
!! !
!
!
!
!!
!!
!!
!!!
! !
!!
!!
!
! !!
!
!
!!!
!
!
!
!
!
!!
!
!
!
!!
!!
!
!
!!
!
!!
!
!!
!
!!
!
!
!!
!
!
!!
!! ! !! !
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!!
!
!! !!
!!
!
!
!!
!!!!
!!!
!
!!
!!
!!
!
!
!!
!!
! !!!
!
!!
!
!!
!
!
!
!
! !
!
!
!!
!!
!
!!
! !
!
!
!
!!
!!
!!
! !
!
!
!!
!!
!!
!
!
! ! !
!
!
!
!!
!!!
!
!
!!
!
! !
!
!
! !
!
!
!
!!
!
!!!! !!
!!
!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!
!!!
!
!
!
!
!!
!
!!
!
!!
!
!
!!
!
!
!
!
!
!!
!!
!
!
!!
!
!!
!
!
!
!!
!
!
!
!
!
!!
!
! !!
!
!
!
!
!
!!
!
!
!
!!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!!!
!
!!
!
!
!
!!
!!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!!!
!
!
!
!
!
!
!
!
!
!!!
!
!
!
!
!
!!!!
!
!
!
!
!!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!
!
!!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!
!
!
!!!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!
!!
!
!
!
!
!
!!
!!!
!
!
!
! !
! !
!
!
!
!
!
!!
!!
!!!
!
!
!!
!!!
!
! !
!
!!
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!!
!
!
!
! !!!
!
!
!
!
!
!
!
!
!
!
!
!!
!!!
!
!
!
!
!!
!!!
!!
!
!
!!!
!!
!
!!
!
! !
!
!
!!
!
!
!
!
!
!!
!!
!!
!
!!
!
!
!!
!!
!
!!
!
!
!
!
!!
!
!
!
!! !!
!
!
!
!
! !
!!
!!
!!! !!
!
!!
!!!
!! !!!
!
!
!
!
!!
!
!
!
!!
!!
!!
!
!
!
!!
!
!
!!
!
!
!
!! !
!
!
!
!!
!
!
!
!
!
!
!!!
!!!!!
!
!!
!!
!
!
!!
!
!
!! !
!
!!!
!
!
!
!
!
! !
!
!
!!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!!!!
!
!
!
!
!
!!
!!
!
!!!
!
!
!!
!
!
!
!
!
!
!!!
!
!
!!
!
!
!
!
!
!!!!!
!!
!!!
!!
!
!
!
!
!
!!
!!!
!
!!
!
! !
!
!
!
!!
!!
!
!!
!
!
!
!!
!
!!
!
!!
!!
!
!
!
!!!!
!!!
!! !
!!
!!
!!
!
!
!!!
!
!!
!
!
!!
!!
!
!! !
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
! !!!!!
!
!!
!
!! !
!!!!
!
!
!
!!
!!
!
!
!
!!
!
!!
!!
!
!
!!
!
!
!
!
!
!!
!
!!
!
!
!!
!!
!
!
!!!!
!
!
!
!
!! !
!
!
!!
!
!!
!
!
!
!!
!
!!
!
!
!!
!
!
!
!
!!
!
!
!!
!!!!!
!!!
!!!
!
!
! !
!
!!
!
!
!
!
!
!! !
!
!
!
!
!
!
!
!
!
!
!
!!!
!
!!
!
!
!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!!
!!!
!
!
!
!!!
!
!!
!!
!
!!
!
!!!!
!
!!
!
!!
!
!
!!
!!
!
!
!!!
!
!!! !
!
!!!!!
!!
!
!!
!
!
!!
!
!
!
!!!
!
!
!!
!
!!
!
!
!!
!
!
!
!!
!
!
!
!!
!!
!! !!
!
!! !
!
!
!
!
!
!
!!
!
!
!
!
!
!
! !
!!
!
!!
!!!!!
!!
! !!
!!
!!
!
!!!
!
!
!!!!
!!!
!!!
!!
!
!!!
!
!
!!
!!!
!
!
!
!
!!
!
!
!
!
!!!
! !
!!
!! !
!
!
!
!
!!!!
!!
!
!
!!
!
!!
!
!!
!
!!!!!
!!
!
!
!
!
!!
!
!!!
!
!!!
!!
!
!
!!
!
!!
!
!
!
! !
!
!!!
!
!
! !
!
!!
!!
!
!!
!
!
!
!
!
!
!
!!
!!
!!!!
!
!
!
!!
!
!!
!!
!
!
!!
!!!
!!
!
!
!
!
!!
!
!
!
!! !
!
!!
!!
!
!
!
!
!
!!
!
!
!! !
!
!! !
!
! !
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!!!!
!
!!!
!!!
!!
!
!
!!
!!
!! !
!
!
!!
!!
!!
!
!!
!!!
!
!!
!
!
!!!! !!!
!
!!!
!!!!
!!!!
!
!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!!
!
! !!
!!!
!
!
!!
!
!
!!
!!
!!
!
!!!
!
!!!
!!!
!
!!!
!
!
!!
!
!!!
!
!
!!!
!!
!!
!
!
!
!
!!
!
!
!!
!
!
!
!
!!!
!
!
!
!
!
!
!!
!
!
! !
!
!
!
!!!
!
!!
!!
!
!
!!
!
!! !
!
!!
! !!
!
!!
!!
!
!
!!
!
! !
!
!!
!!!
!
!
!
!
!!!
!
!!
!
!
!
!!
!
!
! !!
!
!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!!
!
!!
! !
!
!
!
!!
!
!!
!
!
!
!!
!
!
!
!!!
!!!!
!
!!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!!
!
!
!
!
!!
!
!
!!!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!!
!
!
!!
! !
!
!
!
!
! !
!
!
!
!
!
!
! !
!
!
!!
!
!
!!
!!
!
!!!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
! !
!
!
!
!
!
!!!
!
!
!
!
!
! !
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!!
!!
!!
! !
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!!
!
!!
!
!!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!!
!
!!
!
!
!!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
! !
!!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!!
!!
!
!
!
!
!
!!
!
!!
!
!
!!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!!
!
!
!!
!!
! !
!!
!!
!
!
!!
!
!!
!
!
!
!!
!!
!!
!
!
!
!
!!
!!
! !
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!!
!!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!!
!!
!
!!
!
!
!
!
!!
!
! !
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!!
!!
!!
!!
!
!
!
!!!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!!
!!!
!!
!
!
!!
!
!
!!
!
!
! !
!
!
!
!
!
!
!!
!
!
!
!
!
!! !
!!
!
!!
!
!
!!
! !
!
!
!
!
!!
!
!
!
!!
!
!
!
! !
!
!
!
! !!
!!
!!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!!
!!
!!
!
!
!
!
!!!
!
!
!
!
!!!
!
!!!
!
!
!
!
!
!!
!
!
! !!
!! !
! !
!
!! !
!!
!
!!
!
!
!
!
!
!
!
!!
!
! !
!!
!
!
!!
!!!
!
!
! !!
!
!!!
!
!
! !!
!
!
!
!
!
!
!
!!
!
0.6 0.7 0.8 0.9 1.0
−3−1
12
3
invlogit(predict(rf))
resi
d(rf)
/resi
d.se
(rf)
!
!
!
!
!
!
!
!!!
!
!
!
!
!
!
!
!!
!!
!
!!
!
!!!
!
!!
!
!
!
!
!
!!
!
!
! !
!!
!
!
!
!!
!
! !
! !!
!
!!
!!
!
!
!!
!
!!!
!
!!
!
! !!
!
!!
!!
!
!!
!!
!
! !!
!
!
!
!
!!
!
!
!!
!
!
!
!
!
!
!!!
!!
!!!
!!
!!
! !
!!
!!!
!!
!!
!
!
! !
!!
!!
!!
!
!!
!
!
!
!
!
!!!
!!!!!
!
!
!
!
!
!
!
! !
!
!
!!
!
! !!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!
!
!
!
!
! !
!
!
!
!
!!!
!!
!! !!
!
! !
!
!
!
!
!!
!!
!
!
!
!
!
!!
!!
!
!
!
!
!
!!!!
!
!
!!
!
! !
!
!
!
!!
!
!
!
!
!
! !
!!!!
!
!
!
! !!
!!
!
!!!
!
!
!
!
!!!
!
!
!!
!
!!
!
!
!
!
!!
!
!
! !
!
!
!!
!
!
!
!!
!
!
!!
!!
! !
!
!
!
!
!
!
!
!
!
!!
!!
!!
!!
!!
!!
!
!
!
!
!
!
!
! !!
!
!!!
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!!
!
!!!
!
!
!
!!
!
!
!
!
!
!
! !
!
!
!! !
!!
!
!!
!
!
!
!
!!
!
!
!
!!
!
! !
!!
!! !
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!
! !
!
!
!
!
!
!!
!!
!!!
!
!
!!
!!
!
!! !!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!!!
!
!
!
!
!
!
!!
!
!
!!!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!
!
!
!
!!
!
!!
!
!
!!
!
!
!! !
!! !
! !!
!
!!
!
!
!
!
!
!
!!
!
!
!!!
!
!
!!
!
!
!
!
!
!
!! !
!
!
!!
!!
!
!!
!!
!
!
!
!
!
!
!
! !
! !
!
!
!!!
!
!
!
!
!
! !!
!
!
!
!
!
!
!!
!
!
!!
!
! !
!
!
!
!
!!
!
!!!
!
!
!
!!
!
!!
!
!
!!
!
!
!
!
!
!
!
! !!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
! !!
!
!
!
!
!
!!!
!!
!!
!
!
!!
!
!
!!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!!
!
!!! !!
!
!
!
!!
!
!
!
!
!
!!
!!!
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
! !
!
!
!
!
!
! !!!
!
!
!
!
!
!
!
!
!!!!
!
!
!
!
!!!!
!
!
!
!
!!
!
!
!
! !
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!!!
!
!
!!!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
! !!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!!
!
!
!
!
!
!
!
!!!
!
!
!
!
!
!
!
!
!
!
!!
!
! !
!
!
!!
!
!
!
!
!
!!
!!!
!!
!
!!
! !
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!!!
!
!!
!
!!
!
!!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
! !
!
!!
!
!!
!
!
!
!
! !
!
!
!
!!!
!
!
!
!
!!!
!!
!!
!
!
!!!
! !
!
!!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!!
!!
!
!!
!
!
!
!
!!
!
!
!
!!
!!
!
!
!
!
!!
!
!
!!
!!!!
!
!
!
!! !!!!!
!
!
!
!!
!!!!
!
!
!!
!!!!
!
!!
!
!
!!
! !
!
!
!
!
! !
!
!
!!
!!
!
!
!
!
!
!!!!!!!
!
!
!
! !!
!
!
!!
!
!
!! !
!
!
!!
!
!
!
!
!
! !
!
!
! !!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!!!!
!
!
!
!
!
!
!
!!
!
!!
!
!
!
!!
!
!!
!
!
!
!!! !
!
!!
!!
!!
!!!
!!
!!!
!
!!
! !
!
!
!
!
!
!!
!!!
!
!!
!
!!
!
!
!
!!
! !
!!!
!
!
!
!!
!
!
!
!
!
!!
!
!
!
!
!!
!!
!
!!
!! !
! !
!
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!! !
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
! !!
!! !
!
!!
!
!! !
!
!!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!!
!
!
!!
!
!
!
!
!
!!
!
!!
!
!
!!
!!
!
!
!!!!
!
!
!
!
!! !
!
!!!
!
!!!
!
!
!!
!
!!
!
!
!!
!!
!
!
!!
!
!
!!
!
!!!!
!
!!
!
!!
!
!
! !
!
!
!
!
!
!
!
!
!!!
!!
!
!
!
!
!
!
!
!
!
!!!
!
!!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!!
!
!
!
!!!
!
!
!
!!
!
!
!
!
!!!!
!
!!
!
!!
!
!
!!
! !
!
!
!!
!
!
!
!! !
!
!
!!!!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!
!!
!
!
!
!
!
!
!
!
!!
!!
!! !!!
!
! !!
!
!
!
!
!
!!
!
!
!
!!
!
!!
!
!
!
!!
!!
!!!
!!
! !!
!!
!
!
!
!! !!
!
! !
!!
!!!
!!
!
!!
!
!!!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!!
! !
!
!
!
! !
!
!
!
!
! !!!
!!
!
!
!!
!
!
!
!
!!
!
!!
!!
!
!!
!
!
!
!
!!
!
!!
!
!
!!!
!!
!!
!!!
!!
!
!
!
!
!
!
!!
!!
!
!!
!
!!
!!
!
!!
!
!
!
!!
!
!
!!
!!!
!!!
!
!
!
!!
!
!
!!
!
!
!!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!! !!
!
!
!
!
!
!
!
!
!
!!
!
!
!
! !
!
!! !
!
! !
!
!
!
!
!
!
! !
!
!!
!
!
!
!
!
!!!!
!
!!!
!
! !
!!
!
!
!!
!!
!
! !
!
!
!!!!!!
!
!
!!!!
!
!!!
!
!
!!!
!!!
!
!!!
!
!
!!
!!!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
! !!
!
!
!
!
!
!!
!
!
!
!
!!
!!
!
!!
!
!
!!
!
!!
!
!
!
!!
!
!
!
!
!
!!!
!
!
!!!
!!
!!
!
!
!
!
!!
!
!!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!!
!
!!
!
!
!
!!!!
!!
!
!!
!
!!
!
!
! !
!
! !!!!
!
! !!!
!
!!!
!
! !
!
!!
!!
!
!
!!
!
!!!
!
!!! !
!!
!!
!!
!!
!
!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!!!
! !
!!
!
!
!
!!
!
!!
!
!
!
!!
!
!
!
!!!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!!!
!!!
!
! !
!
!
!
!
!
!
!
!
!!
!
!
!!
!
!
!
!!
!!
!
!
!
!
! !
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!!
!
!
!!
! !
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
!!
!
!!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!
!
!
!
!!!
!
!
!
!
!
! !
!!
!
!!
!
!
!
!
!!
!
!
!
!
!
!
!!!
!!!
!
!!
! !
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!!
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!!
!
!
!!
!
!!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!!
!
!!
!
!
!!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!!
!
!
!
!
!
!
!
!!!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
! !
!
!
!
!
! !
!
!!
!
!
!!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
! !
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!!
!!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!
! !
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!
!
!
!
!
!
! !
!
!!
!
!
!
!
!!
!
!
!
!
!
!!
!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!
!
!
!
!!
!
!
!
!
!!
!! !
!!
!!
!
!
!
!
!
! !
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!!
!!
!!
!!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!!
!
!
!!
!
!
!
!
!
!
!!
!!
!
!
!!
!
!
!
!
!
!
!!
!!!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!! !
!
!
!
!!
!
!
!
!
! !
!
!
!
!
!!!
!
!
!
!
!
!!
!!
!!
!
!!
! !
!
!
!
!
!
!
!
! !
!
!!
!!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!!!
!
!
!!
!
!!
!
!!
!
!
!
!
!
!!!
!
! !!
!
!
!
!!
!!
!
!! !!
!!
!
! !
!
!!
!
!!
!
!
! !
!
!
!
!
!
!
!!
!
! !
!!!
!
!!
!!!
!
!
!!!
!
!!!
!
!
! !!
!
!
!
!
!
!
!
!
!
!
0.6 0.7 0.8 0.9 1.0
−20
12
3
resi
d(rf,
inv
= TR
UE)
/resi
d.se
(rf, i
nv =
TRU
E)
!
!
!
!
!
!
!!!!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!!!
!
!
!!
!
!
!
!
!!!
!
! !
!!!
!
!
! !
!
! !
! !
!
!
!!
!!!
!
!
!
!! !!!
!!
!
!!!
!
!
!
! !!!
!! !!
! !! !
!
!!
!!
!!
!!
!
!
!
!
!
!
!!!
!!!!!
!!
!!
! !
!!
!!!
!
!
!!
!
!! !
!
!
!!!
!!
!!
!!
!
!!!
!!
!!!!!
!!
!!
!
!
!
!
!
!!
!
!
!
! !!!
!
!
!!!
!
!
!
!
!
!
!!
!
! !!!!
!!
!
!!
!
!!
!!!!
!!
!!! !
!
! !
!
!!
!
!!
!!
!
!
!
!
!!
!!!
!
!
!
!
!!!! !
!
!!!
!
!
!!
!
!!
!
!
!
!
!
!
! !!!
!
!
!
!
!
! !
!
!!
!
!!!
!
!!
!
!!!
!
!
!!
!
!
!!
!!
!!!!
! !
!!
!
!!
!
!
!!!
!
!
!!
!!!
!!
!
!
!
!!
!
!
!
!!! !
!!!!
!!
!!
!
!
!!
!
!
!! !!!
! !!
!
!
!
!
!!
!
!
!
!
!
!
!! !
!
!
!
!!
!
!!!
!
!!
!!
!!
!
!
!!
! !
!
!
!!!
!!
!
! !! !!
!!!
!
!!
!!
!
! !
!!
!!
!!
!
!
!!
!
! !
!!
!
!
!
!
!
!
!
!!!
!
!
!
!
!!!
!
!!
!
!!!
!
!!
!
!
!
!
!
!!
! !
!
!!
!
!
!!!
!
!
!!
!!
!
!!!
!
!
!
!!
!!
!
!!
!!
!!
!
!
! !
!
!
!!
!!
!
!!
!
!
!!
!
!
!!!
!!
!!!
!
!
!
!
!
!
!!!
!
!!
!
!!!
!
!
!
!
!!
!
!!
!!
!!
!
!
!! !
!! ! !
!! !
!!!!
!!
!
! !!!!
! !!!
!
!!
!
!
!
!
!
!
!! !
!
!
!!
! !
!
! !!
!
!!
!
!
!
!!
! !
! !
!
!
!!
!
!
!!
!
!
! !!
!
!
!
! !
!!!
!
!
!!
!
! !
!
!
!!
!!
!
!!!
!!
!! !!!
!
!
!
!!
!
!
!!
!
!
!
! !!!
!!
!
!!!
!
!
! !!
!
!!
!!
!
!
!! !
!!!!!!!
!!
!
!
!!
!
!
!
!!
!
!!
!
!
!
!
!
!!
!
! !!
!
!!
!
!
!!!
! !
!
!
!
!
!!
!
!
!!
!
!
!!
!
!
!
!
!
!!
!! !
!
!
!
!
!
!
!!
!
!
!!
!!
!
!
!!
!
!
!
!
!
!!
!
!
!
!
!
!!
!!!
!
!!
!
!
!
!
!
!
!
!
!
!!!
!
!
!!
!
!!
!!
!
!
!
!
!
!!! !
!
!
!
!
!
!
!
!
!!! !
!!
!
!!!
!!!
!
!!!!
!
!!
!!
!!
!
!
!!!
!
!!!
!
!
!
!
!
!
!
!
!
!!!
!
!!!
!!
!
!!!
!
!
!!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!! !
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
! !!
!!
!
!
!
! !!!
!
!
!
!
!
!
!
!!
!
!!
!
! !
!
!
!!
!
!
!
!
!!!
!!!
!!
!
! !
! !
!
!
!!
!
!
!
!!
!
!
!
!
!
!!
!!!
!
! !
!
!!
!
!
!!
!
!!!
!
!!
!
!
!!
!
!
!
!
!
! !
!
!
!
! !!
!
!
!!
!
!!!
!
!!
!
!!
!!
!
!!
!!! !
!!
!
!!!!
!
!!
!
!!!!! !
!!!
!
!
!
!
!
!! !
!
!
!
!
!!!
! !!
!
!!
!
!
!! !!
!
!!
!
!!
!!!!! !!! !!
! !
!!! !
!!!
!
!!! !!!!!! !!!!!!!!
! !!
!!!!
!!!!! !!
!!!
!
!
!!
! !!
!
!
!!
!
!
!
!!
!!
!
!
!
!
!
!!!!!!!
!
!
!!
!!
!
!
! !
!
!
!! !
!
!!!
!
!
!
!
!
! !
!
!
!!!
!
!!
!
!
!
!
!!
!
!!
!
!
!!!!
!
!!
!
!
!!
! !
!!!!
!!
!!
!!!
!!
!
!!!!
!!
!
!!
!!!!!!!
!!!
!
!!
!!
!
!
!!
!
!!
!!
!!!
!
!
!!
!
!
!!!
! !
!!!
!
!!
!!
!
!!!
!!!
!
!!
!
!! !!
!
!!!!
!
!!
!!
!!
!
!
!!!!
!!
!!
!!
!!!!
! !!
!
!!
!
!
!
!!
!!!
!
!
!
!
!
!!!
!!!
!
!!
!
!! !
! !!!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!!
!!!!
!
!
!
!
!
!!
!!
!
!
!
!!!!
!
!
!!!!
!
!
!!!! !
!
!!!!
!!
!
!
!!!
!!!!
!
!!
!!!!!!
!
!
!!
!!!
!
!!
!!
!!!!
!
!!
!
!!
!
!
!
!
!
!!!
!!
!!
!
!
!
!
!
!
!
!!!!
!!
!
!!!
! !!
!
!
!
!
!
!!
!
!
!
!
!!
!!!
!
!
!
!!!
!!
!!!
!!!
!
!!!!!!!!
!!!!
!!
! !
!
!
!!!
!
!!!!
!!!!!
!!!
!!
!!
!!!
!
!!
!!!
!
!
!!
!
!!
!
!
!!
!
!
!
!!
!
!
!
!!
!!
!! !!!
! ! !!!
!
!
!!!!
!
!
!
!
!
!
! !! !
!
! !
!!!!!
! !
! !!
!!
!!!!! ! !!! !!!
! !!! !!!!
!!!!!
!
!!!
!!
!!
!!!
!
!
!
!
!
!!!
! !
!!!
!!
!
!
!
!
!!!
!
!!!
!
!!!
!!
!
!! !
!!!
!!!!
!
!
!!
!!!
!!!
!
!! !
!!
!!!
!!
!!
!
!!
!!
!!! !!
!
!!
!
!!
!!
!
!!
!
!
!
!!
!
!
!!
!!
!!!!
!
!
!
!!
!
!!!!
!
!
!!
!!!
! !
!
!
!
!!
!
!
!
!!!!
!
!!
!!
!
!
!
!
!
!!
!!
!! !
!
!! !
!
! !
!
!
!
!!
!
!!!
! !!
!
!
!
!
!!
! !!!!!
!! !
!!
!!
!!
!!
!
! !!!
!!
!!!!
!
!!!!!
!
!! !
!!!
!! !!!
!
!!!
!!!!
!
!!!
!
!
!!
!!
!
!
!
!!!
!
!
!
!
!
!
!!
!
! !!
!! !!
!
!!!
!
!!
!!
!!
!
!!!
!
!!!
!!!!
!! !
!
!
!!
!
!!!
!
!
!!!
!!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!!
!
! !
!
!
!
!!!!
!!!
!!
!
!!
!
!!!
!
! ! !!!
!
!!
!!
!!
!!
!
! !!
!!!!!
!!!
!
!! !!!!
! !
!!!
!!
! !! !
!!
!! !
!
!
!
!
!!
!
!
!
!
!
!
!!
!
! !
!
!
!
!!
!
!!
! !
!
!
!
!!
!
!!
!!
!
!!
!
!
!
!!!!
!!!
!
!!
!!
!
!!
!
!
!
!
!
!
!
!
!
!
!! !
!
!
!
!
!!!
!
!
!!!
!!
!!
!
!
!
!
!
!!
!
!
!
!
!
!!!
!
!
!
!!
!!
!
!
!
!
! !
!!
!
!
! !
!
!
!
!
!
!
!!
!
!
!!
!
!
!!! !
!
!!!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!!
!!
!
!
!
!!
!
!
!
!
!!!
!
!
!
!
!!
!!
!!
!
!!
!
!
!
!!
!
!!
!
!
!
!!
!
!
!
!
!
! !
!
!
!
!
!
!!!
!
!
!
!
!!
!
!!
!
!!
!
!!
!
!!
!!
!
!
!
!
!!
!
!!!!
!!
!!!
!!
!
!
!
!
!
!!!
!!
!
!!
!!
!!
!! !
!
!
!
!
!
!!
!
!!
! !
!!
!!
!
!
!!
! !!
! !!
!!
!
!
!
!
!!
!
!!
!
!!
!
!!
!
!
!!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!!!
!
!!
!
!!
!!
!! !!!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!!
!!
!
!!
!
!
!!
!
!
! !
!
! !!
! !!
!
!
!
!
!!
!
!!!
!!
!
!
!!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
! !
!
!
!
!!
!
!!
!!!
!!
!
!
!!
!!
!
!
!
!
!
!
! !!
! !
!
!
!
!
! !
!!
!
!!
!
!
!
!
!
!
!!
!
!!
!
!
!!
!
!!
!
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!!!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!!
!
!
!!
!
!
!
!
!!
!
!!
!
!
!
!
!! !
!
!!
!
!!!
!
! !!
!
!
!
!
!!
!
!
!
!!!
!
!
!
!!!!
!
!
!!
!
!
!
!
!
!!
!!
!!!
!!
!!
!!
!! !
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!
!!!
!!
!!
!!
!!!
!
!
!!!
!
!
!
!
!
!!
!
! !!!
!
!
!!
!
!
!!
!
!
!
!
!!
!!
!!
!
!
!!
!
!
!
!!
!
! !!
!! ! !!
!
!!
!
!! !
!!
!!
!
!
!
!!
!
!!
!
!
!
!
!
!! !
!
!
!!!
!
!
!!!
!
!
!
!
!
!!!
!
!
!!
!!
!!
!
!!
!
! !!
!!
!
!
!
!
!!
!!!
!!
!!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!!!
!!
!!
!!!
!
!!!
!!
!
!
!!!!
! !!
!
!
!
!!
!!
!
!! !!
!!
!
! !!
!!
!!
!
!
!! !
!
!
!!
!
!
!!!
!!
!!!!
! ! !!!
!
!
!!!
!
!! !
!
!
! !!
!
!
!!
!
!
!
!!
!
0.6 0.7 0.8 0.9 1.0
−20
24
resi
d(rf,
inv
= TR
UE)
/resi
d.se
(rf, i
nv =
TRU
E)
!
!
!
!
!
!
!!!!
!
!
!
!
!
!
!
!
!
! !
!
!!
!
!!!
!
!
!!
!
!
!
!
!
!
!
!
! !
!!
!
!
!
!!
!
!!
!!
!!!!
!!!
!
!!
!! !
!!
!!
!
!
!!
!
!
!
!!
!
!!
! !
!
!!!! !
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!!
!!
!!!
!!
!!
!!
!
!
!!
!
!
!
!!
!
!
!!
!
!
!!
!!
!
!!
!
!
!
!!!!
!
!
!!
!!
!!
!
!
!
!
!
! !
!
!
!!
!
! !
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!!!
!!
!!
!
! !
!
!!
!
!
!!
!
!
!!!
!!
! !
!
!
!
!
!!
!
!
!
!
!
!
!
!! !
!
!
!
!
!
!!!! !
!
!!
!
!
!!!
!
!!
!
!
!
!
!
!
! !!!
!! !
!
!
!!
!
!!
!
!
!!
!
!!
!
!! !
!
!
!!
!
!
!!
!
!
!
!!
!
!
!! !
!
!!
!
!
!
!!
!
!
!!
!!!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
! !
!!
!!
!
!
!!
!
!
!!
!!! !!!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!!!
!!
!
!!
! !!
!
!!
!
!
!
!!
!
! !
!!
!
!
!!
!
!
!
!
!
!!!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!!
!!
!
!!
!
!
! !
!
!
!
!
!
!!
!!
!!!
!
!
!
!
!!
!
! !!
!
!
!!!
!
!
!
!
!
!!
!
!
!
! !
!!
!
!
!!
!
!!
!
!!
!
!!
!
!
!!
!
!
!!
!!!
!! !
!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!
!
!
!
!!
!
!!
!!
!!
!
!
!!!!
!!
! !!!
!!
!!
!!
!
!
!!
!!
!!!
!
!
!!
!
!
!
!
!
!
!
! !
!
!
!!
!
!
!
!!
!!
!
!
!
!!
!!
!!
! !
!
!
!!
!
!
!!
!
!
! ! !
!
!
!
!!
!
!!!
!
!!
!
! !
!
!
! !
!
!
!
!
!!
!!!! !
!
!!
!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!!
!!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!
!
!
!
!
!!
!!
!
!
!!
!
!
!
!
!
!
!!
!!
!
!
!
!!
!
! !!
!
!
!
!
!
!!
!
!
!
!!
!
!
!!
!
!
! !
!
!
!!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!!!!
!
!
!
!
!
!!
!
!
!
!
!!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!!!
!
!
!
!
!
!
!
!
!!
!
!!
!
!
!!
!!!
!
!
!
!!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!!
!
!
!!!
!
!
!!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!
!
!
!!!
!
!
!
!
!
!
!
!
!
!
!!
!
! !
!
!
!!
!
!
!
!
!
!!
!!!
!
!
!
! !
! !
!
!
!
!
!
!!
!!
!!!
!
!
!!
!!!
!
! !
!
!!
!
!
!
!
!
!!!
!
!!
!
!
!
!
!
!
!
!
!
! !
!
!
!
! !!
!
!
! !!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!!!
!
!
!
!
!!
!!!
!!
!
!!!!!!
!
!!
!
! !
!
!
!!
!
!
!
!
!
!!
!
!
!!
!
!!
!!
!!
!!
!
!!
!
!
!
!
!!
!
!
!!
!!!
!!
!
!
! !
!!
!!
!!! !!
!
!!
!!!
!! !!!
!
!
!
!
!!!
!
!!
!!
!
!!
!
!
!
!!
!!
!!
!
!
!
!! !
!
!
!
!!!
!
!
!
!
!
!!!
!!!!!
!
!
!!!
!
!
!!
!
!
!! !
!
!
!!
!
!
!
!
!
! !
!
!
!!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!!!!
!
!
!
!
!
!!
!!
!
!!!
!
!
!!
!
!
!
!
!
!
!!! !
!
!!
!!
!
!
!
!!!!!
!!
!!!
!!
!
!
!
!
!
!!
!!!!
!!
!
! !
!
!
!!
!
!!
!!!
!
!!
!
!
!
!!
!
!
!!
!!
!
!
!!!!
!
!!
!! !
!!
!!
!!
!
!
!!!
!!
!
!
!
!!
!!
!!
! !!
!
!
!
!
!
!
!
!
!!!
!
!
!
!
!
! !!
!!!
!
!
!
!
!
! !
!!!!
!
!
!
!
!
!!
!
!
!
! !
!
!!
!!
!!
!!
!
!
!
!
!
!!
!!
!!
!
!
!
!!
!
!!
!!!
!
!
!
!
!!!
!
!
!!
!
!
!!
!
!
!!
!
!!!
!
!
!
!!
!
!
!!
!
!
!!
!!!!!
!!!
!!!
!
!
! !
!
!
!!
!
!
!
!
!!!
!!
!
!
!
!
!
!
!
!
!
!!!
!
!!
!
!
!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!!
!!!
!
!
!
!!!
!
!!
!!
!!!
!
!!!!!
!!!
!!!!
!!
! !
!
!
!!!
!
!!! !
!
!!
!!!!!
!
!!
!
!
!!
!
!
!
!!!
!
!
!!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!!
!!
!! !!
!
!! !
!
!!
!
!!
!!
!
!
!
!!
!
! !
!!
!
!!
!!!!!
!!
! !!
!!
!!!!!
! !
!
!!!!
!!!
!!!!!
!
!!!!
!
!!
!!!
!!
!!
!!
!
!
!
!
!!!
! !
!!
!
! !
!
!
!
!
!!!!
!!
!
!
!!
!
!!
!
!! !
!!!!!
!!
!
!
!!
!!
!
!!!
!
!!!
!!
!
!
!!
!
!!
!
!
!
! !
!!!!
!
!
! !
!
!!
!!
!
!!
!
!
!
!!
!
!
!
!
!!
!!!
!
!
!
!
!!
!
!!
!!
!
!
!
!
!!!
!!
!
!
!
!
!!
!
!
!
!! !
!
!
!
!!
!
!
!
!
!
!!
!
!
!
! !
!
!! !
!
! !
!
!
!
!!
!
!!
!
! !!
!
!
!
!
!!!!
!
!!!
!!!
!!
!!
!!
!!
!! !
!!
!!!!
!!
!
!!
!!!
!
!!
!!
!!!! !!!
!
!!!
!!!!
!!!!
!
!
!!
!!
!
!
!
!!!
!
!
!
!
!
!
!!
!
! !!
!!!
!
!
!!
!
!
!!
!!
!!!
!!!
!
!!!
!!!
!
!
!!
!
!
!!
!
!!!
!
!
!!!
!!
!!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!!
!
!
!
!
!
!
!!!
!
! !
!
!
!
!!!
!
!!
!
!!
!
!!
!
!!
!
!
!!! !!
!
!!
!!
!
!
!!
!
! !
!
!!
!!!
!!
!
!
!!!
!
!!
!!
!!!
!!
! !!
!
!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
! !
!
!
!
!!
!
!!
! !
!
!
!
!!
!
!! !
!
!
!
!
!
!
!
!!!
!!
!!
!
!!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!!
!
!
!
!
!!
!
!
!!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!!
!
!
!!
!!
!
!
!
!
! !
!
!
!
!
!
!
!!
!
!
!!
!
!
!!
! !
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
! !
!
!
!
!
!
!!!
!
!
!
!
!
! !
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!!!!
!!
! !
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!!
!!
!! !
!
!
!
!
!
!!
!
!!
!!
!
!
!!!
!
!!
!
!!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!!
!
!!
!
!
!!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!!
!
!
!
!
!
!!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!!
! !
!!!!
!!
!!
!!
!
!
!!
!
!!
!
!
!
!!
! !
!!
!
!
!
!
!
!
! !!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!!
!! !
!
!
!
!
!
!
!!
!
!
!
!
!!
! !!
! !
!
!
!
!
! !
!
!!
!!
!!
!
!
!
!
!
!!
! !
!
!
!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!
!
!
!! !!
!
!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
! !!!
!
!
!
!
!
!
!
!
!
!!
!!
!
!!
!
!
!
!
!
!
!! !
!
!!
!
!
!
!
!
!
!!
!
!!
!
!
!
!!
!!
!!
!!
!!
!
!
!
!!!
!
!
!!
!
!
!
!
!!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!!
!
!
!
!!
!
!
!
!
!
!
!
!!
!!!
!!
!
!!
!
!
!!
!
!
! !
!
!
!
!
!
!
!!
!
!
!
!
!
!! !
!!
!
!!
!
!
!!
!!
!
!
!
!
!! !
!
!
!!
!
!
!
!!
!
!
!
! !!
!!
!
!
!
!
!
!
!!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!!
!
!
!!
!
!
!
!
!!!
!
!
!
!
!!!
!
!!!
!
!
!
!
!
!!
!
!
! !!
!!
!
! !
!
!! !
!!
!
!!
!
!
!
!!
!
!
!!
!
! !
!!!
!
!!
!!!
!
!
! !!
!
!!!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
0.6 0.7 0.8 0.9 1.0
−20
12
3
resi
d(rf,
inv
= TR
UE)
/resi
d.se
(rf, i
nv =
TRU
E)
13
Common method variance
!
!!!!
!!
!!
!
!
!!
!
!!
!
!
!
!!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!
!
!
! !
!
!
!
!
!
!
!
!
!
!!
!
!!!
!
!
!
!!
!!
!!
!
!!
!
!
!!!!!!
!
!
!
!!
!!!
!
!
!
!
!!!!!!
!!
!
!!
!!!!
!
!
!
!
!!!
!!
!
!
!
!!
!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!
!!!!!
!
!
!!!!!!!!!! !!!!!!!
!
!!!!!!!!!
!
!!!!!!! !
!!
!!!!!!
!!
!!!!!!!!!!!!!!!
!!
!!!!!!
!!!!!!!!!!
!!
!
!!!
!!
!
!!!!!!!! !
!
!!!! !
!
!
!
!!
!!
!!!
!!
!
!
!! !!
!!
!!
!
!! !!!
!!
!!
!!
!
!
!
!!
!
!!
!!!
!!
!
!!! !!!!
!
!!!!!
!
!!!
!!!
!!
!
!
!
!!!!
!!
!!!
!
!!!!!
!
!
!!!!!
!!!!
!!!!!!!
!
!!!!!
!!
!
!
!
!!
!
!
!!
!
!
!!
!!!!
!
!! !
!!
!!!
!!
!!!!
!
!
! !
!
!!
!!!!
!
!
!
!!!
!
!
!!
!
!!
!
!!!
!!
!!!
!!
!!!
!
!!
!
!!!
! !
!!!
!
!!!
!
!
!!
!
!
! !!
!!
!
!!
!
!
!
!! !!! !
!
!!!!
!!!!!
!!
!!!!
!!
!
!!!!!
!
!!!
!!
!!!!
!!! !!!
!
!!!
!
!
!
!
!!
!
!!
!
!!!!
!!!!!!
!
!
!
!!
!
!
!!!!!!!!!!!!!!!!!!
!!!
!!
!
!!!
!
!!
!!!
!
!!
!!!!!!!!!!!!!!!
!
!! !!!!!!!!!
!!
!
!!
!
!
!!!!!!!!!!
!!
!
!!!!
!!
!!!!!!
!
!
!!!!
!!
!!!!
!!
!!!
!!
!
!
!!!!!
!!!!
!!!!!!
!
!
!
!!
!!!!!!!
!!!
!!!
!!
!
!!
!
!
!!
!
!
!! !!!
!
!!!!!! !
!
!!
!!
!!!! !
!
!
!!!!!!! !!
!
!
!! !
!!!!
!
!
!!!
!
!!! !
!!!!
!
!
!
!
!!!!!!!!
!
!!!
!
!!!
!
!
!
!!!!!!!
!
!
!!!
!!!!
!
!
!
!!!!
!!!!
!
!!!!!!!!!!!!
!!!!!!!!!!!!!!!
!
!!
!
!!!
!!!!
!
!!!!!!
!
!!
!
!!
! !
!! !!!!!!!
!
!!!!!!
!
!
!
!
!
!!!!
!!!!!!!!!!!!
!
!!
!
!!!
!
!
!!!
!
!!!!!! !!
!!
!!!!!!!!!!!!!!!!!
!!!!
! !!!!!!!!
!!!!!!!
!!!!!!!!!!!!!!!!
!!
!!
!
!
!!!! !!!
!
!!!!!!!
!
!
!
!!
!!!
!!
!
!
!!!!!!
!
!!!
!
!!!
!!!
!!!
!
!
!
!
!!!!! !!!!!!!!!
!
!!!!!!!!!
!
!!!!!!!
!
!!!!
!!!
!
!!!!
!
!!
!!
!
!!!
!
!!
!!!
!
!!
!!
!!
!!!!!
!
!!!!
!
!
!!
!!
!!
!
!
!!
!!
!!
!!
!
!
!!!
!!!
!
!
!!
!
!
!!
!!!!!!!!!!!!
!!
!!
!!!
!
!
!!
!
!
!!!!
!!!!
!
!!!
!!!!!!!!!!
!!!!!!!!
!!!!
!!!
!!
!!
!!!
!
!
!
!
!
!
!
!
!!!
!
!!!!!
!
!!
!
!!
!!!
!
!!!!!!!!!!!
!
!!!!!!
!!!!!!!!!
!!
!
!!
!!
!
!!!
!!!!!!
! !
!!
!
!
!!
!!!!
!
!!!!
!!
!
!
!!!!
!!!!
!!!!
!
!!
! !
!!!!!
!!!!! !!!!
!!
!!!!!!
!
!!
! !!!
!
!
!!
!!
!
!!!
!!!!!
!!!!!
!
!
!
!!!
!!
!!!!!!!!
!
!!
!!!!!!!!!
!!!
!
!
! !! !!!
!!!
!!!!!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!!!
!!!!!!!!!!
!!!
!!!!!!!!!!!!!
!!!!!!
!!!
!
!
!!!!!!!!
!
!!! !!!!
!!!!
!!!!!!!!
!
!!
!!!!!!
!!
!!!!!!!!!
! !!
!!
! !!!!!!!!!!!!
!!!
!
!!
!!!!!!
!!!!
!
! !!!!!!!!!!
!
!
!!!
!
!
!!!
!!!
!!!
!
!
!!
!
!
!!!
!!!
!!!!
!
!
!!! !
!
!!!
!
!!!
!!!
!
!
!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!
!!!!!!! !!
!!!!!!!!!!!!!
!
!!!
!!!!! !!!!!!
!
!!! !!!!
!!!!
! !!!!
!!
!!!!!!
!
!!!!
!!!!!!!!!! !
!!!!!
!!!!
!
!!!!!
!!
! !
!
!!!!!!!
!
!!
!
!!!! !! !
!!!
!!!!
!
!!!!!!!!!
!
!
!
!!!!!! !!!
!
!
!!
!
!
!!
!!!
!! !
!!!!
!!
!
!!
!!!
!!
! !!!
!
!!!
!
!
!!
!
!
!!!!!
!!!!!!!
!!
!!
!!
!!!
!!!!
!!
!!!!!!
!
!!!!!!!
!!
!!!
!!
!!
!!
!!
!!
!!
!!
!
!
!!
!
!
!
!
!!
!!!!
! !
!!
!
!
!!
!!
! !
!!!
!!!
!! !!
!
!
!!!!!!!
!!! !!!!!!!!
!
!
!!
!
!
!
!
!
!
!!
!
!!!
!!!!
!!
!
!
!
!!!
!
!!!
!
!!
!
!!!!
!!
!!
!!
!
!
!!!!!!!
!
! !!!!!!!
!!!!
!!!!
!!!!!!
!!!!!!!!!
!
!!
!!!!
!!!!!!!!!!!
!!!
!!!
!!!!!!! !
!!!!
!
!!!!
!!
!
!!!
!!
!!!
!!
!!!!!
!!!!! !!!
!
!!!
!!!!
!!!!
! !!!
!!!!
! !!! !!
!!
!!!
!
!!!
!!! !!
!!!
!
!!!!!
!!!
! !!!
!!!!
!!!!!!!!
!!!!
!!!!
!!!!
!!! !!!
!!! !!
!!
!!
!!
!
!!!!!!
!!!
!!!!
!
!!
!
!!!!
!!!
!!!
!!
!
!!!
!!!!
!!
!!! !!! !
!
!
!!!!
!!
!!!!!!
!
!!!!!
!!!
!!! !
!!!!! !
!
!
!!!!!!
!!!
!!!
!!!
!!!
!!!
!!!!
!!
!!!
!!!
!!!
!
!! !!!!!!
!!!
!!!
!!!
!!!!!
!!!! !!!
!!
!
!
!!
!!! !!!
!!!!
!!
!!!!!!!!!
!!
!
!!!
!
!
!!
!
!
!
!
!
!
!
!
!
!!!
!!!!!!
!
!!
!
!!!
!! !!!!!!
!!!
!!! !!
!!!
!
!!!
!!!!
!
!!!!!!!
!
!
!!!! !
!! !!!!!!!!!!!!
!!!!!!!!!!!!
!!!!!!!!!! !! !!!!!!!!!
!!!
!!! !!!!!!!!!!!! !!!!!!
!!!
!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!
!!!
!!!!!!!
!
!!!!
!!
!
!!
!!!
!
!!!
!!!
!!!
!!
!
!!!
!
!!
!!!
!!
! !!!!!
! !
!
!!
!!
!!
!
!!!
!!
!
!!!
!
!!!!!
! !!
!
!
!
!!!
!!! !!!
!
!!
!
! !
!
!
!!!
!! !!!!!!!!!!!
!!
!!!
!!!!
!!
!!
!
!!
!!!!
!!!
!!!!!!!!!!
!!!!!!!!
!
!!!
!!!
!!
!
!!
!!!!
!!
!!
!!!!
!
!!!!!!!
!!!!
!!!!
!!!!!!!
!!!!
!
! !!!
!!!!!!!!!!!!
!!! !
!
!!!!
!!!
!!!!!!
! !!
!
!!
!!
! !
!! !!!!
!!!!
!!!!
!!!!
!!
!
!
!
!
!
!
!
!
!
!!
!!
!!!
!!
!!!!!!
!
!!!!!
!!!!!!
!!!!!!
!
!
!
!
!
!!
!
!
!!
!
!
!!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!!
!!!
!
!!!!!
!!!!!!
!!!
!!! !
!!
!!
!!
!!!!
!!!
!!!!
0 1 2 3 4 5 6
−3−1
12
predict(rf)
resi
d(rf)
!
! !!!
!!
!!
!
!
! !
!
!!
!
!
!
!!!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!!!
!!
!
!!
!
!
!!
!
!
!
!
!
!!!
!!!
!
!
!
!!
!!!!
!
!
!
!
!
!!!!!
!
!
!
!
!!
!!!
!!
!
!
!!!!!
!
!!
!
!
!
!!!!!
!
!
!
!!!!!
!
!
!
!
!!!!
!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!
!!!!!!!!!!
!!
!!!!!
!
!!!
!!!!!!
!
!!!!!!! !
!!!!!!!!
!!
!!!!!!!!!!
!!!!!
!!!
!!!!
!!!!!
!!!!!!
!!!
!!!
!!
!
!!!
!!!!! !
!
!!!! !
!
!
!
!
!
!!!!!!
!!
!
!! !!
!!
!!
!
!!!!
!
!!
!!
!!
!
!
!
!!
!!!
!!!!
!
!
!!! !!!!
!
!!
!!!
!
!!! !!!
!!
!
!
!
!!!!!!!!!
!!!!
!!
!
!
! !!!!!!!
!!!!!!!!
!
!!!!!
!!
!!
!
!!
!
!!!
!
!
!!
!!!
!!
!! !
!!!!!
!!
!!!!
!
!
! !
!
!
!!
!!!
!
!
!!!!
!
!
!!
!
!
!
!!
!!
!!
!!!
!
!!!!
!
!!
!
!!!
!!
!!!
!
!!!
!
!
!!
!
!
!!
!
!!
!
!!
!!
!
!!!!! !
!
!!!
!!
!!!!
!
!
!!!!
!!
!
!
!!
!
!!!!!
!!
!!!!!!
!!!!
!
!!!!!
!
!
!!
!
!!
!
! !!!!!!!!!
!
!
!!
!
!
!
!!!!!!!!!!!!!!!!!!! !!
!!
!
!!!
!
!!
!!!!
!!
!!!!!!!!!!!!!!!
!
!! !!!!!!!!!!!
!
!!
!!
!!! !!!!!!
!!
!!
!!!!
!!
!!!!!!
!
!!!!!
!!
!
!!!!!!!!!
!!
!
!!!!
!!!!
!
!!!!!!!
!
!
!!
!!!!!!!
!!
!!!
!!
!
!
!!
!
!!
!
!
!
!! !!!
!
!!!!!! !
!
!!
!!
!
!!! !
!
!!!!
!
!!! !!
!
!
!! !
!!! !!
!
!!!
!
!!! !!!!!
!
!
!
!
!!!!!!!!
!
!!!
!
!!!
!
!
!
!!!!!!!!
!
!!!
!!!!
!
!
!
!!!
!!!!!
!
!!!!!!!!!!
!!!
!!!!!!!!!!
!!!!
!
!!
!
!!!
!!!!
!!!!!!!
!
!!
!
!!
! !
!!!!!!!
!!
!
!!!!!!
!
!
!
!
!
!!!!!
!!
!!!!!!!!!!
!!
!!!
!
!
!
!!!
!
!!!
!!! !!!
!
!!!!!!
!!!!!!!!!!!
!!!!
! !!!
!!!!
!!!!!!!!
!!!!!!!!!!!!!!!!
!!
!!
!
!!!!
! !!!
!
!!!!!! !
!
!
!
!!
!!!
!!
!
!!
!!!!!!
!!!
!
!!!
!!!
!!!
!
!
!
!
!!!!!!
!!!!!!!!!
!!!!!!!!!
!
!!!!!!!
!
!!!!
!!!
!
!!!!
!
!!!!
!
!!!
!
!!
!!!
!
!
!
!!!
!
!!!
!!
!
!!!
!!
!
!
!!!
!!
!
!!!!
!
!!
!!
!
!
!!!
!!
!!
!
!!
!
!
!!
!!!!!!
!!!!!
!!!
!!
!!!
!
!
!!
!!
!!!!
!!!!
!
!!
!
!!!!!!!!!!
!!!!!!
!
!!!
!!!!!!
!
!!!!!
!
!
!
!
!
!
!
!
!!!
!
!!!!!
!
!
!
!
!!
!!!
!
!!!!!!!!! !!
!
!!!!!
!
!!!!!!!!!
!
!
!
!
!!!
!
!!!
!
!!!!!
! !
!!
!
!
!!!
!!!
!
!!!!
!!
!!
!!!!
!!
!!
!!!!
!
!!
! !
!!!
!!
!!!
!!!!
!!
!!
!!!
!!!
!
!
!
!!!
!
!
!!
!
!!
!
!!!
!!!!!
!!!!!
!
!
!
!!!
!!
!!!!!!
!!
!
!!
!!!!!!!!!!!!!
!
! !!
!!!!!
!!!!!!
!!
!!
!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!
!!
!!!!!!!!!!
!!!!!! !
!!!!!
!!!!!!!!
!!!!!!
!!
!
!
!!
!!!!!!
!
!!! !!!!
!!!!
!!!!
!!!
!!
!!!!!!!!
!!
!!!!!!
!!!! !!
!!
! !!!!!!!!!!!!
!!
!
!
!!
!!!!!!
!!!!
!
! !!!!!!!!!!
!
!
!!!
!!
!!!
!!!
!!!
!
!
!!
!
!
!!!!
!!!!!!
!
!
!!! !
!
!!!
!
!!!
!!!
!
!
!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!
!!!!!!!!!
!
!!!!!!!!!!!!!
!
!! !
!!!!! !!!!!!!
!!! !!!!
!!!!
! !!!!
!!!!!!! !
!
!!
!!
!
!!!!!!!!! !!
!!!!
!!!!
!
!!!!
!
!!
! !
!
!!!!!!!
!
!!
!
!!!! !! !
!!!
!!!!
!
!!!!!!!!!
!
!
!!
!!!!! !!!!!
!!
!
!
!!
!!!!!
!!!
!!
!!
!
!!
!!!!!
!!
!!
!
!!!!
!!
!
!
!
!
!!!!!!!!!!!
!
!!
!
!
!
!!!!!!! !
!!!!!
!!
!
!! !
!!!!!!!!!
!!
!!
!!
!
!
!!
!!
!!
!
!
!!
!
!
!
!
!
!
!
! !!
!!
!!
!
!
!!
!!
!!
!!!!!!
!!!!
!
!
!!!!!!!
!!! !!!!!!!!
!
!
!!
!
!
!!
!
!!
!!
!!!
!!!
! !
!
!
!
!
!!!
!
!!!
!
!!
!
!!!!!
!
!!
!!
!
!
!!!!!!!
!
! !!!!!!!
!!!! !! !!
!!!!!!
!!!!
!!!!! !
!!
!!!!
!!!!!!!
!
!!!!!!
!!!
!!!!!!! !!
!!
!
!
!! !!!!!
!!
!!! !!
!
!!
!
!
!!!
!!
!!
!!!!
!
!!!
!!!!
!!!!
! !!!
!!!!
!!!! !!!!
!!!
!
!!!!
!!!!
!!
!!
!!!!!
!!!
! !!!
!!!!
!!!!
!!!!
!!!!
!!!!
!!!!
!!! !!!
!!! !!
!
!!!!
!
!
!!!!!!
!!!
!!!!!!
!
!
!!!!
!!!
!!!
!!
!
!!
!
!!!!
!!
!!! !!! !
!
!
!!!!
!!
!!!!!!
!
!!!
!!
!!!
!!! !!!!!
! !
!
!
!!!!!!
!!!
!!!
!!!
!!!
!!!
!!!!
!!
!!!
!!!!!!
!
!!!!!
!!!
!!!
!!!
!!!
!!!!!
!
!!! !!!
!!
!
!
!!
!!! !!!
!!!
!!!
!!!!!!!!!
!!
!
!!!
!
!!
!
!
!
!
!
!
!
!
!
!
!!!
!!!!!!
!
!!
!
!!!
!! !!!!!!
!!!
!!! !!!!
!!
!!!
!!!!
!
!
!!!!!!!
!
!!!! !!
!!!!!!!!!!!!
!
!!!!!!!!!
! !! !!!!!!!!!!!
! !!
!!!!!!!!!!
!!! !!!!!!!!!!!!
!!!!!!
!!!
!!! !!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!
!!!
!!!!!!!!
!!!!
!!
!
!!
!!!
!
!!!
!!
!!!!
!!!
!!!
!!!!
!!
!!! !!!
!!! !
!
!
!!!!
!!
!
!
!!!!
!!!
!!!!!!
!!
!!
!
!!!!
!!!
!!!
!!!
!
!
!!
!
!!!!! !!
!!!!!!!!!
!!!!! !!!!
!!!!
!
!!!!
!!
!
! !
!!!!!!!!!!!
!!!!!!!
!
!!!
!!!
!!
!
!!
!!!!
!!!!
!!!!!!!
!!!!!!!!!
!!!!
!!!!!!!!
!!!!
!!!!
!!!!!!!!!!!!
!!! !!
!!!!!
!! !!!!
!!
! !!
!
!!!!
!!
!
!!!!!
!! !!
!!!!
!!!!
!!
!
!
!
!
!
!
!
!
!
!!
! !
!!
!
!!
!!!!!!
!
!!!!
! !!!!!!!!!!!!
!
!!
!
!
!!
!
!!
!!
!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!!
!!!
!
!!!!!
!!!!
!!!!!
!!! !!
!!!!
!
!!!!!!
!
!!
! !
0 1 2 3 4 5 6
−3−1
12
3
predict(rf)
resi
d(rf)
!
!!!!
!!
! !
!
!
! !
!
!!
!
!
!
!!!!!
!
!
!
!
!
!!!
!
!
!
!
!
!!!
!!
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!!!
!
!
!
!
!
!
!!!
!
!!
!
!
!!!!!
!!
!
!
!!
!!!
!!
!!
!!!!!
!
!!
!
!!
!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!
!!!!!!!!!
!
!!!
!!!!
!
!!!!!!!!!
!
!!!!!!! !!!!!!!
!!
! !
!!!!!!!!!!!
!!!!!
!
!!!!!!
!!!!!!!!!!
!
!!
!!!
!!
!
!!!!!!!! !
!
!!!! !
!
!
!
!!
!!
!!!!!
!
!
!! !!
!! !
!
!
!!!!
!
!!
!!
!!
!
!
!
!!
!
! !!! !!!
!
!!! !!!!
!
!!!!!
!!
!! !!!!
!!
!
!
!!!!!!!!!
!
!!!!!
!
!
! !!!!!!!!
!!!!!
!!
!
!!!!!!
!
!!
!
!!
!
!!!
!
!
!!
!! !!
!
!! !
!!
!!!
!!
!!!!
!
!
! !
!
!
! !!!!
!
!
!
!!!
!
!
!!
!
!!
!!!!
!!
!!!
!
!!
!!
!
!!
!
!!!
!!
!!!
!
!!
!
!
!
!!
!
!
!!!
!!
!
!!
!
!
!
!! !!!!
!
!!!
!
!!!!!
!
!
!!!!!!
!
!!!!
!!!!!
!!
!!!!!
!!!!!
!
!!!!!
!
!!
!!
!!
!
!!!!!!!!!!
!
!
!
!!
!
!
!!!!!!!!!!!!!!!!!!!
!! !!
!
!!!!
! !!!
!!
!!
!!!!!!!!!!!!!!!
!
!! !!!!!!!!!!!
!
!!
!!
!!! !!!!!!!
!!!
!!!!
!!
!!!
!!!
!
!
!!!!!
!!!!!!!!!!
!!
!
!
!!!!!!!!
!
!!!!!!!!!
!!
!!!!
!!!
!!!
!!!
!!
!
!!
!
!
!
!
!
!
!! !!!
!
!!!!!! !
!
!!
!!
!!!! !
!
!
!!!!!!! !!
!
!
!! !
!!! !!
!
!!!
!
!!! !!!!!!
!
!
!
!!!!!!!!
!
!!!
!
!!!
!
!
!
!!!!!!!!
!
!!!
!!!!
!
!
!
!!!!
!!!!
!
!!!!!!!!!!!!
!!!!!!!!!!!!!!!
!
!!
!
!!!
!
!!!
!!!!!!!
!
!!
!
!!
! !
!!!!!!
!!!
!
!!!!!!
!
!
!
!
!
!!!!
!
!!!!!!!!!!!
!
!!
!
!!!
!
!
!!!
!
!!!
!!! !!!
!
!!!!!
!!!!!!!!!!!!
!!!!
!!!!!!!!
!!!!!!!
!
!
!!!!!!!!!!!!!!!
!!
!!
!
!!!!
! !!!
!
!!
!!!! !
!
!
!
!!
!!!
!!
!
!
!!!
!!!!
!!
!
!
!!!
!!!
!!!
!
!
!
!
!!!!!!
!!!!!!!!!
!!!!!!!!!
!
!!!!!!!
!
!!!
!!!!
!
!!!!!
!!!!!
!!!
!
!!
!!!
!
!
!
!!!
!!
!!
!!
!
!!!!!
!
!
!!!
!
!!
!!!!
!
!!!!
!
!
!!
!!!!
!
!!!
!
!
!!
!!!!!!!!!!!!!!
!!
!!
!
!
!
!!
!!
!!!!
!!!!
!
!!!
!!!!!!!!!!!!!!!!
!!
!!!!!!
!!!
!!!!!
!
!
!
!
!
!
!
!
!!!
!
!!!!!
!
!!
!
!!!!!
!
!!!!!!!!! !!
!
!!!!!
!
!!!!!!!!!!!
!
!
!!!
!
!!
!
!!!!!!
! !
!!
!
!
!!!
!!!
!
!!!!!!
!
!
!!
!!
!!!!!!!!
!
!!
! !
!!!!!
!!!
!!
!!!!
!!
!
!!!!!
!
!!
!!!!
!
!!
!!!
!
!!!
!!!!!!!!!!
!
!
!
!!!!
!!!!!!!!!
!
!!
!!!!!!!!!
!!!
!
!
! !!!
!!!!!
!!!!!!!!
!!
!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!
!!!!!!!!!
!!!! !! !!!!!!!!!!!
!!
!!!!!!!
!!!!
!!
!!!!!!
!
!!! !!!!
!!
!!!!!!
!!!
!!!
!!
!!!!
!!!
!!!!!!!!!
!!!
!!
! !!!!!!!!!!
!!!!
!
!
!!
!!!!!!
!!!!
!
! !!!! !!!!!!
!
!
!!!
!!
!!!!
!!
!!!
!
!
!!
!
!
!!!!
!!!!!
!
!
!
!!! !!
!!!
!
!!!!!!
!
!
!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!
!!!!!!!! !
!
!!!!!!!!!!
!!!
!
!!!
!!!!! !!!!!!
!
!!!
!!!!!!!!
! !!!!!
!!!
!!! !
!
!!!!
!!!!!!!!!! !
!!!
!!
!!!!
!
!!!!!
!!
!!
!
!!!!!!!
!
!!
!
!!!! !
! !!!
!
!!!!
!
!!!!!!!!!
!
!
!!
!!!!! !!!
!
!
!!
!
!!
!!
!!!
!!
!!!!
!!
!
!!
!!!!!! !!
!
!
!
!!!
!
!!!
!
!!!
!!!!!!!!!
!!
!
!
!
!
!!!
!!!!
!
!!!!
!!!
!!
!!!!!!!
!!!!
!!
!!
!!
!!
!!
!!!
!
!
!
!!
!
!
!
!
!!
!
!!
!
!!
!!
!
!
!!
!!
!!
!!!!!!
!!
!!
!
!
!!!!!!!!! ! !!!!!!!!
!
!!!
!
!
!
!
!
!
!!
!
!!!
!!!
!!
!!
!
!
!!!
!
!!!
!
!!
!
!!!!
!!
!!
!!
!
!
!!!!!!!!
!!!!!
!!!!!!! !! !!
!!!!!!
!!!
!!!
!!!!
!!
!!!!
!!!!!!!
!!!!
!!!
!!!
!!!!!!! !!
!!!
!
!!!!
!!!
!
!!!! !!
!
!!
!!!!
!
!!!!! !!!
!
!!!
!!!!
!!!!
! !!!
!!!!
!!!! !!!!
!!!
!
!!!!!! !!
!!!!
!!!!!
!!!
!!!!
!!!!
!!!!
!!!!
!!!!!!!!
!!!!
!!!!!!
!! ! !!
!
!!!!!
!
!!!
!!!!!!
!!!!!!
!
!
!!!!
!!!
!!!
!!
!
!!!
!!!!
!!
!!! !!! !
!
!
!!!!!!
!!!!!!
!!!
!!!
!!!!!
! !!!!!! !
!
!
!!!!!!
!!!
!!!
!!!
!!!
!!!
!!!!!!
!!!
!!!
!!!
!
!!!!!
!!!
!!!
!!!
!!!
!!!!!
!!!! !!!
!!
!
!
!!
!!! !!!
!!!!!!
!!!!!!!!!
!!
!
!!!
!
!!
!
!
!
!
!
!
!
!
!
!
!!!
!!!!!!
!
!!
!
!!!
!! !!!!!!
!!!
!!! !!!!
!!
!!!
!!!!
!
!
!!!!!!!!
!!!! !!! !!!!!!!!!!!
!!!!!!!!!!
!!! !!!!!!!!!!!! !!!!!!!!!
!!!!!! !!!!!!!!!!!! !!!!!!
!!!
!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!
!!!
!!!!!!!
!
!!!!
!!!
!!
!!!
!
!!!
!!!
!!!!!!
!!!
!!!
!!!
!!! !!!
!
!
! !
!
!
!!!
!!
!
!!!
!!!
!!!
!!
!!!!
! !!!
!
!
!!
!
!!
!!!!!!!
!
!
!!
!
! !!!! !!!!
!!!!!!!!
!
!!! !!!!!!!!
!
!!
!!!!
!! !
!!!!!!!!!!
!!!!!!!!
!
!!!
!!!!!
!
!!
!!!!
!!!!
!!!!
!!!!!
!!!!!!!
!!!!
!!!!!!!!
!
!!!!
!!!
!!!!!!!!
!!!!!
!! !!
!!!!
!!!
!!!!!!!!!
!
!! !!
! !
!!!!!
!!!!
!
!!!!
!!!!
!!
!
!
!
!
!
!
!
!
!
!!
!!
!!!
!!
!!!!!!
!
!!!!! !!!!!!
!!!!!!!
!!
!
!
!
!!
!!
!! !!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!!!!!
!
!!!!!
!!!!
!!!!!
!!! !!
!!!!!!
!
!!!
!!
!!
!!
0 1 2 3 4 5 6
−3−1
12
predict(rf)
resi
d(rf)
!
!!!!!!
!!
!
!
!!!!!!
!
!!!!!!!!!!!!!!!!!!
!
!!! !!!!!!
!
!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!!!!!!
!
!
!!!!!!!!
!! !!!!!!
!
!
!!!!!!!!
!!
!!!!!!
!
!
!!!!!!!!!
!!!!!!!!!! !! !!!!!
!!
!!!!!!!!!
!!!!!!! !!!!!!!!!
!! !!!!!!!!!!
!!!!!
!!
!!!!!!
!!!!! !!!!!
!!!
!! !
!!
!
!!!!!!!!
!!
!!!! ! !!
!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!
!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!! !!
!
!!!! !
!!
!
!
!
!!
!
!
!!
!
!
!!
!!!!
!
!! !
!!
!!!
!
!
!!!
!
!!
!!
!
!
! !!!!
!
!!!!!!!!!
!
!!!!!!
!!
!!!!!!!!
!
!!!!!!
!!
!!!!!!!!
!
!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!
!
!!!
!
!!!! !!!!!!!!!!!
!!!! !
!!!!!!!!!!!!!!!!!!
!!!
!!
!
!!!
!
!!
!!!
!
!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!
!!
!!!!!!!!!!!!!!!!!!
!
!!
!
!!
!! !!!!
!!!!!! !
!
!!!!!
!!! !
!
!!
!!!!!! !! !
!
!! !!
!!!!
!
!! !
!
!!! !!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!
!!!!!!!!
!!
!!!!!!!!!
!!!!!!!!
!!!!!!
!!!!!!!!!! !!!!!
!!!!!!
!!!!
!!!!!!!
!
!! !!!
! !!! !!!!!!!!!!!!!!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
!!
!!!
! !!!!
!!!!!! !
!
!!!
! !!!
!!
!
!!
!!!!!!
!! !!
!! !!!
!!!!
!!
!!
!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!
!!!!
!!!
!
!!!!!!! !!
!!!!
!!! !!!
!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!
!!!
!
!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!
!!
!
!
!!!
!
!!!!!
!
!!!!!
!!! !!!!!!!!!!!!!!!!!!!
!!!!!!!! !!!
!!!
!!
!
!!
!
!!!!!!
! !
!!
!
!
!!
!!!
!
!
!!!!
!!
!!!!!!!!!!!!!!!!
!!
!!!!!!!!!!!!!!!!!
!!!!!!
!
!!
! !!
!
!
!!
!! !
!
!!!!!!!!!!!!!
!
!
!
!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!
!!
!!
!!!
!
!!!
!
!
!!!!!
!
!
!
!!!
!!! !
!!!
!!
!!
!
!
!
!!
!!!!!
!
!!!!!!!
!!!!!!!
!!!
!!
!
!!
!
!
!!!!
!!
!!!!!
!
!!!
! !
!
!!
!!
!!!!!!!!!
! !
!!!
!
!!
!!!!!
!!!!
!
!
! !!!!!!!!!!
!
!
!!!!
!!!!!!! !!!
!
!!!
!!!!!!!!!!!!
!
!!!! !
!
!!!
!
!!
!
!!!
!
!
!!!!!!!
!
!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!
!!!!!!!!
!
!!!!!
!!!!!
!
!!
!
!!!
!
!!!
! !!!!!!
!
!
!! !!! !
!!!!
! !!!!
!!
!!!!!
!
!
!!!!
!
!!!!!!
!!! !!!!
!!
!!!!
!
!!!!!
!
!! !
!
!
!!!!!!
!
!!
!
!!! ! !!
!!!!
!!!!
!
!!!!!! !!
!
!
!
!
!!!!!! !!!
!
!
!
!
!
!
!
!
!!!
!
! !!!
!!
!! !
!!
!!!
!!
!!!
!
!
!!!!
!!
!
!
!
!!!!! !
!!!! !!!!!
!
! !!!
!!! !! !!!!!
!!!
!
!!!!!!!!! !!! !!
!!
!!!!!!!!!!
!
!
!!
!!
!
!
!!!!!!!!!!
!
!
!!
!!!!!!!!!!!!!!!!
!!!!!!!!
! ! !!!!!!!! !!!!!!
!!
!
!
!!
!
!!!
!!!!
!!!!!
!!!
!
!!! !!!!!!!!
!!!!!!!!!!!!
!!!!!!!!!
!!!!!!! !!!!!!!!!!
!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!! !!!!
!
!
!! !!!!!!!!!! !!!!!
!
!!!!
!!!!! !!! !!!!!!!!!!!!
!!
!!
!!!!
! !!!
!! !!
!!!!!!!!
!! !!
!!
!
!
!!!!!!!!
!!!!
!!!!!!!!!!!!
!!!!!
!!!
!!!!!!!!!!
!!!
!!!!!!
!!
!
!!!!!! !!!!!!!
!!
!
!
!!!!
!!!
!!!
!
!
!
!!!!!!
!!!
!!
! !!! !!!!!!!
!!
!!!!!!!!!!
!! !!!!!!
!!!!
!!
!!!!!!!!!
!!! !!!!!!
!!!
!!!
!!!!
!!
!!!!!!!!!
!
!!
!!!!!!!!!!
!! !!!!!!!!
!!!!!!!!!!
!
!!
!!!!!!!!!
!!
! !!!!!!!!!
!!
!
!!!
!
!!!
!
!
!
!
!
!
!
!
!
!!!
!
!!
!!
!
!
!!
!
!!!
!!!
!!!!! !!!
!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!! !!!!!!!!!
!!! !!!!!!!!!!!! !!!!!!!!!!
!!!!! !!!!!!!!!!!! !!!!!!!!!!!! !!!!!!!!!!!! !!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!
!!!!!!!!
!
!!!!!!
!!
! !
!
!
!!
!!
!
!
!!
!
!!!!
!!
!!!
!!!
!!
!
!
!
!
!!!!!!
!!!!!!
!
! !
!
!
!
!!!! !!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!
!
!!
!!!!!!!!!!!!!!!!!!
!!!!!!!!!
!
!!
!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!! !! !
!!!!
!!!!!!!!!
!!!
!!!!!!!!
!! !
!
!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!! !
!!!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!!
!!
!!
! !!!! !!!!!!!!!!!!!!!!!!! !!!! !!!
!!!!
0.5 0.6 0.7 0.8 0.9 1.0
−0.1
0.1
0.3
invlogit(predict(rf))
resi
d(rf,
inv
= TR
UE)
!
!!!!!!
!!
!
!
!!!!!!
!
!!!!!!!!!!!!!!!!!!
!
!!! !!!!!!
!
!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!!!!!!
!
!
!!!!!!!!
!! !!!!!!
!
!
!!!!!!!!
!!
!!!!!!
!
!
!!!!!!!!!
!!!!!!!!! ! !! !!!!!!
!!!!!!!!!
!
!!!!!!! !!!!!!!!!
!! !!!!!!!!!!
!!!!!
! !!
!!! !
!!!!!
!!!!!!
!!!!
! !
!!
!
!!!
!!!!! !
!
!!! ! ! !
!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!! !!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!
!!
!
!!!!!
!!
!!
!
!!
!
!!!
!
!
!!
! ! !!
!
!! !
!!!
!!
!
!
!!!!
! !
!!
!
!
! !!!!
!
!!!!!!!!!
!
!!! !!!
!!
!!!!!!!!
!
!!!!!!
!!
!!!!!!!!
!
!! !!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!
!
!!
!!
!!!!!!!!!!!!!!!!!!! !
!!!!!!!!!!!!!!!!!!! !!!!
!
!!!
!
!!
!!!!
!!
!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!
!!
!
!!
!! !!!!
!!!!!! !
!
!!!!!
!!! !
!
!!!!!
!!! !! !
!
!! !!!! !!
!
!! !
!
!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!
!!
!!!!!!!!!
!!!!!!!!!!
!!!!
!!!!!!!!!!!!!!
!!!
!!!!!!!!!
!! !!!!
!
!! !!!! !!! !!!!!!!!!!!!!!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! !!
!!!
! !!!!
!!!!!! !
!
!!!
! !!!!
!
!
!!
!!!!!!
!! !!
!! !!!
!!!!
!! !
!
!!!!!!!!!!!! !!!!!!!!!!!!
!
!!!!!!!!
!!!!!!!
!
!!!!!!!!!
!!!
!
!
!! !!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!
!!
!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!
!!
!
!
!!!!
!!!!!
!
!!!!!
!!! !!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!
!!
!!!
!
!!
!
!!!!!!
!!
! !
!
!
!!!
!!!
!!!!
!!!
!!!!!!!!!!!!!!!!
!!
!!!!!!!!!!!!!!!!!
!!!!
!!
!
!
!
!!!
!
!
!!
!
!! !
!!!!!!!!!!!!!
!
!
!
!!!!!!!!!! !!!
!
!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!!
!!! !
!!
!!
!!!
!!
!!!! !
!!
!
!!!
!! !!
!
!!
!!
!!
!
!
!
!!
!! !!!
!
!!! ! !!!
!! !!
!!!!
! !!
!
!
!!!
!
!!!!!
!
!!!!!
!!!!
!!
!
!!
! !! !!!!!!!!
! !
!!!
!
!!
!!!!!!
!!!!
!
! !!!!!!!!!!
!
!
!!!!!!!!!!!!!!
!
!!!
!!!!!!!!!!!!
!
!!!! !
!
! !!
!
!!!
! !!
!
!
!!!!!!!
!
!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!
!!!!!!!
!
!
!!! !!!!
!!!
!
!!
!
!! !
!
!!!! !!! !!!
!
!!! !!!! !!!
!
! !!!!
!
!
!!!!! !
!
!!
! !
!
!! !!!!
!!!!!
!!!!
!!!!
!
!!!!
!
!!
! !
!
!
!!
!!
!!
!
!!
!
!!!! !
!!!
!!
!!!!
!
!!!!!!!
!!
!
!
!
!!!!!! !!!
!
!
!
!
!
!
!
!
!!!!! !!!
!!
!! !
!!
!!!
!!
!!!
!
!
!!!!
!!
!
!!
!!!!!!!!!!!!!
!!
!
!!!!! !!!
! !!!! !!!!
!
!!!!!!!!!!!! !!
!!
!!!!!!!!!!
!
!
!!
! !
!
!
!!!!!!!!!!
!
!
!!
!!!!!!!!!!!!!!!!
!!!!!!!!
!! !!!!!!!! !
!!!!!
!!
!
!
!!
!!!!
!!!
! !!!!!
!!!
!
!!! !!!!!!!!
!!!!!!!!!!!!
!!!!! !!!!!!! !!!! !!!!
!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!! !!!!
!
!
!!!!!!!!!!!! !!!
!!
!
!
!!!
!!!!! !!! !!!!
!!!!!!!!
! !!!
!!!!
!!!!
!!!!
!!!!!!!!!!!!
!!
!!
!!!!!!!!
!!!!
!!!!
!!!!
!!!!!!!!!
! !!!!!!!!!!!!
!!!
!!!!!!!
!
!
!!!!!! !!!!!!!!!!
!
!!!!!!!
!!!
!!
!
!!!
!!!!!!
!!! !!! !!!!!!!
!!
!!!!!!!!!!
!! !!!!!!
!!!!!
!!!!
!!!!!!!
!! !!!!!!!!!
!!!
!!!!
!!
!!!!!!!!!
!
!!!!!!!!!
!!!
!! !!!!!!!!
!
!!!!!!!!!
!
!!
!!!!!!!!!
!!! !!!!!!!!!
!!
!
!!!
!
!!!
!
!
!
!
!
!
!
!
!
!!!
!
!!
!!!
!
!!
!
!!
!
!!!
!!!!! !!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!! !!!!!!!!!!!
!!!!!!!!!!
! !! !!!!!!!!!!! ! !!!!!!!!!
!!! !!! !!!!!!!!!!!! !!!!!!!!!!!! !!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!
!!!!!!!!!
!!! !!!
!!
! !
!
!
!!!
!
!
!
!
!
!!!!!
!!
!!!!
!
!
! !!
!
!
!
!!!!!! !!!!!!
!
!
!!
!
!
!!!! !!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!
!
!!
!!!!!!!!!!!!!!!!!!
!!!!!!!!!
!
!! !!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
! !!
!!!!!!! !!
!!!!!!! !!! !
!
!
!
!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!
!!!!
!!!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!!!
!!!
!
!!!!! !!!!!!!!!
!!!!!!!!!!
!!!!!!!
!!!!
0.5 0.6 0.7 0.8 0.9 1.0
−0.2
0.0
0.2
0.4
invlogit(predict(rf))
resi
d(rf,
inv
= TR
UE)
!
!!!!!!
!!
!
!
!!!!!!
!
!!!!!!!!!!!
!!!!!!!
!
!!! !!!!!!
!
!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!!!!!!
!
!
!!!!!!!!
!! !!!!!
!!
!
!!!!!!!!
!!!!!!!
!!
!
!!!!!!!!
!!!!!!!!!!
!!!!!!!!
!!
!!!!!!!!
!
!!!!!!!
!!! !!!!!!
! !!!!!!!!!! !!
!!!!! !
!!!
!! !!
!!!!!!!!!
!!!
!!!
!!
!
!!!!!!!! !
!
!!!! ! !
!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!
!!!!!!!!!
!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!
!!
!!
!!
!!
!!
!
!
!
!!!
!
!
!!
!! !!
!
!
! !
!!
!!!
!
!
!!!!
! !
! !
!
!
! !!!!
!
!!!!!!!!!
!
!!! !!!
!!
!!!!! !!!
!
!!!!!!
!!
!!!!!!!!
!
!! !!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!
!
!!!!
!!!!!!!!!!!!!!
!
! !!!!
!!!!!!!!!!!!!!!!!!!
!!
!!
!
!!!!
!!
!!!
!!!
!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!
!!
!!!!!!!!!!!!!!!!!!!
!
!
!
!!
!! !!!
!!!!!!! !
!
!!
!!!
!!! !
!
!!!!
!!!! !! !
!!! !
!!! !!
!
!! !
!
!!!!!!!!!!!
!!!!!!!!!
!!!!!!!!!!
!!!!!!!!!
!!!!!!!!
!!
!!!!!!!!!
! !!!!!!!!!!!!
!!!!!!!!!!!!!!!
!
!! !!!! !!!!!
!!!!!!
!
!! !!!! !
!! !!!!!!!!!!!!!!!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!
!!!
!! !!!
!!!
!!!! !
!
!!!!!
!!!
!
!
!!
!!!!!!
!! !
!!! !
!!!
!!!
!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!
!!!!
!!!
!
!!!!!!!!! !!!
!
!!! !!!
!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!
!
!
!!!!
!!!!!
!
!!!!!
!!!!
!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!
!!
!!!
!
!!
!
!!!!!!
! !
! !
!
!
!!!
!!!
!
!!!!
!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!
!!!!!!
!
!!
!!!!
!
!! !!
!!
!!!!!!!!!!!!!
!
!
!
!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!
!!
!!!!!
!
!!! !!
!! !!!
!
!
!
!
!!!
!
!!!
!!!
!
!
!!
!
!
!!
!!!!!
!
!!! !!!!
!!
!!!!!
!!!!
!
!
!
!!
!
! !!
!!
!
!! !!!
!
!!!! !
!
!!
! !!!!!!!
!!!
! !
!!!
!
!!!!!!!!
!!!!
!
! !!!!!!!!!!
!
!
!!!!
!!!!!!!!!!
!
!!!
!!!!!!!!!!!!
!
!!!! !
!
!!!
!
! !!!!
!
!
!
!!!!!!!
!
!!!!!
!!!!
!
!!!!!!!!!!!!!!!!!!
!!!!!!! !
!
!!!!!!!!!!
!
!!
!
!!!
!
!!!
! !!! !!!
!
!
!!!!!
!!!!
!! !!
! !
!
!
!!!
!!!
!
!
!
!!
!
!!!!!!
!!!!!!!
!!
!!!!
!
!! !!
!
!
!! !
!
!
!!
!!!!
!
!!
!
!!
!! !
!!!
!!
!!!!
!
!!!!!!!
!!
!
!
!
!!!!!! !!!
!
!
!
!
!
!
!
!
!
!!!
!!
!!
!!
!! !
!!
!!!!! !
!!
!
!
!
!!!!!
!
!
!
!!! !!!!!!!!!!
!!
!
!!!!!
!!!! !!!!
!!!!
!
!!!!!!!!
!!!! !!
!!!!!!!!!!!!
!
!
!!
!!
!
!
!!!!!!!!!!
!
!
!!
!!!!!!!!!!!!!!!!
!!!!!!!!
! ! !!!!!!!! !
!!!!!
!!
!
!
!!
!!!!
!!! !
!!!!!
!!!
!
!!! !!!!!!!!!!!!!!!!
!!!!!!!!! !!!!!!!!!!! !!!!
!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!! !!!!!
!
!! !!!!!!!!!! !!!!!
!
!!!!
!!!!! !!! !!!!!!!!!!!!
! !!!
!!!!
!!!
!!!!!
!!!!!!!!!! !!
!!!!
!!!!!!!!
!
!!!
!!!!
!!!!
!!!!!!!!!
!!!
!!!!!!!!!!
!! !!!!!!!!
!
!
!!!!!! !!!!!!!!!
!
!
!
!!!!!!
!!!
!!
!
!!!!!!
!!!
!!! !!! !!!!!!!!
!
!!!!!!!!!!
!! !!!!!!
!!!!!!
!!!
!!!!!!!!! !!!!!!
!!!
!!!
!!!!!
!
!!!!!!!!!
!
!!!!!!!!!!!!
!! !!!!!!!!
!
!!!!!!!!!
!
!!
!!!!!!!!!
!!
! !!!!!!!!!
!!
!
!!!
!
!!!
!
!
!
!
!
!
!
!
!
!!!
!
!!!!!
!
!!
!
!!
!
!! !
!!!!! !!!
!!!!!!!! !!!!!!!!!!!!!!!!!
!!!!! !!!!!!!!!!!!!!
!!!!!!!!!!
!!! !!!!!!!!!!!! !!!!!!!!!
!!!!!! !!!!!!!!!!!! !!!!!!!!!!!! !!!!!!!!!!!! !!!!!!!!!!!!!!!!!!
!!!
!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!
!!!!!!!!
!!!!!!!
!
!
! !
!
!!!
!!
!
!
!!
!
!!!!!!
!!!
!
!!
!!!
!
!
!
!!!
!!
! !!!!!!
!
!
!!
!
!
!!!! !!!!!!!!!!!!!!!! !!!!!!!!
!!!!!!!
!
! !
!!!!!!!!!!!!!!!!!!
!!!!!!!!!
!
!!
!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!
!!!!
!!!!
!!! !!!!!!
!!!
!!!!!!!!
!! !
!
!
!
!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!! !!!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!!
!!!
!
!!!!! !!!!!!!!!
!!!!!!!!!!!!
!!!!!
!!!!
0.5 0.6 0.7 0.8 0.9 1.0
−0.1
0.1
0.3
invlogit(predict(rf))
resi
d(rf,
inv
= TR
UE)
!!!!!!!
!!!!
!!!
!!!!
!!!!!!!!!!
!!
!!!!!!
!!!! !!!!!!
!
!!!
!!!
!!!!!!!
!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!
!!
!!!!!!
!!
!!!!!!!!
!!!!!!!
!!
!!!!!!!!
!!
!!!!!!
!!
!!!!!!!!
!!!!!!!!!!! ! !! !!!!!!!!!!!!!!!
!!!!!!!
! !!!!!!!!!!
! !!!!!!!!!!!!!!!!! !!!!!!
!!!!! !!!!!
!! !!
! !
!!
!!!!!!!!
!!
!!!
!! ! !!!!!!!!!!
!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!
!!!!!!!!!!
!
!!!!!!!!!!!!!! !!
!
! !!! ! ! !!!!
! !!
!!!
!
!!!
!!!!
!!! !
!!!!!
! !!!!!
! !
! !!
!! !!!!
!
!
!!!!
!!
!!!
!!!!!!
!!
!!!!!!!!
!!!
!!!!
!!!!!!!!!
!!!! !!!!
!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!! !!!!!!
! !
!! !!!!!!!!!!!
! !!! !!!!!!!!!!!!!!!!!!!!
!!!!!!! ! !
! !!! !! !! !!!!!!!!!!!!!!!
!!! !!!!!!!!!
!!!!!
!!!!!!!!!!!!!!!!!!!
!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!
!!
!!!!!!!!!!!!!!!!
!!
!
!!!
!!!! !!!
!!!!!!! !
!
!!
!!!
!!! !!
!!
!!!!!! !!
!
!!! !
!!!!!
!
!! !
!!!
! !!!!!!!!
!!!!!!!!!
!!!!
!!!!
!!
!!!!!!!!!
!!!!!!!!
!!
!!!!!!!!!
!!!
!!!!!!!!!!
!!!!!!!!!!! !!!!! !!
!!!!!
!!!
!
!!!!!!!!!
!!!
! !!! !!!!!!!!
!!!!!!!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!
!!!!
!!
!!!! !!!
!!!!!!! !!
!!
!! !!!
!!
!
!!
!!!!!!
!! !!
!! !!!
!!!!
!
!!!
!!!!!!!!!!!!!!!!!!!!!!!
!!
!!!!
!!!
!!!!
!!!!
!
!!!!
!
!! !!!
!! !
!!!
!!!
!
!!!!!!!!!!!
!
!!!!!
!!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!!!!!
! !!!
!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!!
!!!!!!!
!
!!!!!!
!!
!!!
!!!!
!!!!!!!!!!!
!!!!!!!
!!!!!!!! !! !!
!!!!!
!!!
!!!!!!! !
! !!
!!!!
!!!
!
!!!!!! ! !
!!!!!!!!!!!!!
!!! !
!!!!!!!!!!!!!!!!
!!!!!!!! !! !! !! !! !! !!!
!!!!!!!!!!!!
!!
!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! !! !
!!! !!! !! !!! !! !
!!!!!!
!!
!! !! ! !!!! !
!!!
!!
!! !!!!!!
!
!!! !!!!!!!!!!!
!!!!!!!
!!
!!!!!
!!!!!! !!!!!
! !!
! !! ! !!!!
!!!!!! !
!!!!
!!!!!!!!
!!!!
!! !!!!!!!!!!
!
!!!! !
!!
!!!!! !!!
!
!!!
!!
!!!!!!!!!!
!
!!!
! !!
! !!
!! ! !!!!!
!
!!!!!!!!
!!!! !!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!! !
!!!!!!!!!!!
!!!
!!! !
!!!! ! !!!!!!
!
!! ! !!! ! !!! !! !!! !!
!!! !!! !!
!!!!
!!!!!!!
!!! !!!!!!!!!!
!!!!!
!
!!
! !
!!
!!!!!!
!!!
!!!
! ! !! !!
!!
!!!!
!!!!!!
! !! !
!
!!
!!!!!! !!!!!! !
!!
! !! !!! ! !!!!!!!
!!!
! !!! !! !!!
!!
!!!!!!
!
!
!!!!! !!!!! !!
!!!!
! !!!
! !! !! !!!!!
!!!
!! !!!!!!!! !!!
!!
!!
!!!!!!!!!!
!
!
!!
! !
!!
!!!!!!!!!!
!
!
!!
!!
!!
!!!!!!!!!!!
!
!!!!!!!!
! ! !!!!!!!! !!!!!
!
!!
!!
! ! !!!!
!!!! !!!
!
!!!!
!!!! !
!!!!!!!
!!!!!!!!
!!!!!!!!!!!!!
!!!!!!! !!!!
!!!!!!!!!!!!
!!! !!!!!!!!!!!!!!!!!!!!!
!!!
!!!!!!! !!
!!!
!!! !!!!!
!!!
!! !!!
!!
!!!!!!!!!! !!! !!!!
!!!!
!!!!
! !!!!!!!
! !!! !! !!!!!
!
!!!!!! !!
!!! ! !!!!!!!!! !!!
!!!!!!!!!!!!
!!!!! !!!!!!!!!!!!!
!! ! !!!!!!!!
!!!!!!!!!!!!!
!
!
!!
!!
!!!
!!!
!!!
!!!
!!!!!!! !!
!!!
!!! !!
!
!!!!!!
!!!!!!
!!!!
!!!!!
!!!!!!!!
! !!!
!!!!!!!!!
!!!!!!
!!!
!!!
!!!!!!!!!
!!!!!!
!!!
!!!!!!
!!!!!! !!!
!!!!!! !!!!!!!!!
!
! !
!!!!!!
!!!!
!!
!!!!!!!!!
!!
!
!!!
!
!! !
!
!!
!!
!
!
!
!
!!!!
!!!!
!
!
!!
!!!
!
!!!
!!!!! !!!
!!!!!!!
!!
!!!
!!!!!!!!!!
!!!
!!!!!!!!!!!!!!!!!!!
!
!!!!
!
!!!
!
!!!!!!!
!
!!!!!!! !!!!!!!!!!!!
!!! !!!!!!!!!!!!
!!!!!!
!!!!!!
!!!!!!!!!
!!!
!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
!
!!!!
!!!
!!!!!!!!!
!!!!!!
!!!
!!!
!!!!!!!!!!!!!!! !
!
! ! !!!!
!!!
! !!!
!!!! !!!!!
! ! !!!! !!!
!!! !!!!
!!
!! !
!
!! !!!! !!!!!!
!!!!!!!!!! !!!!!!!!
!!!!!!!
!! !!!!!!!!!!!!!!!!!!!!!!!
!!!!!
!!!
!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!
!!!!!!! !!!!!!! !! !
!!!!
!!!!!!!!
!!!!
!!!!!!!!!
! !
!
!
!
!
!!
!
!!!!!!!!!!
!!!!!!!
!!!!!!!!!!!
!!!!!!!
!!!
!!!!!
!! !!!!!!
!!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!!!!!!!
!! ! !! !!!! !!!!!!!!!
!!!!!!!!!! !!!! !!!
!!!!
0.5 0.6 0.7 0.8 0.9 1.0
−20
−10
0
invlogit(predict(rf))
resi
d(rf)
/resi
d.se
(rf)
!
!!!!
!!
!!
!
!
!!!
!!
!
!
!
!!!
!!
!
!
!
!
!
!
!!
!
!
!
!
!
!!!
!!
!
!!
!
!
!!
!
!
!
!
!
!!
!
!!!
!
!
!
!
!
!!!!
!
!
!
!!
!!!!!
!
!
!
!!!!!!!!!!
!!!!!!!!
!
!!!!!!!
!
!
!
!!!!
!
!
!
!
!!!!!!!
!
!!
!!!!!
!
!
!
!!
!!!!!
!
!
!
!!!!!
!
!
!
!!
!!!!!
!
!!!!!!!!!
!!
!!
!!!!!
!!
!!!
!!
!!!!
!!!!!!
!!
!!!!!!
!!
!
!!
!!!!!
!!!!
!
!!!!
! !!
!!
! !
!
!!!!
!!!!!!
!!!
!! !
!!
!
!!
!!!!!
!
!
!
!!! !
!
!
!
!
!!
!
!
!!!
!!
!
!
!!!
!
!
!!!
!
!!!
!
!
!!
!!
!!
!
!
!
!!
!
!
!
!!
!!
!
!
!!!!!!!
!
!!!!!
!
!!!!!!!!
!
!
!
!
!!!!!!!!
!!!
!
!!
!
!
!!!!!!!
!
!!
!
!!! !!
!
!!!!!
!!
!!
!
!!
!
!!!
!
!
!!
! ! !
!!
!! !
!!!!!
! !
!!!!
!
!
! !
!
!
!
!
!!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!
!!
!!
!
!
!
!
!!!
!!
!
!!
!
!!
!
!
!
!!
!
!
!!
!
!!
!
!
!
!
!
!
!!
!!!!
!
!!!
!!!!!!!!
!!!!!
!
!!
!!
!
!
!
!!!
!!
!!!!!
!
!!!!
!
!!!!!
!
!
!!
!
!
!
!
!
!!!!!!!!!
!
!!!
!
!
!
!!!!!!
!
!!!!!!
!
!!!!!
!!!!
!
!!!
!
!
!
!!!
!
!!
!!!!!!!!!!!!!!!
!
!!
!!!!!!!!!
!!
!
!!
!!
!!!!!!!!!
!
!!
!
!!!!
!!
!!!!!
!
!
!
!
!!!!!
!
!!!
!!!!!
!
!
!
!
!!!!
!
!!
!!
!!!!!!
!
!
!
!!
!
!
!!!!!
!
!!!!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!!
!
!!!
!
!
!!
!
!
!
!!
!
!
!
!
!
!!
!
!!!!
!
!
!
!! !
!
!
!!
!
!
!!
!
!
!!
!
!!!!!
!
!
!
!
!!!!!!!!
!
!!!
!
!!!
!
!
!
!!!!!!!!
!
!!!!!!!
!
!
!
!!!!!!!!
!
!
!!
!
!
!!!!
!!!
!
!!!!
!!!!
!!
!!!!
!
!!
!
!!
!
!
!!
!
!
!!
!!!
!
!
!!
!
!!
!!
!!
!!!!
!!!
!
!!!!!!
!
!
!
!
!
!
!
!!
!
!
!!!!!!!!!!
!
!!
!
!!
!
!
!
!!!
!
!!
!
!!!!!
!
!
!!!!
!
!!
!!!!!
!!!
!!
!
!
!!
!!
!
!
!!!
!
!!!!!!!
!
!
!!!!!!!!!!!!!!!
!!
!!
!
!
!!
!
!!
!!
!
!!
!!!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!!
!!
!
!
!!
!
!!
!
!!
!
!
!
!
!
!!
!
!
!
!
!
!!!!
!
!!
!
!!!!!!!
!!
!
!!!!
!!!
!
!!
!!
!!!
!
!!
!!
!
!
!!!
!
!!
!
!
!!
!!!
!
!
!!!!!
!!!!!
!
!!!!!
!
!!!!!!!
!!!!!!!
!!!
!
!!!!!!!!
!!!
!
!!
!!!!!!!!!
!!!!!
!!
!!!
!
!
!!
!
!
!!!
!
!!!!!
!
!!
!!!!!!!!!!
!!!!!!
!
!!!!!!!!!
!
!!!!!
!
!
!
!
!
!
!
!
!!!
!
!!!!!
!
!
!
!
!
!
!!!
!
!!!!!
!!!!!!
!
!
!!!!
!
!!!!!!!!!
!
!
!
!
!
!!
!
!!
!
!
!!!!
!
! !
!!
!
!
!!
!
!!!
!
!!!
!!
!!
!
!!
!!
!
!!!
!
!!!!
!!
! !
!
!!
!!
!!!!!!!!!
!!
!
!!!
!!
!
!
!
!!!
!
!
!
!
!
!!
!
!
!!
!!!!!!!!!!
!
!
!
!
!!!!
!!!
!!
!!!
!
!!
!!!!!!!
!!!!
!!
!
!!!
!!!
!!!!!!!!!!!!
!!!
!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!
!
!
!
!!!
!!
!
!!
!
!
!!!!
!
!
!!!!
!
!
!
!
!!
!! !
!!
!
!!
!
!!
!
!
!
!
!
!!!
!!
!
!!!! !!!
!
!!!
!!!
!!
!
!
!
!
!!!!
!!!!!!
!!!!!!
!!!
!
!!
!
!
! !!!
!!
!!!!!
! !!!
!
!
!
!
!!!!
!
!
!!!
!
!
! !!!!!!!!!!
!
!
!!
!
!
!
!
!!
!!
!
!!
!
!
!
!!
!
!
!
!!
!
!!
!
!!!
!
!
!!
! !
!
! !!
!
!!!
!!!
!
!
!
!!!
!!!
!
!!!!!!!!!
!
!!!!!!!!!!!!!!!!!
!
!!!!!!!
!
!
!!! !!!!
!!!
!
!!
!
!!
!
!
!!!! !!!!!!
!
!
!! !!!! !!!!
! !!!!
!!
!!!!
! !
!
!
!
! !
!
!!!!!!
!!!
!
!
!!
!!
!!
!!
!
!!!
!
!
!
!
!!
!
!
!!
!!
!!
!
!!
!
!!!! !
!
!!
!!
!
!!!
!
!!!!!
!!
!!
!
!
!!
!!!!! !!!
!
!
! !
!
!
! !
!!!
!!
!!!!!
!
!
!
!!
!!!
!!
!
!!
!
!
!
!!!
!
!!
!
!
!
!!!
!!
!!!!!!!
!
!
!
!
!
!!!!
!!!
!
!!!!!
!!
!
!
!!!!!
!!!!!!
!!
!!
!!
!
!!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!!!
!!
!!
!
!
!!
!!
!!
!!
!
!!!
!!
!!
!
!
!!!
!
!!!
!
!! !!!!!!!!
!
!
!!
!
!
!
!
!
!
!!
!
!!!
!
!!
!!
!
!
!
!
!!!
!
!!!
!
!!
!
!!!!
!
!
!!
!
!
!
!
!!
!
!!!!!
!!!!
!
!!!
!
!!
! !!!
!
!!
!
!!!
!!!!
!!
!!! !
!
!
!!!!
!!!!!!!
!
!!!
!!
!
!
!
!
!!
!!
!
!!!!!
!!
!
!! !!
!!
!
!
!
!
!!
!!
!
!
!
!
!
!!!
!!!!
! !
!!
!
!!!
!!!!
!
!!!
! !!!
!!
!
!
!!!! !
!!!
!!!
!
!!!
!
!!!
!
!!
!!
!
!!
!
!
!!!
! !!!
!
!
!!
!
!!!
!!!!
!
!!!
!
!!!
!
!!
!
!!
!!!!
!!
!!!
!
!
!!
!
!
!
!
!
!!
!!
!
!
!
!!!
!
!
!
!
!
!
!!
!
!
!
!
!!!
!
!
!
!
!
!
!!!!!
!
!
!
!!!!
!!
!
!!!
!
!
!
!!!
!!!
!
!!
!
!
!
!!!
!!
! !!
!!!
! !
!
!
!!!
!!!
!
!!
!!!
!!!
!!
!
!!!
!!!!
!!
!!!
!!
!
!
!!
!
!!
!!!
!
!!
!!!
!
!
!
!!!
!
!
!!!
!
!!!!!!
!!
!
!
!
!
!!!!
!!
!!!
!
!
!
!!!
!!!
!
!!
!!
!
!!!
!
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!
!!
!!
!
!
!
!
!
!!
!
!
!
!
!
!!
!!
!!!
!
!!
!!!
!
!
!
!
!
!
!!!
!
!
!
!!!!
!!
!
!
!!!! !
!
!
!!!!!!!!!!!!
!
!!!!!!!!
! !!
!!!!!!!!!
!
!
! !!
!!!!!!!
!!!
!!
!!!!!!!!!!
!!
!
!!
!!!!
!
!!
!!
!
!!!!!!!!!
!!
!
!!!!!!
!!!
!!!!
!!!!!
!
!!
!!!!!!!
!
!
!!!
!!!
!
!
!!
!
!
!!!
!
!
!
!!!
!!!
!
!!
!!!
!
!!
!!! !
!!!!
! !
!
!
!!
!!
!
!
!
!
!
!!
!
!
!!
!!!!!!
!!
!!
!
!!!!
!!!
!!!!
!!
!
!
!
!
!
!!!
!!!!!!
!!
!!!!!
!
!
!!!!!!!!!!!
!
!!!!!!
!
! !
!!!
!!!
!!!!!!!!!
!!!
!!!!
!!!!
!
!
!!
!!!!
!
!
!!
!!!!!!!!!!!!!!!!
!!
!
!
!!!!!!!!!!!!
!
!!!
!!!!!!!!
!!!!
!
!!!!
!!!!
!!!!!!!!!
!!!
!
!!!!
! !
!
!!!!
!
!! !!
!!!!
!!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!!
!!!!
!
!!!
!
!!!!!!!!!!!!!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!!
!!!
!
!!
!!!
!!!!!!!!!
!!!!
!
!!!!!!!!!!!!
!!!!
0.5 0.6 0.7 0.8 0.9 1.0
−20
12
invlogit(predict(rf))
resi
d(rf)
/resi
d.se
(rf)
!
!
!!!
!
!
!!
!
!
!!
!
!!
!
!
!
!!!!!
!
!
!
!
!
!!!
!
!
!
!
!
!!
!
!!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!!
!
!
!
!!
!!!!!
!!
!
!!!
!!!!!!
!!!!!!
!
!!
!
!!!!!!!
!
!
!
!!!!
!
!
!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!
!!!
!!!
!!!
!
!!
!
!!!!
!!
!
!!
!!
!!!
!
!!!!
!
!
!
!!
!
!!
!!
!!
!!
!!!!!!
!!
!!
!
!!!!
! !
!
!
!
!!
!
!
!!!
!!!
!!
!
!
!!
!
!!
!
!
!
!!!!!!!
!
!
!
!!
!!
!!
!
!
!!
!
!
!!!
!!
!
!
!!!
!
!
!!
!
!
!!!
!
!
!!
!
!
!!
!
!
!
!!
!
!!
!!!!
!
!
!!!!!
!
!
!
!!!!!!!!!!!!!!
!
!
!
!
!!!
!
!!!!!
!!
!!!
!
!
!!!!!!!
!
!
!!!
!!!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!!
!
!
!!
!
! !!
!
!
!!
!
!
!
!!
! !
!!!
!
!!
!!
!
!
! !
!!!
!
!
!
!!
!
!
!
!!
!
!
!
!
!!!
!!
!
!!
!
!!
!!
!
!
!
!
!!!
!!
!
!!
!
!!
!
!
!
!!
!!!!
!
!!
!
!
!
!!
!
!!!!!!
!
!!
!
!
!!!!!
!!
!!!!!!
!
!!!
!
!
!
!!!
!!
!!!!!
!!!!!
!
!!!!!
!
!
!
!!
!!
!
!
!!!!!!!!!
!
!
!
!!
!
!
!!!!!!
!
!!!!!!
!
!!!!
!
!
!!
!
!
!!
!!
!
!
!! !
!
!!
!!!!!!!!!!!!!!!
!
!!!!!!!!
!!!!!
!
!!
!!
!!!!!!!!!
!
!
!
!
!!!!
!!
!!!
!!
!
!
!
!
!!!!!
!
!!!
!!
!!
!!
!
!
!
!!!!!
!!!!
!!!!!!!
!
!
!!
!
!
!!!!!
!
!
!
!
!!
!
!
!
!!
!
!
!
!
!
!
!
!!
!!
!
!!
!
!!!
!
!
!
!
!
!
!
!!
! !
!
!
!
!!
!
!!
!
!
!
!
!
!! !
!
!!
!!
!
!!
!
!
!!
!!!!!!
!
!
!
!
!!!!!!!!
!
!!!
!
!!!
!
!
!
!!!!!!!!
!
!!!!!!!
!
!
!
!!!!!!!!
!
!
!
!
!
!
!!!!
!!!
!
!!!!!!!!
!!
!!!!
!
!!
!
!!
!
!
!!
!
!
!!
!!!!
!
!!
!
!!
!
!
!!
!!!!
!
!!
!
!
!!!!!
!
!
!
!
!
!!
!!
!
!
!
!!!!!!!!!!
!!
!
!!
!
!
!
!!!
!
!!
!
!!!!!
!!
!!!!
!
!!
!
!!!
!
!!!
!
!
!
!
!
!
!
!
!
!!
!
!!
!
!!!!!
!
!
!
!!
!
!!!!!!!!!!!!
!
!
!
!
!
!
!!!
! !
!!
!
!
!
!!!! !
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!!!
!
!
!
!
!
!!!
!!
!
!!
!
!
!
!
!
!!
!!
!!
!
!!!!!!!
!
!
!!!!!!!!
!
!!!!
!!!
!
!!
!
!
!!!
!
!
!
!!
!
!!!!
!
!!
!
!
!!
!!!
!
!
!!!!!!!!!!
!
!!!!!
!
!!!!
!!!
!!!!!
!!!!!
!
!!!!!!!!!!
!
!
!!
!!!!!!!!!!!!!!
!!
!
!!
!
!
!!
!
!
!!!!
!
!!!
!
!!
!
!!!!!!!!!!
!!!!!!
!!
!!
!!!!!!!
!
!!!!
!
!
!
!
!!
!
!
!!
!
!
!!!!
!
!
!
!
!
!!
!!!
!
!!!!!!!!!!!
!
!!
!!!
!
!!!!!!!!!!
!
!
!
!
!!
!
!!
!
!
!!!!
!
!!
! !
!
!
!!
!
!!!
!
!!!!
!!!
!
!!
!!
!!!!!!!!!
!!
!!
!
!!
!!
!!!!
!!!!!
!!
!
!!!!
!
!
!!
!!!!
!
!! !
!!
!
!
!!!!!!!
!!!!!
!
!
!
!
!!!
!!!!!!!!!
!
!
!
!!!!!!!!!!!
!!
!
!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!
!!!!!
!
!! !!! !!
!
! !!!
!
!
!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!
!
!!! !!!
!
!
!
!!!
!!
!
!!
!
!
!
!
!!
!
! !!!
!!
!!!!!
!!!!
!!!
!
!
! !!!!!
!!!!!
! !!!
!
!
!
!
!!!!!
!!!
!
!
!
! !!!!!!!
!!!
!
!
!!
!
!
!
!
!!!
!
!
!!
!
!
!
!!
!
!
!
!!
!
!!!
!!
!
!
!
!!!
!
!
!!
!
!
!!!
!!!
!
!
!!!!
!!!!
!!!!!
!!!!
!
!!!!!!!!!!!!!!!!!
!
!!!!!!! !
!
!!!
!!!!!
!!
!
!!
!
!!!
!
!!!
! !!!!!!
!
!
!!
!!!
!!!!
!! !!
! !
!!
!!!
!!!
!
!
!
!!
!
!!!!!
!
!!!
!
!
!!
!!
!!!
!
!
!
!
!!
!
!
!!
!
!
!
!
!
!!
!!
!
!!
!
!!
!! !
!
!!
!!
!!!
!
!
!!!!!
!!
!!
!
!
!
!!!!!!
!!!
!
!
! !
!
!
!!
!
!!
!!
!!!
!!
!!
!
!!
!!
!!!! !!
!
!
!
!!!
!
!!
!
!
!
!!!
!!!!!!!!!
!
!
!
!
!
!!!
!!!!
!
!!!!!
!!
!
!
!
!!!!!!
!!!!
!!
!!
!!
!!!
!
!!
!!
!
!
!!
!
!
!
!
!!
!
!!
!
!!
!!
!
!
!!
!
!
!!
!!
!
!!!
!
!
!!
!
!
!!!!!!
!
!
!!
!!!!!!!
!!
!
!!
!
!
!
!
!
!
!!
!
!!!
!
!!
!!
!
!
!
!
!!!
!
!!!
!
!!
!
!
!!!
!!
!!
!
!
!
!
!!
!
!!!!!
!
!!!!
!!!
!!!!
!!
!!
!!!
!!!
!!!
!
!!
!!! !
!
!
!!!!
!!!!!!!
!
!!!!!
!
!
!
!
!!
!!
!
!! !
!!
!!
!
!! !!!!!
!
!!
!!
!!
!
!
!
!
!
!!!
!!!!
! !!!
!
!!!
!!!!
!!!!
! !!!
!!
!
!
! !!! !
!!!
!!!
!
!!!!
!
! !
!
!!!!
!!!!!
!!
!
!!!!
!
!!!
!
!!!
!
!!!
!!!!
!!!!
!!!!
!!
!!!!
!!
!!!
!
!!!
!
!
!
!
!
!
!!!
!
!
!
!!!
!
!
!
!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!!
!!! !
!
!
!!!
!
!
!
!!!
!
!
!
!
!!
!
!
!
!!!
!
!
!
!
!!
!!
! !
!
!
!!!
!!!
!
!!
!!!
!!!
!!
!
!!
!
!!!!
!
!
!!!
!!!
!!!
!
!!
!!!
!
!
!
!!
!
!
!
!
!!!
!
!
!!!
!
!!!!!!
!
!
!
!
!
!
!!!!!!
!
!!
!
!
!
!!!
!!!
!!
!
!!
!
!!!
!
!!!
!
!
!
!
!
!
!
!
!
!!
!
!
!!
!!
!
!
!
!
!
!!
!
!!!
!
!!!!
!!!
!
!!
!!
!!
!
!
!
!
!
!!!
!
!
!
!!!!
!!
!
!
!!!!
!
!
!
!!!!!!!!!!!!
!!!!!!!!!
!!!
!!!!!!!!!!
!! !!!!!!!!!
!!
!
!!
!!!!!!!!!!
!
!!
!!!
!!!
!!!
!!!
!!!!!!!!!
!!
!
!!!!!!
!!!
!!!!!!!!!
!
!!
!!!!!!
!
!
!
!!!
!!!
!
!
!!!!
!!!
!!!
!!!
!!!
!
!
! !
!
!
!!
!
!!
!!!!
!!
!!
!
!
!
!!
!
!
!
!!
!
!!!
!
!!
!!!
!
!!
! !!
!
!
!
!
!
!
!
!
! !!!!
!!
!
!
!
!
!
! !!!!!!!!
!!
!!!!!
!
!
!!
!!!
!!!!
!!
!
!!
!!!!
!
! !
!!!
!!!
!!!!!!!!!
!!!
!!!!
!!!!!
!
!!
!!!!
!!!!
!!!!
!!!!!
!!!!!
!
!
!
!
!!
!
!
!!!!!!
!
!!!!
!!!
!!
!
!!
!!!
!!!!
!!!
!!
!!!!
!!!
!!!!!!
!!!
!
!!!!
!!
!
!!!!!
!!
!!
!!!!
!!!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!!
!
!!!
!
!!!!!!!!!!!!!
!
!
!
!
!
!
!
!
!
!!!
!!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!
!
!!! !
!
!
!!!
!!!!!!!!!
!!!!!!!
!!!!
!
!!
!!!
!!!!
0.5 0.6 0.7 0.8 0.9 1.0
−20
12
invlogit(predict(rf))
resi
d(rf)
/resi
d.se
(rf)
!
!!!!
!!
!!
!
!
!!
!
!!!
!
!
!!
!
!!
!
!!
!
!!
!!!!
!
!
!
!!!!!!
!!
!
!
!
!
!
!!!
!
!!!!!!
!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!!
!
!!!!!
!
!!!
!!!
!!!!!
!
!
!
!!
!!!!!
!
!
!!!!!
!
!!
!
!!
!
!
!!
!
!!!!!!
!
!!!!
!
!!!!
!
!
!
!!
!
!
!
!
!!!
! !!!!
!
!! !
!
!!
!!
!
!!!!!!!
!
!
!
!!
!!
!!
!
!
!!
!!!!!
!!
!!
!!!!
!!!!
!
!!!!
!!!
!!
!!
!
!
!
!!
!
!!!!
!
!!
!
!!!!!!!
!
!!!!!
!
!!!!!!!!!!
!
!!!!!!
!!!!!!!!!
!
!
!!!!!!!!!!!!
!!!!
!
!!
!! !
!
!
!
!
!
!
!
!
!
!!
!
!
!!
!!
!!
!
!
! !
!!
!!
!
!!
!!!
!
! !
!!
!
!
!!!!!
!
!
!
!!!
!
!
!!
!
!!!!!!
!
!
!!!!!
!!!
!
!!
!
!!!
!!
!!!
!
!!
!
!
!
!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!
!!!!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!!!!!!!!!
!
!
!
!!!
!
!!!!!!
!
!!!!!!
!
!!!
!
!
!
!!!
!
!!
!
!
!
!
!!
!
!
!!
!!!!!!!!!!!!!!!
!
!
! !!!!!!!!!!!!!!
!!
!!!!!!!!!!
!!!!!!!
!!
!!!!!!
!
!
!!!!!!
!!!!!!!!!!
!!
!
!!!!!!!!!!!!!!!!!!
!!
!!!!!!!!
!!!!!!!!
!
!
!
!
!
!
!
!
!
!!
!!
!
!!
!
!!!!
!
!!
!
!
!
!!! !
!
!
!
!!
!
!!
! !!
!
!
!! !
!
!
!
!
!
!
!!!
!
!!
!
!!!!!!
!
!
!
!!!!!!!!
!
!!!
!
!!!
!
!
!
!!!!!!!!
!
!!!!!!!
!
!
!
!!!!!!!!
!
!!
!
!
!!!!
!
!
!!
!
!!!!!!
!
!
!
!!!
!!!
!!
!!!
!
!
!!
!
!
!!
!!!!
!
!!
!
!!
!!
!
!
!!!!!!!!!!!!!!
!
!
!
!
!
!
!!!
!
!
!!!!!!!!!!!
!!
!
!!!
!
!
!!!
!
!!
!!!!!!
!
!
!!!!!!!!!!!!!!!!!
!!
!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!
!!
!!
!
!
!!
!
! !!!
!
!!
!!!!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!!
!!
!!
!! !
!
!
!!
!!
!
!!
!
!
!
!
!
!!
!!
!!
!!!!!!!!
!
!!!!!!!
!
!
!
!!
!!
!
!!
!
!!
!!
!!
!
!
!!!!
!
!
!
!!
!
!!
!
!
!
!
!!!
!
!!!!!!!!!!!
!
!!!!!
!
!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!!
!!!
!
!
!!
!
!
!!!!!!!!
!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!
!!
!
!
!!
!
!
!!!!!
!
!
!
!
!
!
!!!
!
!!!!!!!!!!
!
!
!
!!!!
!
!!!
!!!!!
!
!
!
!
!
!
!!
!
!!
!
!!!!!
!
! !
!!
!
!
!!
!
!!!
!
!!!!
!!
!
!
!!!!!!!!!!!!!
!!
!!
!!!
!!!!!!!!!!!!
!
!!
!!!!
!
!!
!!
!
!
!
!
!!
!
!
!
!
!!!!!!
!!!!!!
!
!
!
!
!!
!
!!!!!!!!!
!
!!
!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!
!!!
!!!
!!
!!!
!
!
!
!!!!
!
!
!
!
!!!!
!!
!
!
!!
!
!
!
!
!
!
!
!
!!!!
!
!
!!!!
!!!
!!!!
!
!!
!
!!
!
!
!
!
!
!
!
!!!!
!
!
!!!!
!
!
!!!
!!
!
!!
!
!!!
!!
!!
!!!
! !
!!
!
!
!!
!!!!
!
!
!!
!
!
!
! !!!!!!!!!!
!
!
!!
!
!
!
!
!
!!
!
!
!!!
!
!
!!
!
!
!!!
!
!!!!!!
!
!
!!!
!
!
!!!
!
!
! !!!!
!
!
!!!!
!!!
!
!
!!!
!!!!!
!
!
!!
!!
!!!!
!!!!!!!!
!
!!!
!!!
!
!
!
!!!!!
!!!!!
!
!
!
!
!!!
!
!
!!
!
!!!!!!
!
!
!
! !!
!
!
!!!!
!!!!
!!
!
!!
!
!!!
!
!
!
!!
!
!!!!
!
!
!!!
!
!
!!
!!
!!!
!
!
!!!
!
!
!
!
! !
!
!
!!
!!
!!
!
!!
!
!!
! ! !
!!!
!
!
!!!!
!
!!!!!
!!!
!
!
!
!
!!!!!!
!!
!
!
!
!
!
!
!
!!
!!!
!
! !!!
!
!
!!
!
!
!
!!!
!!
!!!
!
!
!
!!!
!
!
!
!
!
!
!!
!!
!
!!
!! !!
!
!
!
!
!!
!!
!
!!
!! !
!
!!!
!
!!
!
!
!
!!!
!!!
!
!!!
!!
!!
!!!!!!
!!!!
!
!
!!
!!
!
!
!!
!!!!
!!
!!
!
!
!!
!!
!!
!!!!!!!!!!
!
!
!!
!
!
!!!
!
! ! !!!!!
!!!
!
!
!!
!
!
!
!
!
!
!!
!
!!!
!!!
!
!!
!
!
!
!!!
!
!!! !
!!
!!!!!
!!
!!
!!!
!
!!!
!!!!
!
!!!!!
!!!
!!!! !!!!
!!!
!!!
!!!
!
!!
!!! !
!!!!!!!!!!!!!!!!!!!!
!
!!
!!!!
!
!! !!!!
!
!
!!
!!!!!
!
!!!
!!!
!
!!
!
!
!!!
!!!!! !!!
!!!!
!
!!!
!!!!
!!
!!
!!
!
!
! !!!
!!
!!
!!!
!
!!!!!! !!
!!
!
!
!!!!!
!!!
! !!!
!
!!!
!
!!!
!!!!
!
!!!
!
!!!
!!!!
!!!!!!
!!!
!!
!!
!!
!
!
!
!
!
!!
!!
!
!
!
!!!
!
!
!
!
!
!
!!
!
!
!
!
!!
!
!
!
!
!!!
!
!!!
!!
!
!
!
!
!! !
!
!
!!!!
!
!
!!!
!!!
!
!!
!
!
!
!!!
!!! !
!!!!
! !
!
!
!!
!!!
!
!
!!
!!!
!!!
!
!!
!!!
!!!!
!
!
!!!
!!!
!
!!
!
!
!
!!!!
!!
!!!!
!
!
!!!
!
!
!!!
!
!!!!!!
!!
!
!
!!
!!
!!
!!
!!
!
!
!
!
!!!
!!!
!!
!
!!
!
!!!
!
!!
!
!
!
!
!
!
!
!
!
!
!!!
!
!!
!!
!
!
!
!
!
!!
!
!!
!
!
!!!!!!!
!
!!
!!!!
!
!
!!!
!!!!
!
!
!!!!
!
!
!
!
!!!!!
!
!
!!!!!!!!!!
!
!
!!!!!!!
!
!!
!!!!!!!!
!
!
!
!!!!!!!!!!
!!
!!
!!!!!!!!!!
!
!!
!!!!!!
!!!
!!!
!!!
!!!!!!
!!
!
!!!
!!!
!!!
!!!!!!!!!
!!!
!!!!!!
!
!
!
!!!
!
!
!
!!!!!!
!
!!
!!!
!!!
!
!
!
!
!
!
!!!!!!
!!!!!!
!!
!!
!
!
!
!
!!
!
!
!!
!
!!
!
!
!!
! !!!
!!
!
!
!!
!
!
!!!
!!!!!!!
!!
!
! !
!
!
!!!!!
!!!!!!
!!!!!
!
!!!! !!!!!!!!
!
!!!!!!
!
!!
!!!
!!!
!!!!!!!!!!!!
!
!!!
!!!!!
!
!
!
!!!!
!!!!
!!!!!!!!!!!!!!!!
!!
!!
!!!!!!!!!!!!!
!!
!
!!!!!!!!!!!!
!!!!
!
!!!!!!!
!!!!!!
! !
!!
!!!!
!!
!!
!!!!
!
!
!
!
!!!!
!!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!!!!!
!
!!
!
!
!!!!!!!!!!!!!
!
!
!
!
!
!
!
!
!
!!
!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!!
!!
! !
!!!!!
!!!!!!!!!
!
!!!!!!!!!!!!! !!!
!!!!
0.5 0.6 0.7 0.8 0.9 1.0
−3−1
13
resi
d(rf,
inv
= TR
UE)
/resi
d.se
(rf, i
nv =
TRU
E)
!
!!!!!!
!!
!
!
!!!
!!
!
!
!
!!!
!!
!
!!
!
!
!!!
!!
!
!
!
!!! !!!
!!
!
!
!!!
!!!
!
!!
!
!!!
!
!
!
!!!!!!
!
!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!!
!!!!!
!
!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!!
!!!!!!!!!!
!!!
!!!!!
!!
!!!
!!
!!!
!
!!!!!!
!
!!!!!!!
!!
!
!!!!!!
!
!!!!
!
!!!!
!!!
!!
! !
!
!!!!! !!!
!!
!!!
!
! !
!!
!
!!!
!!!!
!
!
!
!!
! !!
!
!
!
!!
!!!!!!!!
!
!!!!
!!!!
!
!!!!
!!!
!!
!!
!
!
!
!!
!
!!
!!!!
!
!
!!!!!!!
!
!!!!!!!!!!!!!!
!
!
!
!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!!
!!!!
!
!!!!!
!
!
!
!
!
!
!
!
!!!
!
!
!!
!!
!
!
!
!
! !
!!
!
!!
! !
!!!
!
! !
! !
!
!
!
!
!!!
!
!
!
!!!
!!
!!
!
!
!
!!!!
!
!
!!!
!!!!!
!
!!
!
!!!
!!
!!!
!!!!
!
!
!!
!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!
!
!!!
!!
!
!
!
!
!
!
!
!
!
!!
!!!!!!!
!
!!!!
!
!
!!!!!!
!
!!!!!!
!
!!!!
!!
!!!
!
!!
!
!
!
!
!!!
!
!!
!!!!!!!!!!!!!!!
!
!! !!!!!!!!!
!!
!
!!
!!
!!!!!!!!!!!!!!!!!
!!
!!!!!
!
!
!
!!!!!!
!
!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!
!!
!!
!!!!!!!
!!!!!!!!
!
!
!
!
!
!
!
!
!
!
!!
!!
!
!!
!
!!!
!
!
!!
!
!
!
!!
!
!
!
!
!
!!
!
!!! !!
!
!
!! !
!
!
!!
!
!
!!
!
!
!!
!
!!!!!!
!
!
!
!!!!!!!!
!
!!!
!
!!!
!
!
!
!!!!!!!!
!
!!!!!!!
!
!
!
!!!!!!!!
!
!!!
!
!!!!!
!!
!
!
!!!!
!!!!
!!
!!!!
!
!!
!
!!
!!
!!
!
!
!!
!!
!!
!
!!
!
!!
!
!
!!!!!!!!!!!!!!!!
!
!
!
!
!!!!!!
!!!!!!!!!!!!!!
!
!!!
!
!
!!!
!
!!!!!!!!
!
!
!!!!!
!!!!!!!!!!
!!
!
!
!!!!!
!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!
!!
!!
!
!
!!
!
!!
!!
!
!!
!!!!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
!!
!!
!
!
!
! !
!!
!
!!
!
!
!
!
!
!!
!
!
!!
!
!!!!
!
!!
!
!!!!!!!
!!
!
!!!!
!!!
!
!!
!!
!!
!
!
!
!
!!
!
!
!
!!
!
!!
!
!
!!!!!
!
!!!!!!!!!!!
!
!!!!!
!
!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!!
!!!
!
!
!!
!
!
!!!!
!!!!!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!
!!
!
!
!!!
!
!!!!!
!
!
!
!
!
!
!
!
!
!
!!
!!!
!!!!
!
!
!
!
!!!!
!
!!!!!!!!!
!
!
!!
!
!!
!
!!
!
!!!!!
!
!!
! !
!
!
!!
!
!!!
!
!!!
!
!! !
!
!!!!!!!!
!!!!!
!!
!
!
!!!!!!!!!!!!!!!!
!!!!
!!
!
!
!
!
!!
!
!
!
!
!
!
!!
!
!!!
!!!!!!!!!
!
!
!
!
!!!!
!!!!!
!!!
!
!!
!!!!!!!!!!!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!
!!
!
!
!!
!
!
!!!!
!
!
!!!!
!
!
!
!
!!
!! !!!
!
!
!
!
!
!
!
!
!
!
!
!!
!!!
!
!!!!
!!!
!!
!!
!
!!
!
!!
!
!
!
!
!!
!
!!!!
!!
!!!!!
!
!!!
!
!!
!!
! !!!
!!
!!!!!
! !
!!
!
!
!!
!!
!!!
!
!!!
!
!
! !!!!!!!!!!
!
!
!!
!!
!
!!!!!
!
!!!
!
!
!!
!
!
!!!!!!!!!!
!
!!!
! !
!
! !!
!
!!
!
!!!
!
!
!
!!!
!!!
!
!!!!!!!!!
!
!!!!!!!!!!!!!!!!!
!
!!!!!!!
!
!
!!!!!
!!!!!
!
!!
!
!!
!
!
!!!!
!!!!!!
!
!
!! !!
!!
!!!
!
!!!!
!
!
!
!!
!!! !
!
!
!
!!
!
!!!!
!
!
!!!
!!
!!
!!
!
!!!
!
!!!
!
!
!
!
! !
!
!
!!
!
!
!!
!
!!
!
!!
!
! !
!
!!
!
!
!
!!!
!
!!!!!
!!
!!
!
!
!
!!!!!
! !!!
!
!
!!
!
!
!!
!!!
!!
!!!!!
!! !
!!
!!!
!!
!
!!
!
!
!
!!!
!
!
!
!
!
!
!!!
!!
!!!!!!!
!
!
!
!
!
!!!
!!!
! !
!!!!!
!!
!
!
!!
!!!
!!!!!!
!!
!!
!!!!!!
!!!!
!
!
!!
!!
!
!
!!
!
!!!
!!
!!
!
!
!!
!!
!!
!!!!!!!!!!
!
!
!!!
!
!!!
!
!! !!!
!!!!!!
!
!!
!
!
!!
!
!
!!
!
!!
!
!
!
!
! !
!
!
!
!
!!!
!
!!!
!
!!
!
!!!!!
!
!!!!!
!
!!!!!!!
!
! !!!!!!!
!
!!! !!!!
!!!
!!!
!!!
!
!!!!! !
!!!!!!!!!!!!!!!!!!!!
!
!
!
!!
!!
!
!!!!!
!!
!
!!
!!!!!
!!
!!
!!!
!
!!
!
!
!!!
!!!!! !
!!
!!!!
!!!!
!
!
!!
! !!!
!!
!
!
!!!!
!
!!!
!!!
!
!!!
!!
!!!
!!
!
!
!
!!!
!
!!!
! !!!
!
!!!
!
!!
!
!!!!
!
!!!
!
!!!
!!!!
!!!!!!
!!
!!!!
!!!!
!
!
!
!
!!
!!
!
!
!
!!!
!
!!
!
!
!
!!!
!
!
!
!
!!
!
!
!
!
!
!
!!!
!!
!
!
!
!!!! !!
!
!!!
!
!
!
!!!
!!!
!
!!
!
!
!
!!!
!!
! !!
!!!
! !
!
!
!!!
!!!
!
!!
!!!
!!!
!
!
!
!!!
!!!!
!
!
!!!
!!
!
!
!!
!
!!
!!!
!
!!
!!!
!
!
!!!!
!
!
!!!
!
!!!!!
!
!!!
!
!
!
!
!!!
!!
!!!
!
!
!
!!!
!!!
!
!!
!!
!
!!!
!
!
!!
!
!
!
!
!
!
!
!
!
!!
!
!
!!
!!
!
!
!
!
!
!!
!
!
!
!
!
!!
!! !!!
!!!
!!!!
!!
!!!
!!!
!
!
!
!!!!!!!
!
!!!! !!
!
!!!!!!!!!!!
!
!!!!!!!!!
!!!
!!!!!!!!!!
!
! !!!!!!!!!
!!!
!!
!!!!!!!!!!
!!
!
!!!!!!
!
!
!
!!
!
!!!!!!!!!
!!
!
!!!!!!
!!!
!!!!!!!!!
!!!
!!!!!!!!!!!!
!
!
!!!!!!!
!!!
!!!!!!!!!
!!!
!!!!
!!
!!! !
!!
!!
! !
!
!
!!
!!
!
!
!
!
!
!!
!
!
!!
!!!!
!
!
! !!!
!
!
!!!
!!! !!!!!!
!
!
!!
!
!
!!!! !!!!!!
!!!!!
!
!!!! !!!!!!!!
!
!!!!!!
!
! !
!!!
!!!
!!!!!!!!!!!!
!
!
!!
!!!!!
!
!!
!!!!
!
!
!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!
!
!!!!!!!!!!!!!
!!!
!
!!!!
! !
!
!!!!
!
!!
!!
!!!!
!!!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!!
!!!!!!!
!!!
!
!!!!!!!!!!!!!
!
!!
!
!
!!!
!
!!!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!!
!!!
!
!!
!!!
!!!!!!!!!
!!!!!
!!!!!!!!!!!!
!!!!
0.5 0.6 0.7 0.8 0.9 1.0
−20
12
3
resi
d(rf,
inv
= TR
UE)
/resi
d.se
(rf, i
nv =
TRU
E)
!
!!!!
!!
!!
!
!
!!!
!!!
!
!
!!!!!
!
!
!
!
!
!!!
!!
!
!
!
!!!!!!
!!
!
!
!!
!
!!!
!
!
!
!
!!!
!
!
!
!
!!!!!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!
!
!
!
!!!!!
!
!
!
!!!!!!!!
!
!!
!!!
!
!!!
!
!!
!
!!!! !
!
!
!!
!
!
!!!
!
!
!!!
!
!
!
!!
!
!!
!!
!!
!
!
!!!!!!!!
!!
!
!!!!
!!
!
!!
!!
!
!
!!!
!!!
!!
!
!
!!
!
!!
!
!
!
!!!!
!!!
!
!
!
!!
!!
!!
!
!
!!
!
!!!!!!!
!
!!!
!
!!!!
!
!!!!
!!!
!!
!!
!
!
!
!!
!
!!!!!!!
!
!!!!!!!
!
!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!
!!!!
!
!
!!!
!!
!
!
!
!
!
!
!
!!!
!
!
!!
!!
!!
!
!
! !
!!
!
!!
! !
!!!!
! !
!!
!
!
! !!!!
!
!
!
!!!
!!
!!
!
!
!
! !!!
!
!
!!!
!! !
!!
!
!!
!
!!!
!!
!!!
!!!!
!
!
!!!!!!!!!!
!
!!!!!!!!!!
!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!
!!
!
!!!!!
!
!
!
!
!
!
!
!
!
!!
!!!
!!!!
!
!!
!
!!
!
!!!!!!!!!!!!!
!
!!!!!
!
!!
!
!
!!
!!
!
!
!!
!
!
!!
!!!!!!!!!!!!!!!
!
!! !!!!!!!!!!!
!
!!
!!
!!!!!!!!!!!!!!!!!
!!
!!!!!
!
!
!
!!!!!!
!!!!!!!!
!!
!
!
!
!!!!!!!!!!!!!!!!
!
!
!!
!!
!!!!!!!!!!!!!!
!
!
!
!
!
!
!
!
!
!!
!!
!
!!
!
!!!
!
!
!
!
!
!
!!!
!!
!
!
!
!!
!
!!!
!
!
!
!
!! !
!
!!
!!
!
!!
!
!
!!!
!!!!!!
!
!
!
!!!!!!!!
!
!!!
!
!!!
!
!
!
!!!!!!!!
!
!!!!!!!
!
!
!
!!!!!!!!
!
!!
!
!
!!!!!
!!!
!
!
!!!!!!
!
!!
!!!!
!
!!
!
!!
!
!
!!
!
!
!!
!!
!!
!
!!
!
!!
!
!
!!
!!!!!!!!!!!!!!
!
!!
!
!
!!!!
!
!
!!!!!!!!!!!!!
!
!!!
!
!
!!!
!
!!!!!!!!
!!
!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!!!
!
!
!!!!!!!!!!!!!!!
!
!
!
!
!
!
!!
!
! !
!!
!
!!
!!!! !
!
!
!
!
!!
!!
!
!
!
!
!
!!
!!!
!
!
!
!
!
!! !
!!
!
!!
!
!
!
!
!
!!!!
!!
!
!!!
!!!!
!
!
!!!!!!!!
!
!!!!
!
!!
!
!!
!
!
!!
!
!
!
!
!!
!
!
!!!
!
!!
!
!
!!
!!!
!
!!!!!!!!!!!
!
!!!!!
!
!!!!!!!!!!!!!!!!!
!
!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!
!!
!!!
!
!
!!
!!
!!!!!!!!!
!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!
!!
!
!
!!
!
!
!!!!
!
!
!
!
!
!!
!!!
!
!
!!!!!!!!!
!
!
!
!!!
!
!
!!!!!!!
!!!!
!
!
!
!!
!
!!
!
!!!!!
!
! !
! !
!
!
!!
!
!!
!
!
!
!!!
!! !
!!!!!
!!!!!!!!!
!!
!!
!!!!!!!!!!!!!!!
!
!
!!!!
!
!
!
!
!
!!
!
!
!!
!
!!
!
!
!!!!!!!!!!!!
!
!
!
!
!!!
!!!!!!!!!
!
!
!
!!!!!!!!!!!!!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
!
!
!!
!!
!!!
!
!
!!!
! !!
!
! !!!
!
!
!
!
!!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!!!
!
!!! !
!!
!
!!
!!
!
!!
!
!!
!
!
!
!
!
!
!
!!
!
!!
!
!!!
!!
!
!!!
!
!
!
!
!
! !!!
!!
!!
!!!
! !
!!
!
!
!!
!!
!!
!
!
!!!
!
!
! !!!!!!!
!!!
!
!
!!
!
!
!
!!!!
!
!!!
!
!
!
!!
!
!
!
!!!
!!!!!
!
!
!
!!!!
!
!!
!
!
!
!!
!!!
!
!
!!!!
!!!
!
!!!!!
!!!!
!
!!!!!!!!!!!!!!!!!
!
!!!!!!! !
!
!!!
!!!!!
!!
!
!!
!
!!!
!
!
!!
!!!!
!!!
!
!
!!
!!!
!!!!
!
!!!!
!
!
!
!!
!
!!!
!
!
!
!!
!
!!!!!
!
!!!
!!
!!
!!
!!!
!
!
!
!
!!
!
!
!
!!
!
!
!!
!!
!!
!
!!
!
!!
!
! !
!
!!
!
!
!!!!
!
!!!!!
!
!!!
!
!
!
!
!!!!!
!!!
!
!
!!
!
!
!
!
!
!!
!
!!!!
!
!
!! !
!!
!!
!!!!
!!
!
!
!
!!!
!
!
!
!
!
!
!!
!
!!
!!
!!!!!
!
!
!
!
!
!!!
!
!!
!!
!!!!!
!!
!
!
!
!!!!!!
!
!!!
!!
!!
!!!!!!
!!!!
!
!
!!
!
!
!
!
!!
!!!!
!!
!!
!
!
!!
!!
!!
!!!!!!
!!!!
!
!
!!!!!!
!
!
!
! !!!!!
!!! !
!!!
!
!
!!
!
!
! !
!
!!!
!
!!
!
!!!
!
!
!!!
!
!!!
!
!!
!
!!!!
!!
!!
!!!
!
!!!
!!!!!
!!!!!
!!!!
!!! !!!!
!!!
!!!
!!!
!
!!!!! !
!!!!!!!!!!!!!!!!!!!!
!!
!
!!!!
!
!! !!!!!
!
!!!!!!!
!
!!!! !!
!
!!
!
!
!!!
!!!!! !!!
!
!!!
!!!!
!!!!
!!!!
!!
!
!
! !!
! !
!!!
!!!
!
!!!!!
!!
!
!
!!
!
!!!!!
!!!
!
!!!
!
!!!
!
!!
!
!!!!
!!!!
!
!!!
!!!!
!!!!!!
!!
!
!!
!
!!!
!
!
!
!
!
!
!
!!
!
!
!
!!!
!
!
!
!
!
!
!!!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!!
! !
!
!
!!!
!
!
!
!!!
!
!!
!
!!
!
!
! !!!!
!
!!
!!
!!
!!
!
!
!!!
!!!
!
!!!!!
!!!
!
!
!
!!!
!!!
!!
!
!!!
!!!
!!!
!
!!
!!!
!
!
!
!!
!
!
!
!
!!!
!
!
!!!
!
!!!!!!
!
!!
!
!
!
!!!!!!
!
!!
!
!
!
!!!
!!!!!
!
!!
!
!!!
!
!!!
!
!
!
!
!
!
!
!
!
!!!
!
!!
!!
!
!
!
!
!
!!
!
!!!
!
!!!!!!!
!!!
!!!!
!
!
!!
!
!!!!
!
!
!!!!!
!
!
!
!!!! !
!
!
!!!!!!!!!!
!!
!!!!!!!!!
!
!!
!!!!!!!!!!
!!!!!!!!!!!
!
!
!
!!! !!!!!!!!!
!
!!!!!!!!
!!
!!!!
!!!!!!!!!
!!!
!!!!!!!!!!!!!!!!!!
!!!
!!!!!!!!!!!!
!
!
!
!!!!!!
!
!
!
!!!!!!!!!
!!! !
!!
!!!
!!!!!!
!
!
!!
!
!
!
!!
!
!
!
!!
!
!!
!
!
!!
!!!
!
!!
!!
!!
!
!
!!
!
!
!
!!!!!!!
!
!
!!
!
!!!!!
!!!!!!!!!!!!
!!!! !!!!!
!!!
!
!!!!!!
!
! !
!!!
!!!!!!!!!!!!!!!
!!!!
!!!!!
!
!
!
!!!!
!!!!
!!!!!!!!!!!!!!!!!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!! !!
!!!!!!!
!!!!!!!!
!
!
!!!!
!!
!
!
!!!!!
!
!
!
!!!!!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!
!!
!!!!!!
!!!!!!!!!!!!!!!!!!!
!
!
!
!
!
!!
!
!!!
!!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!!!!
!
!!! !
!
!
!!
!
!!!!!!!!!!!!!!!!!!!!
!!!!!!
!!!!
0.5 0.6 0.7 0.8 0.9 1.0
−20
12
3
resi
d(rf,
inv
= TR
UE)
/resi
d.se
(rf, i
nv =
TRU
E)
13
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Conditional variable importanceStrobl et al. (2008), BMC Bioinformatics 2008, 9:307.
Predicting the quality of a survey question from its design characteristics Daniel Oberski
●
●
●
●
●
●●
● ● ● ● ● ●●
●● ● ● ●
● ● ●
● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
Conditional importance of variables for prediction of reliability coefficientC
ondi
tiona
l im
port
ance
mea
sure
domain
concept
labels
scale_corres
questiontype
stimulusrange_corres
0.00
0.02
0.04
0.06
0.08
dom
ain
conc
ept
labe
lssc
ale_
corr
esqu
estio
ntyp
est
imul
usra
nge_
corr
esnc
ateg
orie
sin
terv
iew
ersh
owc_
horiz
grad
atio
nfu
ture
posi
tion
labe
ls_o
rder
bala
nce
cent
ralit
yfo
rm_b
asic
repe
titio
nco
untr
yav
gwrd
_tot
alns
yll_
tota
lfix
refp
oint
sus
ed_W
H_w
ord
visu
alsc
ale_
basi
csc
al_n
eutr
alop
inio
noth
ersh
owc_
star
tsc
ale_
tran
geus
edsh
owca
rdav
gsy_
tota
lnw
ords
_tot
alsc
ale_
uran
gesh
owc_
ques
tso
cdes
irdo
nt_k
now
com
p_as
sist
show
c_le
tter
nnou
ns_i
ntro
nnou
ns_t
otal
nsub
_que
stsy
mm
etry
know
ledg
eav
gabs
_int
rosh
owc_
boxe
ssh
owc_
over
nabs
t_in
tro
mot
ivat
ion
nabs
t_to
tal
intr
opre
sent
inst
r_re
spon
abso
lute
avgw
rd_i
ntro
avga
bs_t
otal
nwor
ds_i
ntro
subj
ectiv
eop
enco
urag
ela
bels
_gra
mm
inst
r_in
terv
past
intr
_req
uest
num
sub_
intr
o
Predicting the quality of a survey question from its design characteristics Daniel Oberski
●
●
●
●
●
●●
● ● ● ● ●● ● ● ●
● ●● ● ● ● ● ●
● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
Conditional importance of variables for prediction of validity coefficientC
ondi
tiona
l im
port
ance
mea
sure
ncategories
domain
concept
labels
scale_corres
range_corres
fixrefpoints
0.00
0.02
0.04
0.06
0.08
ncat
egor
ies
dom
ain
conc
ept
labe
lssc
ale_
corr
esra
nge_
corr
esfix
refp
oint
sst
imul
usin
terv
iew
erba
lanc
efu
ture
labe
ls_o
rder
form
_bas
icpo
sitio
ngr
adat
ion
show
c_ho
rizre
petit
ion
coun
try
scal
e_ba
sic
nsyl
l_to
tal
avgw
rd_t
otal
cent
ralit
ysc
al_n
eutr
alus
ed_W
H_w
ord
show
c_st
art
scal
e_tr
ange
dont
_kno
wsc
ale_
uran
gevi
sual
nwor
ds_t
otal
used
show
card
avgs
y_to
tal
com
p_as
sist
nnou
ns_i
ntro
nnou
ns_t
otal
show
c_le
tter
opin
iono
ther
show
c_bo
xes
show
c_ov
ersh
owc_
ques
tns
ub_q
uest
nabs
t_in
tro
sym
met
rykn
owle
dge
socd
esir
ques
tiont
ype
intr
opre
sent
mot
ivat
ion
avga
bs_i
ntro
abso
lute
nabs
t_to
tal
avgw
rd_i
ntro
nwor
ds_i
ntro
inst
r_re
spon
avga
bs_t
otal
enco
urag
esu
bjec
tiveo
pin
tr_r
eque
stin
str_
inte
rvla
bels
_gra
mm
num
sub_
intr
opa
st
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Back to the example
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Evaluation and future work
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Some of the challenges encountered
• (Coding process)• (MTMM estimation process)• Mapping the old coding system into the new...• Missing data; non-applicable characteristics (→ raimforest)• Multicollinearity, aliasing of interactions (→ RF)• Propagating the uncertainty about the MTMM estimates
(not solved)• Coming up with a reasonable prediction interval (→ RF)• Dealing with main vs. supplementary (estimate an order
effect and assume the coded question is like the mainquestionnaire)
• ...
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Things that still need doing in my opinion• Publish journal articles to let the world know about this
meta-analysis!• Model was cross-validated on ESS R1–3. Also validate
predictions against new ESS MTMM results.• Evaluate practical effect on estimates of interest before &
after correction for reliability and method effect: howdifferent are the estimates in practice?
• Seriously reduce the number of codes• Planned experiments to reduce multicollinearity, aliasing in
the meta-analysis• Propagate uncertainty about the MTMM estimates• Perhaps compare w/ linear regression/other meta-analysis
techniques• ...
Predicting the quality of a survey question from its design characteristics Daniel Oberski
Thank you for your [email protected]
Predicting the quality of a survey question from its design characteristics Daniel Oberski