Reducing Biases in Economics and Business Research
T.D. Stanley (Tom)Honorary Professor, Alfred Deakin Research Institute
Julia Mobley Professor of Economics,
Hendrix College, USA [email protected]
“When reviewing an empirical literature, manyeconomists seem to become Bayesian, holdingstrong priors formed on the basis of theoreticalconsiderations and raising high barriers to anycontrary empirical evidence” –Stanley (2001)
The market is efficient and gov’t intervention only makes things worse
Global Financial Crisis: 2007-2008
Meta-Analysis of Economics Research-Network (MAER-Net)
•Prague, CZ—2015 (Sept 10-12)• Athens, Greece (2014)• Greenwich, UK (2013)• Perth, Australia (2012)• Cambridge University, UK (2011)• Hendrix College, US (2010)• Corvallis, Oregon US (09)• Nancy, France (08)• Sonderborg, Denmark (07)
to join: [email protected]
MAER-Net Colloquium at Prague-Sept 10-12
• http://ies.fsv.cuni.cz/en/node/516/index.htm
Conflicting Empirical Findings• Rarely do single studies provide definitive
answers upon which to base policy or to settle the theoretical disputes.
• Very large research variation is the norm.
The Problems of Economics Research
Econometric Misspecification BiasPublication Selection BiasLow Power- {Chris D., John Ioannidis & I for EJ}Researcher Commitment to Ruling Theory {Doucouliagos & Stanley (2013): the stronger the consensus about theory, the larger the bias.}Ideological Bias
Is Meta-Regression Analysis the Answer?
MRA:• Uses replicable methods to identify and
code all relevant research. • Employs objective statistical methods to
summarize and explain all research results. • Accommodates and corrects selection and
misspecification biases routinely found among reported economic findings.
• Provides a rigorous basis for evidence-based practice.
Our Book“In our view, the central task of meta-regression analysis is to filter out systematic biases, largely due to misspecification and selection, already contained in economics research.”
–Stanley & Doucouliagos (2012, p.16, Meta-Regression Analysis in Economics and Business, Routledge.
“Believing is seeing”–Demsetz (1974, p. 164).
• In Psychology, this is called: confirmation (or experimenter) bias
• For example, when asked “Are you happy with your social life?”—Y(73%); N(22%); Undec(5%)
• When asked: “Are you unhappy with your social life?” —Y(65%); N(27%); Undecided (8%)
• In Economics, Ideology and Conventional Theory frames and shapes how evidence is interpreted.
Believing is Seeing IIPublication Selection Bias• Reviewers and editors are predisposed to
accept papers consistent with the conventional view.
• Researchers may treat the conventionally expected result as a model selection test.
• Everyone may have a preference for ‘statistically significant’ findings.
—Card and Krueger (1995)
Picturing Research and Its Bias: Funnel Graphs
• A funnel graph is a scatter diagram of precision (1/SE) vs. estimated effect.
• In the absence of publication selection, estimates will vary randomly and symmetrically around true effect (β).
• The expected inverted funnel shape is dictated by predictable heteroskedasticity.
0
10
20
30
40
50
60
70
80
90
100
1/S
e
-.8 -.6 -.4 -.2 0 .2 .4 .6r
“Picture this: A simple graph that reveals much ado about research.” (Stanley and Doucouliagos, 2010, Journal of Economic Surveys)
Figure 1: Funnel Plot of Union-ProductivityPartial Correlations
Data: 73 partial correlation coeff’s of union membership and worker productivity
source: Doucouliagos, C. and Laroche, P. (2003)
Funnel Graph is a scatter diagram of all estimates and their precisions (precision= 1/SE)
Figure 2: Value of Statistical Life(in millions of 2000 US $s)
Data: 39 estimates of VSL from hedonic wage equations.
Source: Doucouliagos, Stanley, and Giles (2012), JHE.
Figure 3: Antidepressant Trials
Source: Turner et al. (2008).
Data: 74 effect sizes, Hedges g, from the US-FDA registry, considered the goldstandard. 50 of these were published.
2
3
4
5
6
7
8
9
10
11
Pre
cisi
on
-.4 -.2 0 .2 .4 .6 .8 1
g
FDAPublished
Conventional Neoclassical theory:
• predicts a strict LAW of DEMAND for labor {i,e., an inverse relation between wages and employment}.• Therefore, raising the minimum wage must
reduce employment.• All real economists know that government
intervention into the free market always has negative consequences. . . .
• Recall the controversy in the 1990s around Card and Krueger’s ‘Myth and Measurement.”
Figure 4: A graph seen around the world:Minimum Wage Employment Elasticities
Data: 1,474 US minimum-wage elasticities of employment. (Doucouliagos and Stanley, 2009)
Source: Economic Report of the President (2013)
-50
0
50
100
150
200
250
300
Prec
ision
(1/S
E
)
-20 -15 -10 -5 0 5
Employment Elasticity
Elasticity = -1
Testing for Publication Selection bias and Identifying Genuine Empirical Effect
FAT-PET-MRAej= β + β0Sej + εj (j=1, 2, …L) (1)
{Comment: Always use WLS; 1/Se2 as the weight}
FAT (funnel-asymmetry test): t-test of β0 is a test for publication bias (Egger et al., 1997)PET (precision-effect test): t-test of β is a test of authentic effect, beyond publication bias (Stanley, 2005; 2008; S&D, 2014)
FAT-PET-MRA-WLS– Eq. (1)Table 1: MRA Tests of PB & Genuine Effect
*Robust t-values in parentheses
Variable Union-Productivity
VSL Anti-Depressants
Minimum Wage
Se ( β0 ) .65 (1.72)* 3.20(6.67)* 1.84 (5.47)* -1.6(-4.49)*
Intercept β -.0179(-1.06)* 0.81(3.56)* .13 (2.50)* -.01(-1.09)*
Ave. Effect .03 $9.5 mil .47 -.19Top10 -.0003 $2.00 mil .32 -0.022
PEESE . 0009 $1.67mil .29 -.036n 73 39 50 1,474
Putting it all together with Multiple MRA
effecti=(β +ΣαkZik ) + (β0Sei +ΣγjKijSei )+εi (3)Heterogeneity & Publication BiasMisspecification
Biases
Table 2: Minimum-Wage Moderator VariablesK & Z
VariableDefinition Mean (standard
deviation)
Elasticity is the estimated effect, the dependent variable, and an elasticity -1.69 (2.83)
Se is the elasticity’s standard error 22.76 (28.32)
Panel =1, if estimate relates to panel data with time-series as the base 0.45 (0.50)
Cross =1, if estimate relates to cross-sectional data with time-series as the base 0.13 (0.34)
Adults =1, if estimate relates to young adults (20-24) rather than teenagers (16-19) 0.14 (0.35)
Male =1, if estimate relates to male employees 0.07 (0.26)
Non-white =1, if estimate relates to non-white employees 0.05 (0.22)
Region =1, if estimate relates to region specific data 0.10 (0.30)
Lag =1, if estimate relates to a lagged minimum wage effect 0.13 (0.34)
Hours =1, if the dependent variable is hours worked 0.07 (0.25)
Double =1, if estimate comes from a double log specification 0.42 (0.49)
AveYear is the average year of the data used, with 2000 as the base year -19.17 (11.90)
Agriculture =1, if estimates are for the agriculture industry 0.01 (0.11)
Retail =1, if estimates are for the retail industry 0.08 (0.27)
Food =1, if estimates are for the food industry 0.13 (0.34)
Time =1, if time trend is included 0.37 (0.48)
Yeareffect =1, if year specific fixed effects are used 0.30 (0.46)
Regioneffect =1, if region/State fixed effects are used 0.34 (0.47)
Un =1, if a model includes unemployment 0.56 (0.50)
School =1, if model includes a schooling variable 0.15 (0.35)
Kaitz =1, if the Kaitz measure of the minimum wage is used 0.40 (0.49)
Dummy =1, if a dummy variable measure of the minimum wage is used 0.17 (0.38)
Published =1, if the estimate comes from a published study 0.85 (0.35)
22 Z-var’s + 22 K-var’s = 44 Var’s
Table 3: General-to-Specific MRA Minimum-Wage
Elasticity
Variables: Cluster-Robust FE-Panel
Genuine empirical effects (Z-variables)Intercept ( β ) 0.120 (4.39) 0.102 (6.04)Panel -0.182 (-4.72) -0.149 (-10.5)Double 0.064 (3.20) 0.041 (5.42)Region 0.040 (0.92) 0.090 (6.34)Adult 0.024 (2.68) 0.021 (3.72)Lag 0.026 (1.60) 0.010 (1.59)AveYear 0.004 (4.34) 0.003 (6.38)Un -0.042 (-3.04) -0.042 (-5.79)Kaitz 0.052 (3.06) 0.032 (3.88)Yeareffect 0.069 (1.98) 0.067 (7.44)Published -0.041 (-2.69) -0.037 (-4.89)Time -0.022 (-2.08) -0.017 (-2.46)
Publication bias (K-variables)Se ( β0 ) -0.359 (-0.11) -1.374 (-5.94)Double x Se -1.482 (-3.23) -1.073 (-3.90)Un x Se -0.840 (-1.87) 1.164 (3.08)
Heterogeneity
Publication Selection
Doucouliagos and Stanley (2009), British Journal of
Industrial Relations.
‘Prediction’ and Implications
• Once publication selection is accommodated, No evidence of anadverse employment effect remains.
• Substituting any defensible notion of ‘Best Practice Research’ into these estimated MRA coefficients (Table 3) finds: Nosupport for a practically significantadverse employment effect!
Robustness Checks
• Many sets of robustness checks were investigated. • robust regressions• unbalanced panel MRA and cluster-robust
standard errors to control for potential within study dependence
• the ‘best-set’ (i.e., one estimate per study)• Card and Krueger’s (1995) data• omitting all positive elasticities
{All confirm an absence of an adverse employment effect.}
Replicated by 2 Independent Teams &
A Meta-Analysis of UK’s Minimum Wage:• Followed MAER-NET (2013) guidelines
• Comprehensive search, 2 coders, etc.• Confirmed our US findings
• No evidence of adverse employment• But no evidence of publication bias in UK{Leonard et al. (2014), “Does the UK minimum wage reduce employment? BJIR}
Policy Implications
• 2013 US Economic Report of the President quoted us and reproduced our funnel graph.
• President Obama proposed an increase in the US minimum wage—from $7.25 to $9/hour.
• Many US cities and states (including Arkansas) have already raised their minimum wage.
• Minimum wage will be a campaign issue in the next US election.
• UK’s Low Pay Commission has funded an update of our meta-analysis.
Doucouliagos, Haman, and Stanley (2012): Industrial Relations, 51: 670-703.
Pay for Performance and Corporate Governance Reform
Too often, executive compensation in the U.S. is ridiculously out of line with performance. That won't change, moreover, because the deck is stacked against investors when it comes to the CEO's pay. . . Getting fired can produce a particularly bountiful payday for a CEO. Indeed, he can “earn” more in that single day, while cleaning out his desk, than an American worker earns in a lifetime of cleaning toilets. Forget the old maxim about nothing succeeding like success: Today, in the executive suite, the all-too-prevalent rule is that nothing succeeds like failure.
− Warren E. Buffett, Berkshire Hathaway Report, February 28, 2006.
US corporate scandals, e.g., Bernie Ebbers (WorldCom);
Ken Lay and Jeff Skillings (ENRON), . . . . . . . The 2007 global financial meltdown. . . Large bonuses at Wall-Street banks { often the
same ones that caused the 2007 financial crisis and had to be bailed out}
Predictably, media reports of egregious corporate excesses lead to public calls for greater corporate regulation and reform.
Those Buzzards. . . . .
‘Comply or Explain’ “The essence of the principle is that compliance with the codes is not mandatory, but that disclosure relating to compliance is.”
– (MacNeil and Li; 2006, p.486)
Every few years, the UK has adopted new corporate governance reforms, attempting to strengthen the link between CEO pay and corporate performance and thereby to protect shareholders. Cadbury Report (1992); the Greenbury Report (1995); the Combined
Code of Corporate Governance (1998); the revised Combined Code of Corporate Governance (2003); Companies Act 2006; . . .
CEO remuneration committees are now ‘required’ to include independent non-executive directors
shareholders get to vote on directors’ pay at annual meetings
FAT-PET-MRA-WLS– Eq. (5)Table 4: MRA Tests of PB & Genuine Effect
MRA equation (1): Dependent Variable = r (partial correlations) CEO Pay-
Performance Intercept (β) 0.077(8.09)*
SEi (β0) 0.097(0.33)*
Simple Mean 0.093
WLS weighted average
0.079
Top10 0. 078
PEESE 0.077
n 511
*t-values are reported in parenthesis are from cluster-robust standard errors.
Table 5: Selected MRA Results There is evidence that:
‘Comply and Explain’ has been effective.
The pay-performance link is declining over time.
Higher economic freedom leads to stronger pay-performance linkage.
{RE: greater reliance on markets should lead to stronger pay-performance links}
The ways in which both pay and performance are measured matter.
Bad News: ‘Size’ matters; performance: not so much.
Variable Base model General to specific
Constant 25.337 (2.87) 8.177 (2.85) AverageYear -0.013(-2.84) -0.004 (-2.79) Cadbury 0.058 (3.69) 0.037 (2.43) CombinedCode 0.079 (2.16) - EcoFree 0.102 (1.98) - Cash - -0.082 (-2.46) Changepay - -0.053 (-3.12) Returns - 0.058 (3.41) ROE - 0.205 (2.52) Sales - 0.143 (22.98) Lagperform - -0.032 (-2.62) Panel - -0.048 (-2.11) Education - -0.105 (-1.72) Lagpay - 0.065 (3.78) Time effects - -0.025 (-1.75) Composition - -0.055 (-2.19) Adjusted R2 0.38 0.82
A Medical Application:Nicotine Replacement Therapy (NRT)
Stead et al. (2012) The Cochrane Library:
• 122 placebo-controlled clinical trials (RCTs)• large effects from NRT {quitting by 50-70%}
• coded for ‘risks of bias’ for each RCT• did not control for publication bias.
Our MRAs that control for both Stead et al.’s (2012) risks of bias and publication bias find :
No evidence a NRT Effect
Funnel Graph: High quality NRT trials
0
1
2
3
4
5
6
7
8
9Pr
ecis
ion
(1/s
)
-.25 0 .25 .5 .75 1 1.25 1.5 1.75 2 2.25lnRR
Table 6: MRA of NRT trialsVariable/Estimate 1: Q2 2: Q3 3: Q4 4: Full Sample
Intercept: β̂ -0.02 (-0.23)
0.04 (0.30)
-0.05 (-0.25) -0.09 (-0.64)
Sei : 0β̂ 1.68* (4.96)
1.52* (2.76)
1.84* (2.20) 2.04* (3.89)
Riski: 1α̂ — — — 0.11* (2.93)
si ·Riski : 2α̂ — — — -0.39* (-2.64)
Sample size n 46 23 12 121 Adj R2 25.2% 23.1% 25.8% 12.6% Homogeneity: Q 36.8 19.1 11.2 145.1*
Note: Q2, Q3, and Q4 represent decreasing risk of bias, defined by Stead et al. (2012)
Funnel Symmetry Exceptions{aside from pub’bias of course}
Very large, partial correlation coefficients. Estimating AR(1) coefficients, Time Series If unexplained systematic heterogeneity is skewed and
correlated with SE. Non-Market environmental values when calculated from
an estimated price coefficient. {and any other nonlinear transformation of an regression coefficient}
Due to power analysis? Large effects might be associated with smaller samples; hence smaller precision?
Publication Selection Paradox• Many economists have difficulty seeing that the
suppression of positive price elasticities (or negative VSLs) is somehow ‘biased.’
• It may be individually rational for economists to ‘validate’ their estimate– i.e., to suppress positive price elasticities or negative VSL.
• However, when most researchers suppress these ‘wrong’ signs, our aggregate knowledge worsens.
• Fallacy of Composition {ecological fallacy}
ReferencesCard, D. and Krueger, A.B. 1995. Time-series minimum-wage studies: A meta-analysis, American Economic Review 85: 238-243.Demsetz, H. (1974) ‘Two systems of belief about monopoly’, in H.J. Goldschmid, H.M. Mann and Weston, J. F. (eds.) Industrial
Concentration: The New Learning, Boston: Little, Brown and Company, pp.164–84.Doucouliagos, C.(H) and Stanley, T.D. 2009. Publication selection bias in minimum-wage research? A meta-regression analysis,”
British Journal of Industrial Relations 47: 406-29. Doucouliagos, C.(H), Stanley, T.D. and Giles, M. 2012. ‘Are Estimates of the Value of a Statistical Life Exaggerated?’ Journal of
Health Economics, 31: 197-206..Egger, M., Smith, G.D., Scheider, M., and Minder, C. 1997. Bias in meta-analysis detected by a simple, graphical test, British
Medical Journal 316: 629-34.Glass, G.V. 1976. Primary, secondary, and meta-analysis of research. Educational Researcher 5: 3-8.Stanley, T.D. 2001. Wheat from chaff: Meta-analysis as quantitative literature review. Journal of Economic Perspectives 15: 131-
50.Stanley, T.D. 2004. ‘Does unemployment hysteresis falsify the natural rate hypothesis? A meta-regression analysis’, Journal of
Economic Surveys, 18: 589-612. Stanley, T.D., 2005a. Beyond publication bias, Journal of Economic Surveys 19: 309-45.Stanley, T.D. 2005b. ‘Integrating the empirical tests of the natural rate hypothesis: A meta-regression analysis’, Kyklos, 58: 611-
34. Stanley, T.D. 2008. Meta-regression methods for detecting and estimating empirical effect in the presence of publication bias.
Oxford Bulletin of Economics and Statistics 70:103-127.Stanley, T.D., Jarrell, S.B., 1989. Meta-regression analysis: A quantitative method of literature surveys. Journal of Economic
Surveys 3: 54-67.Stanley, T.D., and Doucouliagos, H(C) 2010. Picture this: A simple graph that reveals much ado about research. Journal of
Economic Surveys 24: 170-191.