Individuals and Organizations as Sources of State Effectivenessand Consequences for Policy Design

Michael BestColumbia

Jonas HjortColumbia

David SzakonyiGeorge Washington

Public Sector Efficiency and Effectiveness:Implications for Developing and Transition Countries,

SITE-Stockholm School of EconomicsDecember 15 2017

IntroductionI Same policy → different outcomes in different settings

I VAT in low/high income countries

I NREGA across India

I Post office returning mail

I Bureaucrats and public organizations different across settings

I How much of variation in policy outcomes due to bureaucracy?

I What do effective bureaucracies do differently?

I Towards: How does optimal policy depend on who implements it?

I We investigate these questions in the Russian public sector

1 / 14

IntroductionI Same policy → different outcomes in different settings

I VAT in low/high income countries

I NREGA across India

I Post office returning mail

I Bureaucrats and public organizations different across settings

I How much of variation in policy outcomes due to bureaucracy?

I What do effective bureaucracies do differently?

I Towards: How does optimal policy depend on who implements it?

I We investigate these questions in the Russian public sector

1 / 14

IntroductionI Same policy → different outcomes in different settings

I VAT in low/high income countries

I NREGA across India

I Post office returning mail

I Bureaucrats and public organizations different across settings

I How much of variation in policy outcomes due to bureaucracy?

I What do effective bureaucracies do differently?

I Towards: How does optimal policy depend on who implements it?

I We investigate these questions in the Russian public sector

1 / 14

IntroductionI Same policy → different outcomes in different settings

I VAT in low/high income countries

I NREGA across India

I Post office returning mail

I Bureaucrats and public organizations different across settings

I How much of variation in policy outcomes due to bureaucracy?

I What do effective bureaucracies do differently?

I Towards: How does optimal policy depend on who implements it?

I We investigate these questions in the Russian public sector

1 / 14

IntroductionI Same policy → different outcomes in different settings

I VAT in low/high income countries

I NREGA across India

I Post office returning mail

I Bureaucrats and public organizations different across settings

I How much of variation in policy outcomes due to bureaucracy?

I What do effective bureaucracies do differently?

I Towards: How does optimal policy depend on who implements it?

I We investigate these questions in the Russian public sector

1 / 14

IntroductionI Same policy → different outcomes in different settings

I VAT in low/high income countries

I NREGA across India

I Post office returning mail

I Bureaucrats and public organizations different across settings

I How much of variation in policy outcomes due to bureaucracy?

I What do effective bureaucracies do differently?

I Towards: How does optimal policy depend on who implements it?

I We investigate these questions in the Russian public sector

1 / 14

IntroductionI Same policy → different outcomes in different settings

I VAT in low/high income countries

I NREGA across India

I Post office returning mail

I Bureaucrats and public organizations different across settings

I How much of variation in policy outcomes due to bureaucracy?

I What do effective bureaucracies do differently?

I Towards: How does optimal policy depend on who implements it?

I We investigate these questions in the Russian public sector

1 / 14

IntroductionI Same policy → different outcomes in different settings

I VAT in low/high income countries

I NREGA across India

I Post office returning mail

I Bureaucrats and public organizations different across settings

I How much of variation in policy outcomes due to bureaucracy?

I What do effective bureaucracies do differently?

I Towards: How does optimal policy depend on who implements it?

I We investigate these questions in the Russian public sector

1 / 14

IntroductionI Same policy → different outcomes in different settings

I VAT in low/high income countries

I NREGA across India

I Post office returning mail

I Bureaucrats and public organizations different across settings

I How much of variation in policy outcomes due to bureaucracy?

I What do effective bureaucracies do differently?

I Towards: How does optimal policy depend on who implements it?

I We investigate these questions in the Russian public sector

1 / 14

Procurement in RussiaI Decentralized procurement with centralized rules. 104,000

Federal, regional, municipal public bodies procure independently.

I No centralized civil service, procurement officers hired much likeprivate sector.

I Key: Organizations observed working with multiple bureaucrats &bureaucrats observed working with multiple organizations.

I Majority (52%) of purchases by electronic descending auction.

I Detailed data on all procurement available in online procurementregister – the Unified Register of Federal and Municipal Contracts

I We collect data on all auction requests, review protocols, bids,and final contracts 2011–2015

2 / 14

Procurement in RussiaI Decentralized procurement with centralized rules. 104,000

Federal, regional, municipal public bodies procure independently.

I No centralized civil service, procurement officers hired much likeprivate sector.

I Key: Organizations observed working with multiple bureaucrats &bureaucrats observed working with multiple organizations.

I Majority (52%) of purchases by electronic descending auction.

I Detailed data on all procurement available in online procurementregister – the Unified Register of Federal and Municipal Contracts

I We collect data on all auction requests, review protocols, bids,and final contracts 2011–2015

2 / 14

Procurement in RussiaI Decentralized procurement with centralized rules. 104,000

Federal, regional, municipal public bodies procure independently.

I No centralized civil service, procurement officers hired much likeprivate sector.

I Key: Organizations observed working with multiple bureaucrats &bureaucrats observed working with multiple organizations.

I Majority (52%) of purchases by electronic descending auction.

I Detailed data on all procurement available in online procurementregister – the Unified Register of Federal and Municipal Contracts

I We collect data on all auction requests, review protocols, bids,and final contracts 2011–2015

2 / 14

Procurement in RussiaI Decentralized procurement with centralized rules. 104,000

Federal, regional, municipal public bodies procure independently.

I No centralized civil service, procurement officers hired much likeprivate sector.

I Key: Organizations observed working with multiple bureaucrats &bureaucrats observed working with multiple organizations.

I Majority (52%) of purchases by electronic descending auction.

I Detailed data on all procurement available in online procurementregister – the Unified Register of Federal and Municipal Contracts

I We collect data on all auction requests, review protocols, bids,and final contracts 2011–2015

2 / 14

Procurement in RussiaI Decentralized procurement with centralized rules. 104,000

Federal, regional, municipal public bodies procure independently.

I No centralized civil service, procurement officers hired much likeprivate sector.

I Key: Organizations observed working with multiple bureaucrats &bureaucrats observed working with multiple organizations.

I Majority (52%) of purchases by electronic descending auction.

I Detailed data on all procurement available in online procurementregister – the Unified Register of Federal and Municipal Contracts

I We collect data on all auction requests, review protocols, bids,and final contracts 2011–2015

2 / 14

What Happens When Bureaucrats Move?

0

1

2

3

−1 0 1 2Time (0 = last day with old bureaucrat)

Sta

ndar

dize

d P

rice

Res

idua

ls

Trajectory 1 to 1

4 to 4

3 / 14

What Happens When Bureaucrats Move?

0

1

2

3

−1 0 1 2Time (0 = last day with old bureaucrat)

Sta

ndar

dize

d P

rice

Res

idua

ls

Trajectory 1 to 1

1 to 4

4 to 1

4 to 4

4 / 14

Decomposing Variation in Prices PaidI Model outcome yi ∈ {log (pi) , ni} for item i purchased for

organization j by bureaucrat b (i, j) as

yi = Xiβ + α̃b(i,j) + ψ̃j + εi

I Xi includes log quantity, good FEs, month FEs, size, region

I Goal: Estimate magnitude of Var (α̃) and Var(ψ̃

)I 4 Challenges

1. Like-for-like comparisons: quality -adjusted prices2. Effects only identified within “connected sets” linked by mobility3. Causal identification requires “exogenous mobility”4. Sampling error inflates variances

5 / 14

Decomposing Variation in Prices PaidI Model outcome yi ∈ {log (pi) , ni} for item i purchased for

organization j by bureaucrat b (i, j) as

yi = Xiβ + α̃b(i,j) + ψ̃j + εi

I Xi includes log quantity, good FEs, month FEs, size, region

I Goal: Estimate magnitude of Var (α̃) and Var(ψ̃

)

I 4 Challenges

1. Like-for-like comparisons: quality -adjusted prices2. Effects only identified within “connected sets” linked by mobility3. Causal identification requires “exogenous mobility”4. Sampling error inflates variances

5 / 14

Decomposing Variation in Prices PaidI Model outcome yi ∈ {log (pi) , ni} for item i purchased for

organization j by bureaucrat b (i, j) as

yi = Xiβ + α̃b(i,j) + ψ̃j + εi

I Xi includes log quantity, good FEs, month FEs, size, region

I Goal: Estimate magnitude of Var (α̃) and Var(ψ̃

)I 4 Challenges

1. Like-for-like comparisons: quality -adjusted prices2. Effects only identified within “connected sets” linked by mobility3. Causal identification requires “exogenous mobility”4. Sampling error inflates variances

5 / 14

Decomposing Variation in Prices PaidI Model outcome yi ∈ {log (pi) , ni} for item i purchased for

organization j by bureaucrat b (i, j) as

yi = Xiβ + α̃b(i,j) + ψ̃j + εi

I Xi includes log quantity, good FEs, month FEs, size, region

I Goal: Estimate magnitude of Var (α̃) and Var(ψ̃

)I 4 Challenges

1. Like-for-like comparisons: quality -adjusted prices

2. Effects only identified within “connected sets” linked by mobility3. Causal identification requires “exogenous mobility”4. Sampling error inflates variances

5 / 14

Decomposing Variation in Prices PaidI Model outcome yi ∈ {log (pi) , ni} for item i purchased for

organization j by bureaucrat b (i, j) as

yi = Xiβ + α̃b(i,j) + ψ̃j + εi

I Xi includes log quantity, good FEs, month FEs, size, region

I Goal: Estimate magnitude of Var (α̃) and Var(ψ̃

)I 4 Challenges

1. Like-for-like comparisons: quality -adjusted prices2. Effects only identified within “connected sets” linked by mobility

3. Causal identification requires “exogenous mobility”4. Sampling error inflates variances

5 / 14

Decomposing Variation in Prices PaidI Model outcome yi ∈ {log (pi) , ni} for item i purchased for

organization j by bureaucrat b (i, j) as

yi = Xiβ + α̃b(i,j) + ψ̃j + εi

I Xi includes log quantity, good FEs, month FEs, size, region

I Goal: Estimate magnitude of Var (α̃) and Var(ψ̃

)I 4 Challenges

1. Like-for-like comparisons: quality -adjusted prices2. Effects only identified within “connected sets” linked by mobility3. Causal identification requires “exogenous mobility”

4. Sampling error inflates variances

5 / 14

Decomposing Variation in Prices PaidI Model outcome yi ∈ {log (pi) , ni} for item i purchased for

organization j by bureaucrat b (i, j) as

yi = Xiβ + α̃b(i,j) + ψ̃j + εi

I Xi includes log quantity, good FEs, month FEs, size, region

I Goal: Estimate magnitude of Var (α̃) and Var(ψ̃

)I 4 Challenges

1. Like-for-like comparisons: quality -adjusted prices2. Effects only identified within “connected sets” linked by mobility3. Causal identification requires “exogenous mobility”4. Sampling error inflates variances

5 / 14

Decomposing Variation in Prices Paid

Yi =Xiβ + αb(i,j) + ψj + γs(b,j) + εi

Var (Yi) =Var(αb(i,j)

)+ Var (ψj) + . . .

Prices (P) (s.e.) Participation (N) (s.e.)(1) (2) (3) (4)

s.d. of Y 2.417 1.355s.d. of Y | good, month 1.646 1.241

s.d. of Bureaucrat Effects 1.031 (0.0462) 0.919 (0.0418)s.d. of Organization Effects 1.068 (0.0496) 0.888 (0.0468)s.d. of Total Bur + Org Effects 1.036 (0.00126) 0.710 (0.00358)

Adjusted R-squared 0.955 0.837Sample Size 11,228,122 11,228,122

6 / 14

Crude Counterfactual: Improving Bureaucrats

7 / 14

Counterfactual: Improving Bureaucrats & Organizations

8 / 14

Correlates of Bureaucrat/Organization Effectiveness

I What do good and bad bureaucrats and organizations dodifferently?

I What attributes do good and bad bureaucrats have?

I Using detailed data on the levers they control at the “qualificationstage”, the “auction stage”, and “the contracting stage” – andhow firms respond to their behaviors

I Large number of rhs variables. For variable selection present

1. Pairwise regression coefficients (standardized rhs vars)

2. Post-LASSO coefficients

9 / 14

Correlates of Bureaucrat/Organization Effectiveness

I What do good and bad bureaucrats and organizations dodifferently?

I What attributes do good and bad bureaucrats have?

I Using detailed data on the levers they control at the “qualificationstage”, the “auction stage”, and “the contracting stage” – andhow firms respond to their behaviors

I Large number of rhs variables. For variable selection present

1. Pairwise regression coefficients (standardized rhs vars)

2. Post-LASSO coefficients

9 / 14

Correlates of Bureaucrat/Organization Effectiveness

I What do good and bad bureaucrats and organizations dodifferently?

I What attributes do good and bad bureaucrats have?

I Using detailed data on the levers they control at the “qualificationstage”, the “auction stage”, and “the contracting stage” – andhow firms respond to their behaviors

I Large number of rhs variables. For variable selection present1. Pairwise regression coefficients (standardized rhs vars)

2. Post-LASSO coefficients

9 / 14

Pairwise Regressions Post−LASSO Regression

−0.10 −0.05 0.00 0.05 0.10 −0.10 −0.05 0.00 0.05 0.10

Supplier Turnover

Supplier Assets

Supplier log Employees

Number of Contract Revisions

Average of Losing Bids / Winning Bid

1[Winner is From Same Region]

1[Auction Held]

Number of Items Purchased

Success Rate

In−house Bureaucrat

Admission Rate to Auction

Number of Applicants

Time to Prepare Documents

Deposit / Reserve price

Number of Products

Lot Size

Standardized Coefficient

Var

iabl

e

10 / 14

Bid Preferences for Domestic Products

I Motivation: classic mercantilist “support local business”

I Each spring presidency issues list of goods receiving preferences

I Winner supplying foreign good receives 85% of winning bid

I Winner supplying domestic good receives 100% of winning bid

I Sources of time × product variation:

I Intra-year: Preferences switch on in late spring, off on 12/31

I Across goods: Different goods on list each year

I ⇒ Difference-in-differences strategy:

I ⇒ Triple-differences: Interact with α̂b, ψ̂j from regular auctions

11 / 14

Bid Preferences for Domestic Products

I Motivation: classic mercantilist “support local business”

I Each spring presidency issues list of goods receiving preferences

I Winner supplying foreign good receives 85% of winning bid

I Winner supplying domestic good receives 100% of winning bid

I Sources of time × product variation:I Intra-year: Preferences switch on in late spring, off on 12/31

I Across goods: Different goods on list each year

I ⇒ Difference-in-differences strategy:

I ⇒ Triple-differences: Interact with α̂b, ψ̂j from regular auctions

11 / 14

Bid Preferences for Domestic Products

I Motivation: classic mercantilist “support local business”

I Each spring presidency issues list of goods receiving preferences

I Winner supplying foreign good receives 85% of winning bid

I Winner supplying domestic good receives 100% of winning bid

I Sources of time × product variation:I Intra-year: Preferences switch on in late spring, off on 12/31

I Across goods: Different goods on list each year

I ⇒ Difference-in-differences strategy:

I ⇒ Triple-differences: Interact with α̂b, ψ̂j from regular auctions

11 / 14

Average Policy Impact: Graphical Analysis

2011

law

com

es in

to E

ffect

2013

law

com

es in

to E

ffect

2012

law

com

es in

to E

ffect

2014

law

com

es in

to E

ffect

−0.4

−0.2

0

0.2

0.4

Pric

e R

esid

uals

Treatment Group

Not Preferenced

Preferenced

−0.5 SD[−0.70]

0

0.5 SD[0.70]

2011 2012 2013 2014 2015

Time

Pric

e R

esid

uals

Difference between Groups: Preferenced − Unpreferenced

12 / 14

Policy Impact Depends on Bureaucratic Effectiveness

●

●

●

● ●

●

●

●●

−0.4

−0.2

0.0

−1 0 1Bureaucrat Effectiveness

Trea

tmen

t Effe

ct (

rel.

to d

ecile

1) Panel A: Heterogeneity in Effect on Price by Bureaucrat Effectiveness

●

●

● ●●

● ●●

●

−0.4

−0.2

0.0

−1 0 1Organization Effectiveness

Trea

tmen

t Effe

ct (

rel.

to d

ecile

1) Panel B: Heterogeneity in Effect on Price by Organization Effectiveness

13 / 14

ConclusionI Procurement as window into determinants of state’s effectiveness

I Weber (1922) “Bureaucracy develops the more perfectly, the moreit is ‘dehumanized’... The individual bureaucrat cannot squirm outof the apparatus into which he has been harnessed”

I ∼ 60% of variation in prices paid due to bureaucrats & publicorganizations. Far from Weberian ideal.

I What do effective bureaucrats do?

I Attract entry by diverse firms: ⇒ policy to encourage participation

I Experienced/in-house burs better: ⇒ training/end-user feedback?

I Policy responses to limited effectiveness

I Optimal bid preferences decreasing in effectiveness

I Tailoring policy to capacity substitute for raising capacity

14 / 14

ConclusionI Procurement as window into determinants of state’s effectiveness

I Weber (1922) “Bureaucracy develops the more perfectly, the moreit is ‘dehumanized’... The individual bureaucrat cannot squirm outof the apparatus into which he has been harnessed”

I ∼ 60% of variation in prices paid due to bureaucrats & publicorganizations. Far from Weberian ideal.

I What do effective bureaucrats do?

I Attract entry by diverse firms: ⇒ policy to encourage participation

I Experienced/in-house burs better: ⇒ training/end-user feedback?

I Policy responses to limited effectiveness

I Optimal bid preferences decreasing in effectiveness

I Tailoring policy to capacity substitute for raising capacity

14 / 14

ConclusionI Procurement as window into determinants of state’s effectiveness

I Weber (1922) “Bureaucracy develops the more perfectly, the moreit is ‘dehumanized’... The individual bureaucrat cannot squirm outof the apparatus into which he has been harnessed”

I ∼ 60% of variation in prices paid due to bureaucrats & publicorganizations. Far from Weberian ideal.

I What do effective bureaucrats do?

I Attract entry by diverse firms: ⇒ policy to encourage participation

I Experienced/in-house burs better: ⇒ training/end-user feedback?

I Policy responses to limited effectiveness

I Optimal bid preferences decreasing in effectiveness

I Tailoring policy to capacity substitute for raising capacity

14 / 14

ConclusionI Procurement as window into determinants of state’s effectiveness

I Weber (1922) “Bureaucracy develops the more perfectly, the moreit is ‘dehumanized’... The individual bureaucrat cannot squirm outof the apparatus into which he has been harnessed”

I ∼ 60% of variation in prices paid due to bureaucrats & publicorganizations. Far from Weberian ideal.

I What do effective bureaucrats do?I Attract entry by diverse firms: ⇒ policy to encourage participation

I Experienced/in-house burs better: ⇒ training/end-user feedback?

I Policy responses to limited effectiveness

I Optimal bid preferences decreasing in effectiveness

I Tailoring policy to capacity substitute for raising capacity

14 / 14

ConclusionI Procurement as window into determinants of state’s effectiveness

I Weber (1922) “Bureaucracy develops the more perfectly, the moreit is ‘dehumanized’... The individual bureaucrat cannot squirm outof the apparatus into which he has been harnessed”

I ∼ 60% of variation in prices paid due to bureaucrats & publicorganizations. Far from Weberian ideal.

I What do effective bureaucrats do?I Attract entry by diverse firms: ⇒ policy to encourage participation

I Experienced/in-house burs better: ⇒ training/end-user feedback?

I Policy responses to limited effectivenessI Optimal bid preferences decreasing in effectiveness

I Tailoring policy to capacity substitute for raising capacity

14 / 14