Impact Evaluation of a Systemic Investor Response Mechanism in...

1

Impact Evaluation of a Systemic Investor Response Mechanism in the Business Ombudsman of Georgia

Concept Note (Revised Version)

06/15/2016

Abstract A good investment climate and market-supporting government institutions are considered to be

fundamental to private sector development. However, there is little empirical evidence on the effect of

specific government institutions on firms’ perceptions of the business climate, their investment choices,

and firm growth. Governments around the world are keen to become more “business friendly,” but

have little guidance on specific interventions that will promote firm investment and growth.

We hope to contribute to the body of knowledge on how institutions affect decisions made by firms by

evaluating one specific government institutional innovation, the Business Ombudsman (BO), in Georgia.

The BO is an independent office within the government that firms can turn to if they have disputes with

other parts of the government bureaucracy. The BO helps firms by providing legal advice, written

opinions and by advocating for firms within the government ministries.

We will induce exogenous variation across firms in the usage of the BO's services through a randomized

information intervention that targets firms that are in the process of submitting an appeal to the

Georgia Revenue Service or the Tbilisi Municipality. Furthermore, in order to better understand what

the BO can provide beyond the type of services provided by law firms in the private market, we will

randomize firms into two different groups that receive different services from the BO. One group will

receive all of BO’s services, the “Business as Usual” treatment, including the informal advocacy the BO

does for firms within the government agencies while another group will receive a limited set of services,

the “Legal Services” treatment, that does not include advocacy.

Using the exogenous variation in usage we look to estimate the effect of the BO’s services on three

types of outcomes - (i) the direct effect the BO has on the issues that the firms had with the

government, (ii) the effect on firms’ perception of the business climate, and (iii) the effect on firms’

investments, growth and future tax payments. Furthermore, we will use the second layer of

randomization of the BO’s services to answer which of the BO’s activities has the largest impact on these

outcomes.

Competitiveness Policy Evaluation Lab

2

Table of Contents Abstract ......................................................................................................................................................... 1

Table of Contents .......................................................................................................................................... 2

1. Background ........................................................................................................................................... 3

2. Intervention to be evaluated ................................................................................................................ 4

3. Theory of change ................................................................................................................................ 10

4. Literature ............................................................................................................................................ 13

5. Hypotheses and Evaluation Questions ............................................................................................... 15

6. Evaluation Design and Sampling Strategy .......................................................................................... 15

7. Data Collection ................................................................................................................................... 19

8. Data Processing and Analysis .............................................................................................................. 21

9. Study Limitations and Risks ................................................................................................................ 23

10. Policy Relevance and Impact .............................................................................................................. 25

11. Dissemination Plan ............................................................................................................................. 27

12. Milestones, Deliverables, and Timeline .............................................................................................. 28

13. References .......................................................................................................................................... 28


3

1. Background

Country Context

Following the collapse of the Soviet Union, Georgia was among the poorest-performing transition

economies in the region, thanks in part to conflict and governance issues. By 2003, real GDP per capita

had recovered to only about 57 percent of its 1990 level.1 Starting in 2004, the Government of Georgia

undertook a number of reforms to improve the business environment, infrastructure quality and the

state of public finance, and to reduce trade barriers. Between 2004 and 2008, Georgia's rank on the

Doing Business indicators improved from 112 to 18, reflecting these efforts. These reforms are also

believed to have stimulated investment, with FDI inflows increasing from US$109m in 2001 to about

US$2b in 2007, amounting to nearly 16.5 percent of GDP. During this period Georgia also managed to

successfully combat corruption and in Transparency International’s Global Corruption Perceptions Index

it went from ranking 85 out of 102 in 2002 to ranking 48 out of 167 in 2015. This is the 4th highest

ranking among the post-Soviet states and better than some EU members such as Hungary, Greece and

Italy.

The twin shocks in 2008 of the global economic crisis and the war with Russia interrupted capital

inflows, with strong effects on private investment and growth – in 2009 GDP contracted by 3.8 percent.

The government was quick to implement a fiscal stimulus to support recovery, including a large public

investment package and increased social transfers. And growth has since recovered, averaging over 5

percent between 2010 and 2013.

However, the Government of Georgia remains concerned at the levels of private sector investment and

is particularly concerned about investors’ perceptions of uncertainty. Although FDI inflows had

recovered to $1.6 b in 2014, this remains short of the 2007 level. Growth too is short of the 10%

averaged between 2004 and 2008, with Georgia’s real GDP still below its 1989 level.

As a part of the activities intended to improve investor protection and confidence, and thus to increase

investment retained, the World Bank Group (WBG) is working with the Georgia Business Ombudsman

(BO) office to implement a Systemic Investor Response Mechanism (SIRM). The SIRM aims to improve

investment retention and investor confidence by,

i. Establishing or enhancing investor grievance mechanisms, and

ii. Developing tracking systems to allow governments to better respond to investor grievances.

These steps are expected to enable the government to respond to grievances in a transparent, fair, and

timely manner and therefore improve investor perceptions of investor protection, and prevent

escalation of grievances to investor-state disputes, which can have high direct (arbitration and

settlement) and indirect costs (damage to perception of investment climate).

The SIRM intervention proposed for evaluation is in the BO office, established by the Business

Ombudsman Law in 2011. The World Bank Group has been providing advice to the BO on various

1 Figures from the World Bank Databank.


4

implementation features to improve its ability to monitor grievances and coordinate with investors and

other government agencies. The intended outcomes of the SIRM are improved investor confidence,

increased investment retention and improved perceptions of Georgia’s investment climate. The impact

evaluation of the SIRM in Georgia will rigorously measure the impact of the BO’s office on these

outcomes and aim to uncover which of the BO’s activities achieve the greatest impact.

2. Intervention to be evaluated Consultation

In Nov 2015, the project team initiated discussions with the BO on a potential impact evaluation, and

together with the impact evaluation team, met with the BO in-person in Jan 2016 to discuss the value of

and get their buy-in for a prospective impact evaluation using a rigorous randomized controlled trial

(RCT) design. The Head of the BO, as well as its technical staff, expressed immediate support both

because it provides an opportunity to showcase the example of the BO to an international audience and

also because the impact evaluation can generate information that will allow the BO to improve the

effectiveness of its service.

The team also organized focus group discussions with three groups of firms, one group who had not

used the BO’s services, and two groups who had used the BO. In addition, the team met with the three

agencies that were the most common counterparties on grievances brought to the BO by firms – the

Georgia Revenue Service, the Tbilisi Municipality and the Georgia Department of Natural Resources. The

Georgia Revenue Service and Tbilisi City Municipality indicated verbally that they are willing to act as

implementing partners in this impact evaluation. This concept note is based on the outcomes of these

discussions.

In May 2016 the impact evaluation team, now including a Tbilisi-based field coordinator, returned to

Georgia for another round of meetings with the involved government agencies to work out details and

get a joint letter of commitment signed by all parties. Again the response from the BO, and from the

relevant units within the Georgia Revenue Service and the Tbilisi municipality was overwhelmingly

positive. We agreed on many of the specifics of the research design. We drafted a letter of commitment,

outlining the responsibilities of all of the involved government agencies, and this document was

reviewed and edited by each agency. The document has been translated into Georgian and will soon be

signed by representatives from all of the involved government agencies.

Overview of interventions

The initial intervention of the RCT will consist of a random assignment of firms to an information

treatment introducing the firms to the services of the BO office, with the expectation that this

intervention will substantially increase the number of firms that approach the BO. The second level of

randomization will determine what type of services the BO will provide to firms in the treatment group.

One treatment will be the full package of the BO’s services, the “Business as Usual” treatment, while

another treatment, the “Legal Services” treatment, will be limited so that it excludes the advocacy

services provided by the BO and therefore more resembles the services that could be provided by a

private law firm.


5

Our preliminary research shows that despite the fact that the BO’s services are free and sometimes

publicized on television, the usage of the BO’s services among firms was very limited. In total, only

around 330 firms (from a population of about 60,000) had used the BO’s services in 2014. Out of a

random sample of 800 medium to large firms surveyed by the IFC for an investor perception survey, only

16 said that they had ever interacted with the BO and only three had actually gotten help from the BO.

In focus group discussions with firms that had not used the BO’s services, it was clear that most firms

either did not know that the BO existed or did not know exactly what the BO did, and had therefore

never considered using the BO’s services. Although this is a problem that the BO should address, it also

provides us with a window of opportunity where a simple pamphlet and contact details to the BO’s

office could induce firms to contact the BO.

In order to maximize the likelihood that treated firms will end up using the BO’s services, one group of

firms that will be targeted are those that have just filed an appeal with the first level of the Georgia

Revenue Service’s dispute resolution cell. There are two main reasons for choosing this group of firms; i)

in 2014 47% of the cases that the BO received were related to tax or customs issues ii) the Georgia

Revenue Service was the most commonly mentioned government institution firms in the focus groups

had had problems with; this was true even for the group of firms that had never used the BO’s services.

The Georgia Revenue Service was often described by firms as overzealous when collecting tax and this

sentiment is also reflected in the high number of appeals to the revenue service’s dispute resolution

mechanism, the annual number of appeals is almost 10,000 from a population of approximately 60,000

active firms.2

The next most commonly mentioned government institution in the focus groups was the Tbilisi

Architectural Service, the department within the Tbilisi Municipality responsible for issuing building

permits. A second group of firms that we will target in the study are those that applied for but did not

receive a building permit at the Tbilisi Municipality. These firms have a choice of either registering a

formal dispute with the Legal Department of the Tbilisi Municipality or re-applying for a permit. Here,

the BO’s assistance may improve the likelihood that a firm obtains a construction permit, the speed with

which it obtains a permit, and the likelihood that the firm escalates the permit denial to the level of a

formal legal dispute, which is costly for both the firm and the agency. We also expect that receiving

assistance with construction permits will have a more direct effect on firms’ investments and growth

and therefore this group of firms will be crucial to study the impact of the BO on these outcomes. In

2015, there were approximately 2500 firms who applied for a construction permit but did not receive

one3.

Institutional set up of the dispute resolution process in the Georgia Revenue Service

Firms who want to appeal the Georgia Revenue Service’s tax decision can bring their case to the First

Level Appeals Committee. In this committee the case is first assessed by administrators and

subsequently presented to the committee to cast a vote on how to treat the appeal. However, due to

the heavy workload, not all cases are actually voted on by the committee - less complex cases are

2 Figures from the Department of Revenue and Geostat.

3 Figure from the Tbilisi Municipality website, where this data is publicly available.


6

instead decided on directly by the administrators. Both the committee and the administrators have the

options to i) fully accept the appeal by the firm and reduce the tax liability to what is claimed by the

firm, ii) partially accept the firms appeal and reduce the tax liability to somewhere in between the

original decision and what is claimed by the firm, or iii) reject the firm’s appeal. Even if the appeal is

rejected the committee has the power to waive any additional fees or penalties levied on the firm.

If the firm is not satisfied with the decision from the First Level Appeals Committee it can bring its case

to a Second Level Appeals Committee, located within the Ministry of Finance. Approximately 20%

(2,000) cases are brought to this committee each year. Here, the firm’s case is reviewed and prepared

by administrators again and voted on by a committee consisting of high level government

administrators appointed directly by the Prime Minister. Currently, one of the members of this

committee is the BO. The Second Level Appeals Committee has the same options as the first level one, it

can either fully or partially accept the appeal or reject it. If the firm is not satisfied with the decision

from the Second Level Appeals Committee, its only option is to take the case to court.

It is possible, but not a requirement, to seek help from the BO at any stage of the dispute resolution

process with the Georgia Revenue Service. When a firm applies for help, the BO reviews their case and

prepares a written legal opinion that can be used by firms to support their case in both committees, as

well as in court. In discussions with members of both of the appeals committees it was clear that the

BO’s opinion was something they took seriously since they respected the knowledge of the staff of the

BO’s office, some of whom had experience of working for the Georgia Revenue Service. However, it is

only in the second stage of the appeal process that the BO is a formal member of the committee and

even there the BO only has one vote on a committee of 15 members where the quorum for a decision is

eight members.

Intervention for firms with cases in the Georgia Revenue Service

Currently very few firms actually apply for the BO’s services when they make an appeal to the Georgia

Revenue Service. From the focus group discussions, we learned that the main reason for this is that

firms do not have a good understanding of what the BO does. The information intervention intends to

increase the knowledge among firms about the BO’s services right after they have submitted the appeal

at the First Level Appeals Committee.

Appeals can be submitted online or in person at one of the Georgia Revenue Service’s field offices. Our

intervention will have a person sitting in several of the revenue service’s field offices and, provided the

firm’s representative is willing to take part in the study, ask for the firm’s contact details. A Senior

Manager will then be contacted for an in-person meeting where the firms will provide formal consent of

taking part in the study. This meeting will be carried out by a survey firm who will also collect baseline

data on the firm. The main Georgia Revenue Service office in Tbilisi informed us that they get

approximately 25 “walk-in” applications per day. We will sample approximately 750 such firms for the

study and hence we expect the intervention to be active for approximately three months depending on

how many walk-in appeals are received and how many firms accept being part of the study.


7

Once the baseline data have been collected, sampled firms will be randomly assigned to the treatment

or the control group. Firms that are part of the treatment group will be given information about BO’s

services as well as the contact details of a person in the BO’s office. These firms will be encouraged to

seek help from the BO’s office for their tax appeal but also informed that they are not required to do so.

Firms in the control group will not be given any information.

Institutional set up of dispute resolution process in the Architectural Service of the Tbilisi Municipality

Building permits are issued by the Architectural Service department of the Tbilisi Municipality. The

Architectural Service receives permit applications electronically and carries out an assessment of

whether the planned building complies with requirements such as the building code and the zoning

plan. Some constructions also need special permissions, such as a clearance from the Cultural Heritage

Commission (for a construction in the ‘Cultural Heritage’ zone), or a special zoning permit (for an

exception to a zoning plan). If a proposed construction complies with all of these requirements, the

Architectural Service issues a building permit.

If a permit application is rejected, an applicant may either resubmit an application or register a formal

appeal with the Mayor’s Legal Department at the Tbilisi Municipality. If a firm resubmits a permit

application, the Architectural Service will review the application once again provided the application is

updated to include any documents or permissions missing in the original application. Alternatively, if the

firm appeals, the case is reviewed by the Mayor’s Legal Department and an oral hearing is carried out.

After having reviewed all the documentation and listened to both the firm and the Architectural

Service’s oral statements, the Mayor’s Legal Department make a ruling on whether the rejection

decision was a correct application of the law or if the rejection was incorrect, and if the rejection was

incorrect the firm can then be given a building permit. The Mayor’s Legal Department is the first and last

stage of the dispute resolution process at the Tbilisi Municipality. The only remaining recourse for firms

unsatisfied with the outcome of this process is to go to court.

Intervention for firms with cases in the Tbilisi Municipality

The list of permit applications rejected is updated daily on the website of the Architectural Service, and

includes the contact information of the applicant, the nature of the proposed construction and the

reason for the rejection of the permit. Using a set of simple rules developed in close collaboration with

the Architectural Service and the BO, we will screen the website daily for cases where the intervention

of the BO can make a difference. Broadly, these are cases where the applicant is a commercial entity,

the proposed construction is for commercial purposes, and the reason for rejection is not “technical” –

for instance, these are the types of cases where the cultural heritage commission refuses to approve the

construction, or the building is not approved for a special zoning permit. If we find that this approach to

screening is unsuccessful – i.e. the types of cases that we are sending to the BO are not cases where the

BO is able to provide much assistance, we plan to hire a legal expert to do this screening. Once we

identify a case where the BO’s assistance can potentially be useful, we will use the contact information

in the website to call the firm to set up an in-person meeting where, provided the firm formally agrees

to participate in the study, the baseline survey will be carried out by the Survey Firm. Sampled firms will

be randomly assigned to the treatment and control groups. Firms in the treatment group will be given

information about BO’s services, contact details of a person in the BO’s office, and encouraged to seek


8

help from the BO’s office but also informed that they are not required to do so. Firms in the control

group will not be given any information.

Based on a preliminary set of rules, we found that approximately 20 cases per-week passed the

screening stage described above. Given that we aim to sample around 250 firms from the Tbilisi

Municipality, we expect to run this intervention for about four months.

Second level of randomization: Variation in services provided by the BO

In order to differentiate between the effects of the different services that the BO provides we will

randomize one group of firms looking to make an appeal to the Georgia Revenue Service into a

treatment where they only receive a limited set of services. Broadly, the “Legal Services” are designed to

include the legal services provided by the BO but to exclude any advocacy services.

To classify the services into these two categories, we consulted with the BO to list all the services they

currently provide to firms. These are listed in the table below under “Full” or “Business-as-usual”

services. For the firms assigned to the “Legal services” treatment, we will ask the BO to withhold some

of these services - specifically, services where the BO communicates directly with the Revenue Service to

reach an agreement over a case. In practice, since the BO has a mandate to provide all of its services to

firms that request them, we only ask the BO to withhold these services where a firm in the “Legal

services” treatment does not explicitly request them.

The differentiation of services cannot be applied to firms that have a problem with the Tbilisi

Municipality since the BO has informed us that for these firms the advocacy service are the core service

that they provide and they cannot help these firms without providing this advocacy service.

Table 1: Differentiation of the BO’s Services

Full Services (“Business as Usual”) Limited Services (“Legal Services ”)

BO assesses a firm’s claim by conducting an

interview with the firm, reviewing relevant

documents and if needed communicating (by

phone or meeting) with the tax agency.

Same as business as usual.

The staff of the BO’s office assesses the validity of

the firm’s claim. If the BO deems that the firm’s

rights have been violated, the BO writes a legal

opinion to the firm.


If it is a complicated case, the BO communicates

directly with the tax agency and tries to come to

an agreement about what the correct ruling is.

This communication can be by phone calls or in-

person meetings.

If the firm insists that the BO’s office holds a

meeting with the Revenue Service, the BO’s

office may decide to do so but the BO will not

actively suggest this.


9

After the decision by the Revenue Service the

firm may consult with the BO if they should bring

the case to the Second Level Appeals Committee

at the Ministry of Finance.


If firm decides to take the case to the Second Level Appeals Committee at the Ministry of Finance

The BO may find it necessary to write a new

recommendation for the firm.


If it is a complicated case, the BO’s office

communicates directly with the Ministry of

Finance and tries to come to an agreement about

what the correct ruling is. This communication

can be by phone calls or in-person meetings.

If the firm insists that the BO’s office holds a

meeting with the Ministry of Finance, the BO’s

office may decide to do so, but the BO will not

actively suggest this.

In the committee meeting where the firm’s case

is decided on, the BO’s representative will make

an oral argument for the BO’s recommendation

to be followed.


If the firm decides to escalate the case to court the BO can no longer help the firm but the firm may

use the BO’s recommendation as evidence in court.

In order to implement this second level of randomization, the World Bank will provide financial support

to the BO office to employ two additional employees. These two employees will only provide the “Legal

Services”. In practice, the way firms are randomized into this treatment is through the content of the

information intervention. The firms that are assigned to the “Legal Services” treatment will be given the

contact details to one of the new employees in the BO’s office and these employees will be instructed to

only give the firms the services listed under the “Legal Services” column above.

Our hope is that this additional level of randomization will provide us with valuable information on

which of the BO’s activities is the most effective. If the effect from the “Legal Services” is the same as

the effect of the “Full” services treatment, we can rule out that advocacy is the main mechanism

through which the BO effects the government bureaucracy. As we explain below, most of the services

provided by the BO can be thought of as free legal services, similar to those provided by private law

firms. However, the advocacy services provided by the BO are harder to obtain in the private market,

and a feature of the institutional design of the BO in Georgia that might be of interest in other settings.

Differentiating the services allows us to estimate the additional effect of the advocacy above and

beyond the “Legal Services” that the BO provides.


10

3. Theory of change The starting point for the theory of change is the lack of awareness about the existence of the BO. Based

on this, the first intervention (box 1 in Figure 1) offers firms information on (i) the existence of the BO,

(ii) what type of cases they can approach the BO with, and (iii) how to approach the BO. We expect this

intervention will be effective in both raising awareness about the existence of the BO and increasing the

likelihood that these firms use the BO.

We hope to establish that the BO helps firms but also, through the second level of randomization,

investigate the relative importance of the different types of services the BO provides. If the firm is

sampled from the Georgia Revenue Service, the second randomization determines which of the BO’s

services will be provided to the firm:

a. Legal Services (box 2a. in Figure 1): Consists of the activities described in detail in the right hand

side of Table 1. The legal advice provided by the BO may complement (or substitute) for legal advice

procured by firms from lawyers or consulting firms. The legal opinion provided by the BO can be

used in the appeals process and as evidence in a potential court case.

b. Full Services (box 2b in Figure 1): As a government institution the BO’s office has the ability to

influence the tax authorities through informal advocacy. For example, the BO can potentially affect

the applications of laws that are not clearly defined. The BO is a politically well-connected individual

and we therefore expect him to be able to be an effective advocate for the firms that he is trying to

help.

Legal advice provided by a third party at an early stage of dispute resolution can be effective in reducing

dispute costs, length, and likelihood of going to court, as our literature review section describes. In the

case of the BO, the legal advice can (i) help firms better understand the strength of their case and affect

the probability of the case being escalated in the dispute resolution mechanism, and (ii) affect the

outcome of the appeals process because the BO are recognized as being experts in tax and permit

related matters, and because the BO has a seat on the second stage dispute resolution committee in the

Ministry of Finance. For firms that would have procured this legal advice from a third party of

comparable expertise in the absence of the BO, anyway, there may still be a reduction in the costs of

dispute resolution as the BO’s services are provided for free. As, as described above, the BO can

advocate directly for a firm in the second stage dispute resolution committee at the Ministry of Finance,

unlike a private law firm.

The full services (box 2b) are likely to amplify the effect of the legal advice alone – as described in Table

1, the BO often call officials in the Georgia Revenue Service to advocate on behalf of the firms. Since

some of the BO’s staff are ex-employees of the Georgia Revenue Service, and the BO is seen as a

politically strongly connected office, these advocacy efforts are likely to have an impact on all of the

outcomes illustrated in the theory of change below.

If the combination of services offered by the BO is indeed effective in reducing the number of disputes

that escalate to court, increasing the likelihood of dispute resolution and increasing the speed of dispute


11

resolution, we expect the firms who benefit to see strong income effects. An increase in the likelihood of

resolution in favor of the firm can reduce any penalties or other dispute related losses. Faster resolution

can also have an income effect because of a reduction in expenses of the dispute itself. Moreover, faster

resolution can reduce any earnings lost by the firm as a result of the dispute – for instance, if

construction is delayed by a dispute over the building permit a faster resolution can reduce the amount

of time a firm has to wait to realize earnings on the constructed property. Finally, the BO services may

also successfully prevent escalation of disputes to court, which has savings for both the firm and the

government (although, it is also possible that the written legal opinion provided by the BO encourages

firms to go to court if pre-court resolution fails4). These are the effects we will refer to as direct effects.

We expect that firms who have positive experiences with the BO will perceive the Georgian investment

climate more favorably. It is also possible that firms who receive the information treatment but who do

not use the BO will perceive the Georgian investment climate more favorably because of their improved

knowledge of the BO. These two channels are represented in the diagram below by the two arrows

leading to the perceptions outcome box.

We then have two potential causal links affecting long-term economic outcomes for firms. One is that

the direct effects described above, such as a lower tax liability or the resolution of a building permit

dispute, will affect firms operations or investment decisions. The other is that if firms have a more

favorable view of the Georgian investment climate, including knowledge about the existence of the BO,

then these firms might be more willing to invest more and grow their operations. Different firms will

grow in different ways, some types of firms may increase investment in physical capital, others firms

may increase in the number of employees.

4 However, we think this may not be a very strong channel because the BO informed us that there were less than 5

cases in total where the BO had intervened that a firm went to court since 2012. Since some of these cases would have gone to court in the absence of the BO, the number that went to court as a result of the BO’s intervention must be lower.

12

Figure 1: Theory of Change

13

Table 2: Table of outcomes

Outcomes Description

Direct outcomes Appeal outcome (success/failure, amount of penalty paid, tax liability reduction, was the construction completed as planned)

Length of appeal process

Number of cases going to court

Amount spent on legal advice for the case

Total resources spent on case, including management time and potential bribes to government officials

Medium-term outcomes Survey questions about investment climate perceptions such as:

How much of an obstacle is government regulation for your business?

How much of an obstacle is tax administration for your business?

How much of an obstacle is obtaining building permits for your business?

How business friendly is the Georgian business climate?

Firms stated intentions to use the BO’s services in the future

Firms stated intentions to invest in the future Long-term outcomes Total investment:

Total investments as per the firm’s financial statement

Investment in project or business section related to the dispute with the government

Firm growth in: revenue, profits and number of employees

Taxes paid in years after tax appeal

Resources spent on legal advice and tax advisors

Usage of BO after first issue was resolved

4. Literature The literature on institutions and economic growth has relied primarily on cross-country variation in

data. Papers have explored the role of the rule of law, enforcement of property rights, and the ‘social

infrastructure (Acemoglu, Johnson and Robinson 2001, Hall and Jones, 1999). Firm level analyses have

corroborated earlier findings (Dollar et. al. 2005, Escribano and Guash, 2005) and also highlighted the

heterogeneity in the effects of institutions across different firms (Hallward-Driemeier and Pritchett,

2015; Xu, 2010).


14

Against this backdrop, this project represents a unique opportunity to develop rigorous evidence at the

micro level on one specific component of institutional quality. While the importance of institutions for

long -run development has been hypothesized by many scholars (see e.g. Rodrik et al., 2004), there is

very little evidence on the specifics of institutions. There is even debate on the extent to which any

government intervention can change the institutions of a country, since these are the result of political

economy equilibria that emerge over decades, and do not easily change absent some strong external or

internal forces. (Acemoglu and Robinson, 2008)

This study could give a credible answer to the question of what effect the BO has on firms and therefore

add to a larger body of evidence on how government institutions shape the economy. Furthermore, the

BO is an extremely interesting case of a government institution, since its role is to improve the working

of the government bureaucracy as a whole. Studying this type of institution can therefore potentially

also teach us about governments’ ability to deliberately change from within to become more inclusive

by empowering firms that otherwise would not have access to legal services or political advocacy.

Since the BO does not have the power to change any laws but has the power to affect how the law is

implemented, this is also a test of the importance of de facto regulation as opposed to de jure

regulation. Hence the study can speak to the body on evidence on the importance of the practice of

implementing laws in low- and middle income countries (Hallward-Driemeier and Pritchett, 2015).

The BO office in Georgia combines aspects of traditional alternate dispute resolution methods such as

Early Neutral Evaluation with advocacy services that exploit the political connections of the BO office to

improve the de facto implementation of legal rights of firms against the state in Georgia. Evidence

suggests that ADR mechanisms can improve the efficiency of the judicial system (Gropper, 2010), save

costs for courts (Wissler, 2004), and increase the speed and decrease the cost of resolution of disputes

(see Love, 2011 for a review). Evidence on the effects of these on firm outcomes is limited, with one

exception being Chemin (2012), who finds that a reform of courts in India improving court speed

increased firms’ investment and access to finance.

On top of these services, the BO offers advocacy services. These advocacy channels are either formal,

with the BO’s being a part of dispute resolution committees in other government agencies, or informal,

relying on the fact that the BO is a politically well-connected office. It is well established that political

connections can be valuable to firms in several different ways (see e.g. Fisman, 2001, Khwaja and Mian,

2005 and Faccio, 2006). This study will show the effect of giving firms access to a type of political

connection in a free, unbiased and legal way through a government institution.

Another important way this study may contribute to the academic literature is its potential effect on the

long standing debate around whether corruption and political connections “grease the wheels” of

development (Bardhan, 1997). Some argue that potentially corrupt political connections between firms

and politicians may actually improve growth by providing firms with ways to side-step regulation that

would otherwise hurt their growth potential (see e.g. Méon and Weill, 2010). On the other hand several

studies have shown how corruption may be detrimental to growth by creating efficiency wedges and

uncertainty about the implementation of regulation (Campos et al., 2010). The institutional innovation


15

of the Georgian BO provides a third way in the policy conclusions from this debate by regulating access

to political advocacy for all firms and by providing this for free in a legal and orderly manner.

5. Hypotheses and Evaluation Questions Based on our theory of change, we have three main hypotheses:

Hypothesis I (see Figure 1 box 1.): Providing firms with information on the BO’s office will

increase the number of firms who use the BO’s services.

Hypothesis II: The BO’s services will help firms in achieving better outcomes in their disputes

with the government, this will subsequently alter firms’ perception of the Georgian business

climate. These two effects will lead to improved long-term economic outcome for the firms.

Hypothesis III.A (see Figure 1 box 2a.): A secondary hypothesis is that providing firms with the

BO’s legal services alone is sufficient to achieve the outcomes described in the TOC section

Hypothesis II above.

Hypothesis III.B (see Figure 1 box 2b.): A secondary hypothesis is that the full suite of the BO’s

services, including informal advocacy, are needed to achieve the outcomes described in

Hypothesis II.

The design of the evaluation aims at answering questions related to both overall effect of the BO as well

as questions regarding the relative importance of the BO’s services. Our main questions are:

1. Testing Hypothesis II: What is the impact of the BO’s services on the direct, medium-term

and long-term firm-level outcomes listed in Table 2 (also see Figure 1. box 1.).

2. Testing Hypothesis III.A vs. Hypothesis III.B: What is the difference in effect between the

BO providing legal services (see Figure 1. box 2a.) and providing full services (see Figure 1.

box 2b.).

These questions were derived from the dual purpose of this study, to estimate the effect of the BO and

to better understand the mechanism behind this effect.

6. Evaluation Design and Sampling Strategy

Identification strategy and stratification

The identification strategy relies on the exogenous variation provided by the random selection of firms

into the information treatment to answer Question 1 and the random selection into Treatment 1 (box

2a. in Figure 1) or Treatment 2 (box 2b. in Figure 1) to answer Question 2.

Since randomization will be carried out on a rolling basis, we will not stratify the randomization along

any baseline variables. If we find, ex-post, that the treatment and control group are unbalanced, we will

control for this in our analysis using baseline covariates.

Figure 2: Randomization arms and sample size


16

Take-up

As the intervention will be the information given to the firms and not the actual BO’s services, this

should be considered an encouragement design where we hope that the knowledge about the BO will

induce firms to contact the BO (i.e. take up the treatment). Our focus groups showed that lack of

information is one reason why firms do not seek help from the BO but there might also be other

reasons. For example, some firms in the focus groups expressed concerns about the neutrality of the

BO’s office if the BO is a government representative. The information intervention will be designed to

minimize such concerns but we cannot expect all firms to be convinced that the BO’s services will be

beneficial to them. Hence the take-up of the actual treatment will not be 100%.

We will define take-up as having at least sought help from the BO, regardless of whether the BO could

actually help or not. Given this broad definition we hope that take-up will be above 50% since the BO’s

services are free and all of these firms would in theory benefit from the BO’s assistance. Care will be

taken to improve take-up as much as possible by, for instance, having well design information material

and, if needed, having the survey firm or the BO calling firms to remind them of the potential benefits of

using the BO’s services.

Sample size

We expect to have a sample size of approximately 1,000 firms where 750 firms will be from the Revenue

Service’s appeals offices and 250 firms from the Tbilisi Municipality’s complaints office. One third of

firms will be assigned to the information treatment leading to the BO’s full services, one third will be

assigned to the treatment leading to only getting legal services from the BO’s office and one third of

firms will be kept as a control group. This is illustrated in Figure 2 above.

Firms recruited from Department of Revenue and Tbilisi Municipality

Sample size (1,000)

Information Treatment (see Figure 1 box 1.)

Any treatment (660)

BO provides legal support (see Figure 1 box 2a.)

Treatment 1 (330)

BO Provides legal support AND advocacy services (see

Figure 1 box 2a. and 2b.)

Treatment 2 (330)

Control (placebo information)

Control (340)


17

While collecting data on approximately 1,000 firms is our expectation we will make full use of the

flexibility that our intervention design gives us. Since the intervention is rolled out gradually we will

monitor the take-up among firms continuously and potentially increase or decrease our sample size

depending on how many firms who actually contact the BO’s office, the proportion of firms assigned to

treatment and control will stay constant throughout the study.

When determining the final sample size we will trade off two competing objectives. The first concern is

to not overload the BO’s office with work as our intervention starts, if many more firms than we expect

approach the BO’s office we may reduce the number of firms we give the intervention treatment in

order not to increase the workload on the BO’s office too drastically. The second objective is to have a

sufficient number of firms actually seeking out the BO’s services. If take-up is lower than we expected

we will therefore increase the sample size in order for our study to maintain an adequate statistical

power. In order to do so without overloading the BO, we will extend our intervention over a longer

period so only a manageable number of firms approach the BO each month.

Power calculations

At this stage any power calculation will be heavily dependent on assumptions and should be considered

with great caution. However, we have performed a few power calculations to get a rough estimate of

what sample size the study requires.

The power calculations have been done using simulated data replicating the distributions of the data

from the World Bank’s Enterprise Survey in Georgia (2013) on investments in fixed assets and how much

of an obstruction tax administration is for the current operations of a firm. In doing so we are assuming

that the firms in our sample will be similar to the firms in the Enterprise Survey. In order to implement

the baseline variable control we had to make an assumption about the inter-temporal correlation of the

outcome variables, we have assumed that the inter-temporal correlation is 0.5 between each survey.

We have defined “power” as the probability of rejecting the null hypothesis of a zero effect at the 95%

confidence level. We measured the power by running repeated simulations of the data, 1,000 iterations

per power calculation.

We have used 4 different possible specifications for our power calculations:

1. Basic specification: 𝑦𝑖 = 𝛼 + 𝛽 × 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 + 휀𝑖

2. Baseline control: 𝑦1𝑖 = 𝛼 + 𝛽 × 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 + 𝛾 × 𝑦0𝑖 + 휀𝑖

3. Midline+Endline: 𝑦𝑡𝑖 = 𝛼 + 𝛽 × 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 + 𝛾 × 𝑦0𝑖 + 𝛿1 × 𝑒𝑛𝑑𝑙𝑖𝑛𝑒𝑡 + 𝛿2 × 𝑒𝑛𝑑𝑙𝑖𝑛𝑒𝑡 ×

𝑦0𝑖 + 휀𝑖

o Where 𝑡 ∈ {1,2} and standard errors are clustered at the firm level

4. Firm fixed effects: 𝑦𝑖𝑡 = 𝛽 × 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖𝑡 + 𝑡𝑖𝑚𝑒𝑡 + 𝑓𝑖𝑟𝑚𝑖 + 휀𝑖𝑡

o Standard errors clustered at the firm level

Where 𝑦𝑡𝑖 represents the outcome variable in time period t, t=0 is the time of the baseline survey, t=1 is

the midline survey and t=2 is the endline survey. 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖𝑡 is a dummy variable for if the firm i was


18

treated in time t. In the endline+midline specification we control for what survey the observation is from

using the 𝑒𝑛𝑑𝑙𝑖𝑛𝑒𝑡 dummy variable and the interaction between this dummy variable and the baseline

value for the outcome variable. 𝑡𝑖𝑚𝑒𝑡 is a time fixed effect and 𝑓𝑖𝑟𝑚𝑖 is a firm fixed effect.

The main assumptions of our power calculations are the following:

Sample size: 1,000 firms

Take-up: 50%

o Please note that in this project in practice, sample size will be adjusted depending on

the take-up we observe. So if take up is lower than 50% sample size will be increased

and vice versa to keep the total number of firms that seek out the BO’s services as a

result of this project to be around 330.

Take-up among firms in the control group: 2%

o The number of firms currently seeking the BO’s help for tax issues divided by the total

number of firms appealing their original tax decision is approximately 0.015.

The probability of a further appeal is 30% (the fraction of firms that appeal the decision of the

first level dispute resolution committee at the Revenue Service to the second level dispute

resolution committee at the Ministry of Finance).

The perceptions and investment variables have the same distributions as these variables do for

Georgian firms in the Enterprise Survey (2013).

o Inter-temporal correlation of perceptions and investment variables is 0.5

Effect on probability of dispute taken to the next level in the appeals process (short-term

outcome): 15 percentage points (control group mean is 30% which is the currently

approximately the rate of disputes taken to the second appeal committee in the Revenue

Service)

Effect on perceptions (medium-term outcome): 1 category (out of 5) change for half of the firms

taking up the treatment, approximately 0.19 standard deviations.

Effect on investment (long-term outcome): 30% increase in investment for firms that would

have had investment and a 30% probability of median investment for firms that would have had

no investment.

o Among firms interviewed after having used the BO’s services the average “investment at

risk of being cancelled” if the BO had not intervened was 33% of total investments in

that year

All treatments vs. Control

Probability of further appeal

Perceptions (categorical variable)

Investment

Basic, no controls 70% 82% 41%

Baseline controls NA 90% 59%

Midline+Endline NA 94% 62%

Firm fixed effects NA 76% 36%


19

One treatment vs. Control

Basic, no controls 60% 73% 33%

Baseline controls NA 79% 43%

Midline+Endline NA 87% 45%

Firm fixed effects NA 63% 28%

Note: percentages corresponds the power each specification has to find a significant effect at the 95% level for a given outcome

variable.

The power calculations above indicate that the study is well powered to find effects on medium-term

outcomes such as perceptions, especially where we have baseline figures for these outcomes. For the

variables related to the final step in our theory of change we are still uncertain about the power of the

study. Above we have estimated the power for investment which is a noisy variable where a few outliers

have a large effect on the mean of the distribution. Due to this feature of the investment distribution we

used to do these power and our conservative assumption that investment only has a year-to-year within

firm correlation of 0.5, the power for this outcome is lower than what we would like it to be. However,

do take into account that other long-term variables such as turnover, employment and tax-payments to

be more stable from year to year and hence we expect more precisely measured estimates for these

outcome variables. Furthermore, to improve our ability to measure the effect of the BO’s services on

investment we will collect data on investments that are directly related to the project or business

segment that had the dispute with the government. In these investments there should be less noise and

a more direct effect from the BO’s services, especially in building permit related cases.

Once we have started to receive baseline data on the actual of firms and the actual outcome variables

that we are going to use in our analysis we will update our power calculation. If we at that stage find

that our study is underpowered we can increase the sample size by letting our intervention run for a

longer time and thereby collect data on a larger set of firms until our power reaches satisfactory levels.

7. Data Collection The study will use two main data sources. One is the survey data we will obtain by interviewing the firms

and the other is the administrative data on the firms’ and their disputes that will be given to us by the

BO, the Revenue Service and the Tbilisi Municipality. All firms participating in the study will have given

written consent to participate in the study and have their administrative data shared by the relevant

government agencies. This consent will be important since it is a requirement from the government to

provide us with administrative data for these firms.

Quantitative Instruments

The main data source will be survey responses from in-person interviews with top managers in each firm

before and after the intervention. As described above, we intend to collect baseline information from

the firms in direct connection with the information intervention. The first follow-up survey is planned


20

about 12 months after the intervention and the second follow-up survey is planned approximately

24months after the intervention. The first follow-up survey will focus on measures directly related to the

BO’s services and measures that we think might change relatively quickly such as firms’ perceptions of

the Georgian investment climate and their intentions to make future investments in Georgia. The

second round of follow-up interviews will focus on the long-term economic outcomes such as turnover,

profits, number of employees and investments that we ultimately hope to see an effect on but that

might take a longer time to change. We will also ask firms about expenditure on lawyers and tax

consultants.

Additionally, in our agreement with the BO, Georgia Revenue Service and Tbilisi Municipality, for all

firms in the study, they will provide us with administrative data on the nature of these firms disputes,

the eventual resolution to the disputes and their overall tax payments for the period 2015-2019. This

data will be submitted to us on a regular basis from all three agencies.

Management of Data Quality

The main responsibility for survey data collection and quality control of this data will be with the survey

firm collecting the data. Georgia has several high quality survey firms with experience of this type of

data collection and we are currently in negotiations with one of Georgia’s top survey firms for this work.

The survey firm will be doing back checks on a rolling basis of surveys on selected questions both by

phone and in person on approximately 20% of the sample (randomly selected). The back check data will

be submitted to us directly so that the research team can make sure that it corresponds to the original

survey data. If more than 1% of the data can be considered incorrect (e.g. if it substantially deviates

from the back checks) that is considered a breach of our contract with the survey firm. In addition to the

survey firms’ back checks our field coordinator will conduct back checks on his own among a few

randomly selected firms.

Qualitative Instruments

To better understand the mechanisms through which the BO operates we will conduct in-depth

interviews with both the BO staff and the officers working at the Revenue Service as well as the Tbilisi

Municipality. During these interviews we will focus on the political economy aspects of how the BO

affects the government agencies decisions. When interviewing the government officers we will also

attempt to understand if some of the BO’s effect can be due to displacement of effort of government

officers.

Implementation monitoring system

We will have administrative data submitted to us regularly from all the relevant government agencies.

This will allow us to monitor our intervention in real time and, if needed, make adjustments to the

implementation protocols.

Our field coordinator will make frequent visits to the Revenue Service’s field offices and the Tbilisi

Municipality to coordinate the intervention. The field coordinator will also work closely with the survey

firm to ensure that information intervention is rolled out correctly and that the quality control protocols


21

are being adhered to. Finally, the field coordinator together with the IFC staff based in Georgia will work

closely with the BO’s office to monitor the increase in cases that come to their office as the intervention

is being rolled out. One main advantage of the intervention being continuously rolled out while we

monitor the initial indicators, such as firms approaching the BO’s office, is that we can make

adjustments to the intervention as needed.

8. Data Processing and Analysis

Main econometric specification

We will estimate an Intent to Treat (ITT) effect which will capture the effect of the information

intervention. Since we do not expect any substantial spillovers between firms in the treatment group we

can also estimate the Treatment on the Treated (TOT) effect of the BO’s services by using the

information intervention as an instrument for usage of the BO’s services, however this interpretation

requires a few additional assumptions. Most notably we have to assume that the information treatment

itself does not affect the firm’s performance though any other mechanism than usage of the BO’s

services. Firms behavior and perceptions cannot, for example, be influenced by simply knowing about

the BO’s services without actually using them. The TOT should interpreted as a Local Average Treatment

Effect (LATE) for the firms whose decision whether to approach the BO is affected by the information

intervention. Since we only expect very few firms in the control group to use the BO’s services, the TOT

effect will approximately be the ITT effect divided by the take-up rate.

For estimating the ITT effect on outcome variables that we do not have baseline data on, such as

information on the solution of the bureaucratic issue (tax or building permit appeal), our main

econometric specification will be a simple OLS regression:

𝑦𝑖 = 𝛼 + 𝛽 × 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 + 𝛾 × 𝑥𝑖 + 휀𝑖

Where 𝑦𝑖 is the outcome variable such as log of penalty paid, 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 is a dummy variable indicating

if the firm was treated with the information intervention or not, 𝑥𝑖 is a vector of pre-intervention firm

characteristics and 휀𝑖 is an error term. We will use heteroscedasticity robust standard errors to make

statistical inference. Since the intervention is randomized and implemented at the firm level we will not

cluster the standard errors.

For variables for which we have panel data both before and after the intervention, such as perceptions

about the Georgian investment climate, we will use a firm and time fixed effects regression as our main

specification.

𝑦𝑖𝑡 = 𝛽 × 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖𝑡 + 𝑡𝑖𝑚𝑒𝑡 + 𝑓𝑖𝑟𝑚𝑖 + 휀𝑖𝑡

Where 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖𝑡 is a dummy variable for if the firm I was treated in time t, 𝑡𝑖𝑚𝑒𝑡 is a time fixed

effect and 𝑓𝑖𝑟𝑚𝑖 is a firm fixed effect. In the firm fixed effects regression no pre-intervention firm

characteristics needs to be controlled for since all of them will be accounted for by the firm fixed effect

𝑓𝑖𝑟𝑚𝑖.


22

Although these will be our main econometric specifications for the ITT effect we will still report the

simple OLS regression specification with just the treatment dummy on the right hand side of the

equation as a robustness check.

As mentioned above, we will use an instrumental variables approach to estimate the TOT effect where

the information treatment will act as an instrument for getting the BO’s services. One potential problem

of interpreting the instrumental variables approach as a TOT is that the information intervention may

have an impact on firms even if they did not approach the BO (as outlined in Figure 1). This potential

problem may make us interpret the instrumental variables estimates with additional caution.

Heterogeneous effects

The main aim of the BO is to facilitate interactions and dispute resolution between firms and the

government. Therefore we expect that the long term effects of the BO on investment and growth will be

the largest for firms in industries where frictions with the government are a binding constraints to firms’

activities. We will attempt to understand for what industries this is most important in our baseline

survey and then look for heterogeneous effects along this dimension.

Another important dimension for heterogeneous effects will be firms’ political connections previous to

using the BO’s services. If the BO’s services is a substitute for a firms political connections we expect the

treatment effect to be the largest for firms without political connections. It may be reasonable to

expect, for instance, that this is correlated with firm size, and therefore, it may be interesting to look at

heterogeneity along firm size.

Multiple hypothesis testing

Given the broad potential for implications of our project, it is natural to estimate the effect of the

intervention on a broad set of outcomes. This may lead to issues where the number of hypothesis that

we are testing are so many that some estimate will be statistically significant by chance. We will avoid

this using three types of counter measures,

i. We will specify a pre-analysis plan and submit this to the American Economic Association's

registry for randomized controlled trials. This will avoid any fear or suspicion of

“specification search” or “data mining” in the analysis of the data.

ii. We will create aggregate measures of our main outcomes, the processing of the initial issue,

firm perceptions of the investment climate and firm growth. Reducing the outcomes into

three main categories reduces the risk of some estimates of the effect becoming significant

by chance.

iii. Finally, when analyzing the effects on a wide variety of specific outcome variables we will

report “family-wise” p-values that are adjusted for the multiple hypothesis testing within

that category of outcomes.


23

9. Study Limitations and Risks

External validity

Given that the sample focuses on firms that already have a problem with a Georgian government

institution we cannot claim that the results of the study will be generalizable to the general population

of Georgian firms. However, since this is the population the BO’s office if set up to serve, the study’s

choice of sample is done so that it is focusing on the relevant part of the population of Georgian firms,

i.e. the firms that have some type of problem with the government.

Given that Georgia is one of very few countries in the world to have a fully functioning SIRM it is hard to

say if Georgia’s BO will be similar to future SIRM that may be set up elsewhere in the world. Even if we

were to look only at the Georgian economic, bureaucratic and political context it is hard to say if we can

draw conclusions from the Georgian experience to other countries. Therefore we see this study more as

a potential proof of concept that a SIRM or a BO can be beneficial in some contexts. Having said that, in

terms of GDP or HDI ranking Georgia is similar to other countries considering adopting SIRM such as the

Dominican Republic, Peru and Albania.

One potential threat to the interest of the BO in scaling up their services is the limited resources at their

disposal. Currently, the BO office has nine full time employees dealing with approximately 300 cases in a

year. For the duration of the impact evaluation, we will hire extra staff at the BO’s office to deal with the

additional cases that we expect our information treatment to generate. We have had conversations with

the BO about these additional staff members and the BO is currently planning to absorb these staff

members into the organization at the end of the research project.

Spillovers

One potentially problematic aspect of the research design is the potential for spillovers from the

treatment to the control group. These spillovers could take two forms:

i. Bureaucrats in the Revenue Service and the Tbilisi Municipality may have limited time (or

attention) and when the BO is involved the bureaucrat may choose to focus on the cases

involving the BO, reducing the attention given to firms in the control group. If this is true,

firms in the control group may get a worse treatment by the bureaucrat than it otherwise

would. This type of spillover has two implications for our study.

a. We may not be able to attribute the entire difference between the treatment and

control group to the impact the treatment had on the treatment group. Instead some of

the difference may come from the negative impact the BO had on the control group.

b. It may not be valid to extrapolate our results to a setting where all firms are informed by

the BO’s services. If the BO shifts bureaucrats focus from one case to another then an

overall increase in the use of the BO’s services may not have the same effect as the

increase in the usage of the BO among a small number of firms.


24

ii. A similar mechanism may be at work in terms of tax revenue. If firms that use the BO’s

services are relieved from more of their tax liabilities and the government still has a certain

revenue requirement, then the Revenue Service may increase tax collection from firms not

using the BO’s services to recuperate the losses caused by the BO.

Spillovers may also exist in the opposite direction if cases resolved by the BO have an impact on

outcomes for firms in the control group. For instance, in the case of ambiguous tax provisions, if the

legal advice provided by the BO or the written legal opinion provided by the BO in court in effect clarifies

the ambiguity in a tax provision that affects several firms, then the effect of the BO’s intervention will

likely impact firms in the control group who are affected by the same issues.

We plan to capture these spillovers through a combination of methods,

i. First, we are expecting to get data from the Revenue Service on the officer assigned to each case

within the treatment and control group. If there is a crowding-out of effort by the case officer, we

expect to find evidence of this by comparing control group firms that have different number of

treatment firms assigned to them.

a. By comparing outcomes across case officers who received a different number of 'treatment’

cases, we can identify if outcomes are worse in cases assigned to officers who received a

higher number of ‘treatment’ cases.

b. By comparing outcomes across the treatment and control group for each case officer, we can

identify if there are differences between treatment and control group firms that had the same

Revenue Service officer.

ii. Second, we will collect data on firms without giving any of them the information intervention for

approximately 2 weeks before the start of the start of the information intervention and 2 weeks

after the end of the treatment intervention. If the firms in the control group are affected by the

treatment, we expect to see a discrete change in the outcomes for the control group firms whose

data was collected during the treatment period and those whose data was collected outside of the

treatment period. Hence we can use a regression discontinuity design to identify if the treatment

has any effect on the control group. The discontinuity would occur at the time the information

treatment is implemented and again when the treatment stops and we can control for time

effects using a flexible parametrical specification, such as a high order polynomial.

iii. Third, we are also complementing these approaches with qualitative data collection from the BO

and the Revenue Service to understand if there are any cases during the study period where the

BO’s intervention has caused a re-interpretation of regulation that affects firms in the controls

group.

A concern about welfare implications

One potential concern is that while some outcomes generated by the BO’s assistance are unequivocally

desirable (faster resolution and increased investment), others are less clear cut (eg. tax paid by firms).

We will discuss the possibility with the BO to provide us as much information as possible in terms of the

details on each case. Among these details should be copies of the legal opinions that the BO provide to

many of their clients. We may then be able to code each case as being in accordance with the tax law or


25

not. Irrespective of our ability to turn these legal opinion letters into a quantitative variable, the legal

opinions may give us general understanding of if the firms’ claims are in accordance with the Georgian

law or not.

Hawthorne, John Henry and survey effects

One potential issue is the threat that the BO will put in extra effort to benefit the firms in our treatment

group. While it is possible that the BO increases its overall effort level during the study period it will not

be possible for the BO to do so for just the firms in the treatment group. This is because we will not

share the data about what firms are included in the treatment group with the BO while the processes

are still on going. At some point we will have to share the data about what firms were part of the study

in order to gain access to the BO’s data on what was done for these firms but this will likely be after

most of the firms’ issues (their tax and building permit appeals) have been resolved.

There might also be Hawthorne effects where the firms change their behavior because they are being

surveyed. However, if such an effect exists we have no reason to believe that it should effect the

treatment and control group differently and therefore such an effect does not pose a plausible threat to

the internal validity of the study. Furthermore, given that all firms will not be explicitly told if they are in

the treatment or control group of the study it is unlikely that such considerations would affect their

behavior.

Given that key outcome variables will be survey data provided by the firms themselves we have to be

concerned about potential survey effects. This is especially concerning for measuring the effect on firms

perceptions since these are very subjective measures and can therefore easily be affected by, for

example, a gratitude effect. One example of a gratitude effect would be if firms in the treatment group

want to show their gratitude towards the surveyor, perhaps for giving them the information about the

BO, by claiming that the investment climate is favorable since this is what they think the surveyor wants

to hear. One way to test for if there is such an effect is to include questions about perceptions that

should not have been affected by the intervention. For example, we can ask firms about their

experience dealing with the Georgia Revenue Service in the year before the intervention. If there is no

effect of the intervention on this answer, we can be more comfortable there that the answers are not

affected by a gratitude effect. Other outcome variables, such as investment or tax payments, will be

relatively objective and therefore are less likely to be affected by such an effect.

10. Policy Relevance and Impact

Policy relevance for the Georgian Government and the BO’s office

Most immediately, the impact evaluation offers lessons for the Georgian Government. If the BO’s

services have a large effect on firm behavior, then in itself, this is evidence that firms are severely

constrained by the government bureaucracy and shows one clear way to relieve this constraint.

Furthermore, it will also be relevant for the government to know what parts of the BO’s services had the

largest effect. If it is the “Legal Services” that helps firms then coming up with strategies to provide legal

support for firms should be a priority. If it is the advocacy work that the BO does that drives the effect,


26

this suggests that the government may have to become more open to listen to firms concerns and may

provide a strong motivation to strengthen these channels, for instance by formalizing these advocacy

channels (e.g. in the case of the second-level appeals committee of the Revenue department, the BO

has a vote on an eight member appeals committee).

There are of course several important questions for the BO’s office that may be answered by the study.

If, as we expect, the information treatment is effective in encouraging firms to use the BO’s services, it

offers the BO a low-cost way to increase take-up of their services among firms in Georgia. The second

layer of variation will test the value of the advocacy services offered by the BO. If these are either found

not to add significant value, or conversely, to add the bulk of the value to the BO’s assistance, this can

help the BO focus its services in the future. This will be particularly important if more firms visit the BO’s

office.

It is possible that the results of the impact evaluation and the attention generated by them can help the

BO petition for additional resources. In order to maximize the likelihood of this scenario, the impact

evaluation and project teams will periodically organize dissemination workshops with results and

insights from surveys and from implementation with the BO and also members of the Ministry of

Economy and Sustainable Development and the Economic Council with the aim of raising the profile of

both the impact evaluation and the BO office itself within the government.

Broader Policy relevance

The impact evaluation will provide important evidence about effective ways to improve government

institutions in an area of increasing importance – investor protection. Surveys conducted by the WBG

have indicated that investors often withdraw their investment or cancel expansion plans due to investor

protection issues.

Improving investor protection is a matter of both improvement in the (“de jure”) legal framework of

investor protection guarantees and of (“de facto”) implementation of laws, regulations and contracts.

Weaknesses in implementation, in addition to affecting perception of investment climate and therefore,

the potential to attract new investments, can have direct costs to countries related to arbitration and

litigation – for instance, the average arbitration cost in such disputes is estimated at $3 million in

administrative costs and $10.4 million in damages per case5. Furthermore, these disputes can affect the

retention of existing investment, which is a significant channel for investment in developing countries

(estimated to be one third of FDI flows to developing countries).

The SIRM aims to prevent investor disputes from escalating into disputes by identifying and resolving

them at an early stage, by providing government with the minimum institutional infrastructure to

identify, track and manage grievances arising between investors and public agencies as early as possible.

The WBG has distilled elements of successful dispute prevention and general aftercare systems (which

have a broader mandate than tracking and managing early-stage disputes) into the current design of the

5 Based on an analysis of 82 ISDS cases worldwide. From Franck, Susan D., Empirically Evaluating Claims About

Investment Treaty Arbitration. North Carolina Law Review, Vol. 86, p. 1, 2007. Available at SSRN: http://ssrn.com/abstract=969257


27

SIRM, but there is little ‘best practice’ information available on such mechanisms. The Collaborative

Network in Bosnia and Herzegovina and the Office of Foreign Investment Ombudsman in South Korea,

perhaps the two most developed examples of such systems, offer two very different approaches.

In sum, this impact evaluation can help (i) make a clear case for using such mechanisms to improve the

de-facto implementation of investor protection, and (ii) test and improve an early design of the SIRM,

one such mechanism. These lessons can provide clear and rigorous information to developing countries

elsewhere interested in implementing such mechanisms. Indeed, the WBG itself is in the early stage of

engagements on similar projects in other countries, and early lessons from the impact evaluation can

also have an important impact on these ongoing engagements.

11. Dissemination Plan

Local dissemination

Given that the Impact Evaluation will affect the functioning of the BO for the period of the evaluation

and that one of the aims of the evaluation is to affect the operations of the BO in the longer term, the

team will coordinate dissemination events at each stage of the evaluation and each time results from

data collection activities are available. In specific, a workshop with the BO office and implementing

partner agencies (Georgia Revenue Department, Tbilisi Municipality) took place in May 2016 to get

formal agreement (Memorandum of Understanding) on the implementation arrangements and to

familiarize the implementation partners with the design and objectives of the impact evaluation. At the

end of the implementation period, a workshop will be planned with information on the encouragement

intervention and to share results from the baseline exercise. One more workshop is planned for

dissemination of short-term results after the conclusion of the first follow-up survey. Once we have

analyzed the results from the final round of data collection we will have a decision making workshop

with the key stakeholders of the project - the BO, the Georgia Revenue Service and the Tbilisi

Municipality. There we will present the results, discuss the implications of the study for the relevant

government actors and help them identify elements of an action plan going forward. Finally the team

will organize a final dissemination event to mark the end of the impact evaluation with a broader

audience including other relevant ministries and national agencies such as the Ministry for Economy and

Sustainable Development, the National Investments Agency, and any firm associations or Chambers of

Commerce in Georgia. At this event we will present the results of the study as well as the action plan for

how to incorporate the policy implications of the study into actual policy making.

Global dissemination

Since the results of the impact evaluation also carry useful lessons for the WBG, events to disseminate

results from each stage of the impact evaluation are planned for audiences within the WBG, including

two regional events at the end of the impact evaluation to disseminate results to teams in the regions

where the demand for these interventions is currently highest. Four more dissemination events are

planned at academic institutions and academic conferences at the end of the impact evaluation in order

to ensure wider dissemination into research.


28

12. Milestones, Deliverables, and Timeline

Milestones Completion Date

Brief proposal Completed, Feb 2016

Peer reviewed methodology Completed, Jun 2016

Plan for baseline data collection and intervention Jul 2016

Pilot data collection and intervention Jul-Aug 2016

Baseline data collection and intervention Aug-Dec 2016

Baseline data analysis Feb 2017

Mediation/advocacy intervention Aug 2016-Jun 2017

First Follow up data collection Aug-Nov 2017

First follow up data analysis Feb 2018

Second Follow up data collection Aug-Nov 2018

Final data analysis Mar-Jun 2019

13. References Acemoglu, Daron, Simon Johnson, and James A. Robinson, "The Colonial Origins of Comparative

Development: An Empirical Investigation," American Economic Review, 91, 5, December 2001,

1369-1401.

Acemoglu, Daron, and James A. Robinson. 2008. "Persistence of Power, Elites, and Institutions."

American Economic Review, 98(1): 267-93.

Alvarez de la Campa, Alejandro. 2009. “The Private Sector Approach to Commercial ADR:

Commercial ADR Mechanisms in Colombia.” Investment Climate Department, World Bank,

Washington, DC. http://www.fias.net/index.cfm.

Bardhan, Pranab. "Corruption and development: a review of issues." Journal of economic

literature 35.3 (1997): 1320-1346.

Besley, T. (1995). Property Rights and Investment Incentives: Theory and Evidence from Ghana.

Journal of Political Economy, 103(5):903–937.

http://www.fias.net/index.cfm


29

Nauro Campos, Ralitza Dimova and Ahmad Saleh. 2010 "Corruption and Economic Growth. An

Econometric Survey of the Evidence." Journal of Institutional and Theoretical Economics.

Chemin, Matthieu. 2010. “Does Court Speed Shape Economic Activity? Evidence from a Court

Reform in India.” Journal of Law, Economics, and Organization, advance access, November 11,

2010.

David Dollar, Mary Hallward‐Driemeier and Taye Mengistae. 2005. Investment Climate and Firm

Performance in Developing Economies. Economic Development and Cultural Change, Vol. 54,

No. 1, pp. 1-31.

Field, E. (2005). Property Rights and Investment in Urban Slums. Journal of the European

Economic Association, 3(2/3):279–290.

Gropper, Akvile. 2010. “ADR in Tax Disputes.” Research Note, World Bank, Washington, DC.

Hallward-Driemeier, Mary, and Lant Pritchett. 2015. "How Business Is Done in the Developing

World: Deals versus Rules." Journal of Economic Perspectives, 29(3): 121-40.

Hall, Robert, and Chad I. Jones, “Why Do Some Countries Produce So Much More Output per

Worker than Others?” Quarterly Journal of Economics, 114, 1, February 1999, 83-116.

Jorquiera, Carlos Eugenio, and Gabriel Dabdoub Alvarez. 2005. “The Cost of Disputes in

Companies and the Use of ADR Methods: Lessons from Nine Latin American Countries.” MIF

Study, Multilateral Investment Fund, Washington, DC.

Faccio, Mara. 2006. “Politically Connected Firms”. The American Economic Review, Vol. 96, No.

1, pp. 369-386.

Fisman, Raymond. 2001. “Estimating the Value of Political Connections”. The American

Economic Review, Vol. 91, No. 4, pp. 1095-1102.

Mian, and Khwaja. “Do Lenders Favor Politically Connected Firms? Rent Provision in an Emerging

Financial Market”. Quarterly Journal of Economics 120.4 (2005): 1371-1411. Print.

Méon, Pierre-Guillaume, and Laurent Weill. "Is corruption an efficient grease?." World

development 38.3 (2010): 244-259.

Rodrik, Dani, Arvind Subramanian, and Francesco Trebbi. "Institutions rule: the primacy of

institutions over geography and integration in economic development." Journal of economic

growth 9.2 (2004): 131-165.

Wissler, Roselle. 2004. “The Effectiveness of Court-Connected Dispute Resolution in Civil Cases.”

Conflict Resolution Quarterly 22: 55–88.

Xu, Lixin Colin. 2010. The effects of business environments on development: surveying new firm-

level evidence. Policy Research working paper ; no. WPS 5402. Washington, DC: World Bank.

Date post:	20-Jun-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Impact Evaluation of a Systemic Investor Response Mechanism in...

Documents