Compared Assessment of Foreign Assistance WebsitesJulie Biau Jbiau@brookings.edu Christine Zhang CYZhang@brookings.edu

Global Economy & Development Development Assistance and Governance Initiative

INTRODUCTION

3

• Notes on terminology and user profiles• Evaluation Criteria• Walkthrough of different interfaces

» Interfaces that are sources of data:– AidData– OECD Creditor Reporting System– IATI Registry

» Interfaces that are visualization platforms:– US Foreign Assistance Dashboard– DFID Development Tracker– IATI apps (OpenAidNL, d-portal, AidView)

Outline

4

• The following terms are used in this analysis to evaluate the selected websites:» Clarity: content is easy to locate

» ‘Recently updated’: data is not more than 1-2 years old

» Accuracy : degree to which the data compare to official sources

» Quality : degree to which the data are well organized and reliable (few missing values, no double counting, etc.)

» Comprehensive: data cover most donor and recipient countries (DAC donors; 100-200 recipients; all regions)

» Accessible: portal allows free bulk export of data; portal contains disaggregated data; data is published under an open license

A note on terminology

5

• The usability and relevance of a website varies by who is using it, and what they are using it for. We distinguish by:

A note on user profiles

Technical Expertise

Technical users: Someone with strong data manipulation skills who needs to

download the underlying data in order to perform detailed analysis (e.g.

Researchers, academics, students)

Everyday users: Someone looking for summary figures of aid flows from a

given donor to a given country and/or sector, with no need for further analysis

(e.g. Development practitioners, government officials, general public)

Geographic Scope

‘Macro’ users: Someone needing information on aid flows by donor,

country and sector, but not necessarily interested in project-level information

(e.g. Capitol Hill staffer, USAID regional office, recipient-country finance ministry)

‘Micro’ users: Someone interested in project-level information with sub-sector

and sub-national detail, perhaps less concerned with ‘big picture’ statistics

(e.g. USAID country office, local NGO, recipient-country line ministry)

6

Criterion 1: Ease of using interface

Applicable to

Good Moderate Bad

All users Interface is clear (making it easy to find and understand content), fast, and has useful data visualization / query customization options   

Interface lacks clarity or has uninformative or unwieldy data visualization and query customization options 

Interface is very unclear (making it very hard to find and/or understand content) and/or very slow

Note: the usability of image heavy websites is likely to decline with a limited internet connection. Simplified versions for slow connections would be useful.

7Criterion 2: Relevance of ContentApplicable to Good Moderate Bad

Micro users (e.g. donor agency field office, recipient country finance ministry, local NGO)

Content (i) has been recently updated (1-2 years), (ii) is comprehensive in terms of country coverage, and (iii) covers all aspects relevant to the audience, .e.g.:- Resource flows from all donor

governments to the recipient country by sector

- Project-level information on activity within a country by region

- Historical and forward-looking information

Content covers most aspects relevant to the audience but has not been recently updated (3+ years) and/or is not comprehensive in terms of country coverage

Content covers few or no aspects relevant to the audience

Macro users (e.g. donor government congressperson, donor agency regional office)

Content (i) has been recently updated (1-2 years), (ii) is comprehensive in terms of country coverage, and (iii) covers all aspects relevant to the audience, .e.g.:- Resource flows from donor

government by source (ministry/agency)

- Resource flows from donor government by world region, country and sector

- Historical and forward-looking information

Content covers most aspects relevant to the audience but has not been recently updated (3+ years) and/or is not comprehensive in terms of country coverage

Content covers few or no aspects relevant to the audience

Note: this analysis intends to apply equally to donor and recipient-country users; however poor availability of information in other languages is a limiting factor.

8Criterion 3: Availability of DataApplicable to Good Moderate BadTechnical users (e.g. researcher, academic, economist)

Raw data is available for download in a readily usable format (Excel, SPSS, STATA…) with minimal user effort (low # of clicks); file is clearly organized ; there is sufficient guidance on the data (frequency of update, source, meaning)

Raw data is available for download in a readily usable format with considerable user effort and/or file is not clearly organized; there is some guidance on the data  

Raw data is not available for download in a readily usable format; there is no guidance on the data 

Non-technical users

Data is clearly presented ; specific statistics can be accessed with minimal user effort; there is sufficient guidance on the data (frequency of update, source, meaning)

Data is presented in an unclear or misleading fashion; there is some guidance on the data  

Data is difficult to access; there is no guidance on the data

Note: large data files may be difficult to download with a limited internet connection. Simplified versions for slow connections would be useful.

INTERFACES THAT ARE SOURCES OF DATA

AidData, CRS, IATI Registry:

10

AidData

Clear homepage allows to navigate between data visualization and use

Also allows to obtain ‘quick stats’ by donor, country and sector, with historical information; however these are very aggregated and do not allow to answer a specific donor-recipient query

Research tab allows access to core dataset, research datasets from various academic articles, replication datasets and geocoded data

Ease of use: Good

11

Coverage: Good - 100+ recipients for 2010, 80+ donors (fewer in certain

years) Timeliness: Poor - latest update 2012 but most countries only go to 2011; no

forward-looking informationRelevance of content: ModerateAidData

Micro: project level data with geographic information is available, with factsheets for each project of 1,322,824. However these are hard to filter or aggregate across projects.

Macro: Aid visualization option allows access to several dashboards, one at the very macro level (aggregate flows by donor, recipient, sector, year)… but these are hard to narrow down to a specific donor-recipient-sector query

Relevance of macro content: Moderate

The data needed to answer specific queries for both micro and macro users (# of donors active in a given sector of a country, total aid flows from a donor for a given sector by recipient, etc.) is available but cannot be easily generated without downloading the full dataset and manipulating it.

Relevance of micro content: Moderate

12

AidData

Main dataset can be downloaded in csv or in STATA, making this very useful for technical users; full dataset is so large that it loads incompletely in STATA or Excel, and a statistical transfer software is required; however the ‘thin’ version and aggregates versions allow for easier use.

Guidance: Numerous non-typical datasets are available for download, including on Chinese aid and underreported financial flows, with full description of methodology

A variety of datasets in Excel and STATA can be downloaded from recent academic papers

Data for technical

users: Good

13

AidData

… And downloading the data from a specific query is onerous (requires entering personal information and waiting for a link to be sent via email)

Detailed search options exist by whether one is looking for aggregate or detailed data, allowing to select recipient, donor, sector, and year – but some of the options are presented in a confusing manner and it is unclear whether certain filters are mutually exclusive, or whether they are working at all

..Search results are not presented in a very useable or informative manner (graphs but no table)

Data for everyday

users: Bad

14

OECD Creditor Reporting System

The website not very user friendly to the extent that there is no introduction page and users are launched directly into the dataset, which can be intimidating with no overall explanation of what it is or can doClicking on the information icon next to a dataset name generates a useful explanation of its contents, however

There is no explanation of how the CRS differs from the other OECD DAC datasets also on this page

Ease of use:

Moderate

Filter function works well with many options to customize the query but it may be unclear to users that a filter is currently being applied; queries are slow to load and website sometimes freezes

15

OECD Creditor Reporting System

Macro: Queries are easily customized with a detailed menu of drop-down options by sector, recipient, flow, channel, amount type, flow type, and type of aid. This function allows to select only some donors and recipients to arrive at a very specific query (e.g. US education assistance to Kenya in 2012) without downloading the full dataset, and also enables aggregation (e.g. how much are all donors providing to Kenya in education). Note that this kind of query is answered in fewer steps in CRS than on other websites.

Micro: Customization options allow for more in-depth selection of flows by sector, recipient and region and enables to compare the funding that several donors provide to a given sector in a given country. However, the information is not at the project-level or at the sub-national level so would be of limited interest to local users. Access to project-level data requires downloading the full dataset and cannot be done through web queries.

Relevance of macro content:

GoodRelevance of micro content: Moderate

Coverage: Good – 60+ donors and 180+ recipientsTimeliness: Moderate– includes some information for 2013; more

data for 2012Relevance of content: Good (macro); Moderate (micro)

16

OECD Creditor Reporting System

Customization and table layout options allow to display data exactly as it will be most useful for statistical analysis; but loading is very slow and selection time consuming / confusing

A bulk download of the full dataset, including historically, is available to Excel (however this bulk download option is hard to locate on the website and the file is so large that it loads incompletely in Excel, requiring a statistical transfer software to download it correctly)

Data for technical

users: Good

Drop-down filters allow an at-a-glance check of what certain queries look like before the data is customized and downloaded.

17

OECD Creditor Reporting System

Customization options and categories can be confusing for lay users

Charts and graphs cannot be generated for data queries that are too large

Data for everyday

users: Bad

Unless one looks for a specific query, the website does not provide aggregate statistics and there are no data visualizations

Need for more guidance: non-technical users commonly find CRS data in summary reports published by the OECD, rather than by browsing the database. Website should be clear that it targets technical users, and provide links to these summary reports for others.

18QWIDS (Query Wizard for International Development Statistics)More user friendly display of CRS data and filters, faster to navigate and less confusing; ‘popular queries’ button allows access to useful aggregated tables without needing to download the data.

Hosts data from all the main OECD aid datasets (DAC aggregate tables and CRS)

Data is displayed for the specified query and can be easily downloaded into Excel; however this interface remains complicated and difficult to use (hard to get numbers to add up).

Improves ease of use, but needs further work

19

IATI Registry

The IATI Registry consists mainly of a list of datasets provided by different aid practitioners that can be sorted through with several filters (source, publisher, organization type, recipient country, flow type). However there is no way on the Registry website to aggregate these datasets or even to filter them by sector or type, and no overall statistics are provided for lay users.

Some interfaces have been developed by third parties to facilitate use of the Registry data (particularly d-portal, OpenaidNL and Aidview), which and we will look at these in turn.

“The data is published in the international IATI standard which is easy for computers to read, but very hard for humans”(OpenAid NL). However, the IATI Registry was not intended for use as a research tool, but rather as a data repository, with the expectation that tools for its navigation would be developed separately.

INTERFACES THAT ARE VISUALIZATION PLATFORMS

IATI applications, US Foreign Assistance Dashboard, DFID Development Tracker:

21

OpenAidNL

Each project has a fact sheet with project documents, transactions, and other basic information on budget and implementation. However, the sector links and ‘export’ button do not work.

Presents the Registry’s project-level information in a format that would be very useful to local actors seeking to know which sectors are being funded in which parts of the country by the Netherlands Projects can be filtered by country. However these cannot be readily aggregated in a single dataset and have limited search options.

Filters at the top of the homepage allows to filter by country, region, sector, and budget, with a detailed drop-down menu for each dimension. Filtering produces a list of projects in the selected fields.

Ease of use: Good

22

OpenAidNL

Each project has a fact sheet with project documents, transactions, and other basic information on budget and implementation, which is quite useful for project-level queries. However, the sector links and ‘export’ button do not work, and projects cannot be aggregated.

Relevance of micro content: Moderate

Presents the Registry’s project-level information in a format that would be very useful to local actors seeking to know which sectors are being funded in which parts of the country by the Netherlands Projects can be filtered by country. However these cannot be readily aggregated in a single dataset or into country totals, making the information of limited use for macro queries.

Coverage: Poor – covers >2000 activities but just for Netherlands

Timeliness: Good – updated every 3 monthsRelevance of content: Moderate/Bad

Relevance of

macro content:

Bad

Data cannot be aggregated beyond the project level or downloaded into excel (links are broken)

No overall statistics for everyday users to refer to

Data for everyday

users: Bad

Data for technical

users: Bad

23

d-portal.org“click & scroll”

Ease of use: Good

HOMEPAGE

RECIPIENT PROFILE PAGE: CAMBODIA

d-portal presents IATI data by recipient country

“click&scroll” model makes it easy to click and scroll through project-level info

24

d-portal.orgCambodia example

VERY MACRO LEVELWHERE DOES THE MONEY COME FROM? WHERE DOES THE MONEY GO?

ACTIVE PROJECTS+ RECENTLY ENDED PROJECTS

Macro: able to attain high-level & detail information on projects in Cambodia. Can filter by exact location (if provided), donor, and sector. Cannot filter by project size or year. 2012 = most recent year (not forward-looking)

Both: d-portal identifies US funding to Cambodia as 88mn according to CRS but 0 according to IATI. This is because of insufficient reporting by donor into IATI.

Micro: clicking on an individual donor is confusing because it lists ALL projects (ranked by size), regardless of year

INTERMEDIATE LEVEL

MICRO LEVEL

Coverage: moderate – only allows searches by recipient countryTimeliness: moderate – data until 2012

Relevance of content: moderate/bad – confusing due to data gaps Relevance of

micro content:

BadRelevance of macro content: Moderate

25

d-portal.orgCambodia example

Data for technical

users: Bad

Data for everyday

users: ModerateBoth: One important note is that the

info is organized by recipient country. So there is no way to see all recipients on one page.

Micro: data is only “downloadable” by clicking on a given project, then “view source,” which generates an (unhelpful for manipulation) xml file that requires a CSV conversion tool to transfer into Excel. This tool should be listed on the site.

Macro: layout is user-friendly but not always clearly organized (e.g., donors are by agency rather than country)

26

AidView

Can filter projects by recipient country, sector or organization/donor. Filter by sector leads to projects grouped by sector and then by subsector. Buttons allow to sort projects alphabetically or by budget size

Two things are misleading here: once at the subsector level, each bubble is an individual activity, not a collection of projects in that subsector; and each screen only lists a handful of projects, rather than a comprehensive mapping for that sector, so that the user has to navigate through many screens (in the case of education, 126) to see the full list of projects. The use of bubbles and apparent grouping is therefore actually not a map, and conveys no more informative than using a simple listing of the projects by size (not an aggregation).

Ease of use: Bad

Results can also be viewed on map or as list but navigation between layouts and filters is very slow, sometimes resulting in an error message (with no way back!)

27

AidViewA useful feature for micro and macro users is that the searches can be overlaid: here the screen is showing all projects financed by the World Bank to Bangladesh in education. However this requires applying each filter successively on different webpages, which is more time consuming than selecting the desired combination from a drop down menu.

Relevance of

micro content:

GoodRelevance of macro content: Moderate

Coverage: Good: allows to search full IATI database covering 3000 activities, over 160 donors, and a large range of recipient

countries.Timeliness: Good – includes forward looking information (projects

still in planning stage); but some visualizations may need updating

Relevance of content: Good/Moderate

Project factsheets are very detailed, with project description, transactions, budget, implementing agency, partner agencies, contact details, and link to main project documents.

When more than one dimension (country/sector/organization) is selected, the ‘datafile’ option becomes active and presents information within those filters (e.g. all basic education projects in Argentina) with some level of aggregation. This is especially useful for micro users.

28

AidView IATI information at the project level can be downloaded into Excel from the project factsheet.

However the downloading option is not available at any level other than the project level (data from factsheets at a country-sector level, for example, cannot be downloaded into excel). Hence this is not a large improvement over downloading project information straight from the IATI registry, where it is also available on a project by project basis.

Data for technical

users: Bad

Data for everyday

users: Moderate

When more than one dimension (country/sector/organization) is selected, the ‘datafiles’ present some summary statistics that can be useful to everyday users.

The bubble view gives overall volumes per sector, country or donor.

29

US Foreign Assistance Dashboard

The website is user friendly and well laid out, allowing users through its ribbon and tabs to understand quite easily what kind of information it can provide in its different sections (foreign assistance by agency, by country office, by initiative; data at aggregated or transaction level; notes on methodology…). Information is available on maps and charts as well as tables.

Ease of use:

Moderate

However, the multiplicity of places to look for information – with different cuts of the data presented on different pages rather than all in one place – can make it hard to understand exactly where to go to answer a specific query. A lot of navigation required.

30

US Foreign Assistance Dashboard The dashboard allows to

explore data, including aggregate planned spending, by country office, agency and fiscal year.

Coverage: Poor – only one donorTimeliness: Good – includes information for 2013 (spent) and 2014/5

(planned)Relevance of content: Good

The country office and sector pages also give project/transaction-level information for that country, by sector, year and agency (downloadable in Excel)

Clicking on a transaction accesses a detailed project page. However, data could also be linked to other relevant sources, e.g. MCC transaction-level data, geocoded data from USAID or information from FBO on tenders.

This includes data on planned spending (2014/5), which is not usually available on aid websites

Relevance of micro content: Moderate

Relevance of macro content: Moderate

31

US Foreign Assistance Dashboard

Note: data is not yet available for every agency but this is work in progress. The fact that State Department spending is not yet on the website means that the information is a very incomplete picture of US aid flows.

Full dataset is easily available for download in Excel at transaction-level and at aggregate level. Data is available until 2015 with information on agency, operating unit, sector, and amount.

Data for technical

users: Good

32

US Foreign Assistance Dashboard

Some useful aggregations are available under the ‘where is the money going?’ tab, where spending figures and charts can be obtained by office, by recipient country or region, by sector, and by agency.

Data for everyday

users: Good

Sector and country specific breakdowns with their own summary statistics allow to answer specific queries (e.g. how much education assistance is the US providing to Kenya) without downloading the full dataset).

Guidance: website provides some explanation on terminology used; but further clarity on US-specific budget terminology could make this clearer to external audience.

33

DFID Development Tracker

Clear homepage with a search bar allowing to search projects by keyword

Top statistics and achievements are useful for those looking for an overview of UK aid

Ease of use: Good

Quick access to searching for projects by location or by sector, with some data visualization

34DFID Development Tracker Micro: For more detail, a country’s project

list can be filtered by status, sector, agency, budget, and date, and directs the user to project factsheets. Project factsheets also contain separate tabs with information on transactions and project documents.

Relevance of micro content:

Good

Macro: Presents summary information on the country before going to the project-by-project detail: total budget and total number of projects; also allows to search for regional projects

Relevance of macro content:

Good

Coverage: Poor – only one donorTimeliness: Good – information on planned projects up to start date

of 2018Relevance of content: Good

Each country has a fact sheet with summary of ongoing projects by budget and sector

35

DFID Development Tracker

Macro: Presents summary information on the each sector (share of total budget) before going to the sector-by-sector detail: also allows to sort sectors by name and budget %

Each sector is organized into detailed sub sectors

Relevance of micro content:

Good

Relevance of macro content:

Good

Micro: Each sub-sector has a project list that can be filtered by status, sector, agency, budget, and date, and directs the user to individual project factsheet. Project factsheets also contain separate tabs with information on transactions and project documents. Importantly, there is also information on implementing organizations (i.e. subcontractors). This level of detail is unique amongst the websites reviewed, and important for traceability.

Coverage: Poor – only one donorTimeliness: Good – information on planned projects up to start date

of 2018Relevance of content: Good

36

DFID Development Tracker

However, this link leads to an xml file that requires a CSV conversion tool to transfer into Excel. The website should make clear to users how to use this tool, or should provide data directly in Excel (insufficient guidance).

Data for technical

users: Bad

Each project and country factsheet has a button allowing download of IATI data for that project

The only option to download the full dataset is to access the Annual Report Excel files, which are poorly advertised on the main website. Main website has no functionality to sort and filter data other than at the project level. For example an everyday user looking for the budget going to a specific country and sector would need to manually add the budgets of all individual projects in that category.

Data for everyday

users: Moderate

CONCLUDING THOUGHTS

38

EASE OF USE

RELEVANCE OF CONTENT

– MICRO

RELEVANCE OF CONTENT

– MACRO

DATA FOR TECHNICAL

USERS

DATA FOR EVERYDAY

USERS

AidData

OECD Creditor Reporting System

US Foreign Assistance Dashboard

DFID Development Tracker

OpenAidNL

d-portal.org

AidView

Bad

Moderate

Good

Summary Table

39

Summarizing…• Visualization ≠ Value

» Data visualizations must be useful in addition to aesthetically pleasing; excessively image-heavy sites can be difficult to navigate with limited internet connection

• Aggregation is important

» It is often difficult to find totals (by donor/recipient country or by sector), unless one downloads the entire dataset (if it is available on the site)

• Who is the audience?

» Everyday Users versus Tech-Savvy users

» Sometimes it can’t (or shouldn’t) be both… but this should be made clear at the outset

• Looking forward…

» Planned in addition to disbursed aid – so recipients can plan ahead (greater predictability)

» Outputs (project results) in addition to inputs (project budgets)

» Data can only be as robust as a donor presents it (must increase reporting)

40

Thank you!