FINAL EVALUATION REVISED REPORT - GOV UK · The final evaluation considers the change processes...

Itad 2019

FoodTrade East and Southern Africa DFID

FINAL EVALUATION

REVISED REPORT

Date: 05 April 2019

Submitted by Itad

Itad 2019

Disclaimer

The views expressed in this report are those of the evaluators. They do not represent those of any of the individuals and organisations referred to in the report.

‘Itad’ and the tri-colour triangles icon are a registered trademark of ITAD Limited.

FINAL EVALUATION

Page | iii

Contents

Introduction .......................................................................................................................................... 1

Purpose, objectives, audience and communications 1

Revisions to the terms of reference and any departures 2

Structure of the report 3

Programme background and ToC .......................................................................................................... 4

Background 4

ToC and expected results 6

Grant portfolio 9

Evaluation design and methods ...........................................................................................................10

EQs 10

Design 12

Evaluation modules 13

Data collection methods 14

Analysis and synthesis 17

Independence 18

Limitations and qualifications 19

Findings ................................................................................................................................................21

To what extent is FTESA a collection of individual interventions or a coherent portfolio (EQ1)? 21

To what extent have improved trade support systems increased production and trade (EQ3)? 26

To what extent have improved availability and use of inputs and application of good agricultural practices increased production and trade (EQ4)? 32

To what extent and how has FTESA brought (or facilitated) smallholder farmers into structured regional markets (EQ5)? 37

To what extent is FTESA likely to improve the functioning of national and regional staple food markets and generate systemic change (EQ2)? 46

To what extent has FTESA benefited consumers (EQ6)? 51

To what extent have FTESA approaches to supporting reform to relevant policies, regulations, etc. contributed to change (EQ7)? 52

Does FTESA offer VfM in the results it achieves, compared with possible alternatives (EQ8)? 55

Main findings and conclusions .............................................................................................................59

Lessons and recommendations ............................................................................................................63

Annex 1: Common and output theories of change .......................................................................................66

Annex 2: Evaluation matrix ..........................................................................................................................69

Annex 3: Methodology .................................................................................................................................73

Annex 4: Evaluation questions and main modules .......................................................................................80

FINAL EVALUATION

Page | iv

Annex 5: Generic data collection templates .................................................................................................81

Annex 6: Module evidence assessment frameworks ...................................................................................82

Annex 7: Strength of evidence and triangulation .........................................................................................83

Annex 8: Programme context ......................................................................................................................84

Annex 9: Results against DFID logframe .......................................................................................................87

Annex 10: Grant portfolio ............................................................................................................................89

Annex 11: Grant summaries .........................................................................................................................93

Annex 12: Key documents consulted ...........................................................................................................97

Annex 13: Final Evaluation Terms of Reference .........................................................................................100

FINAL EVALUATION

Page | v

Figures, tables and boxes

Figure 1: FTESA common theory of change ............................................................................................. 7 Figure 2: Evaluation sequence/structure ............................................................................................... 10 Figure 3: Modular evaluation design ..................................................................................................... 13

Table 1: Evaluation criteria ................................................................................................................... 12 Table 2: VfM by metric.......................................................................................................................... 56

Box 1: Joseph Initiative ......................................................................................................................... 38 Box 2: Challenges developing and rolling out G-Soko ............................................................................ 40 Box 3: Effects on poverty (Joseph and Kaderes) ................................................................................... 42 Box 4: Gendered benefits ..................................................................................................................... 44 Box 5: FTESA grantee capacity .............................................................................................................. 50

FINAL EVALUATION

Page | vi

Acronyms

AAER Adopt-Adapt-Expand-Respond

ACTESA Alliance for Commodity Trade in East and Southern Africa

AGRA Alliance for a Green Revolution in Africa

CF Challenge fund

COMESA Common Market for Eastern and Southern Africa

CToC Common Theory of Change

CUTS Consumer Unity & Trust Society

DF Development fund

DFID Department for International Development (UK)

EAC East African Community

EAGC East African Grain Council

EATIH East Africa Trade and Investment Hub

EMU Evaluation Management Unit

ENAS Ets Nkubili Alfred & Sons

EQ Evaluation Question

FAO Food and Agriculture Organisation

FRA Food Reserve Agency

FTESA FoodTrade East and Southern Africa

G-Soko Grain soko (market)

IAPRI Indaba Agricultural Policy Research Institute

M&E Monitoring and Evaluation

MoU Memorandum of Understanding

MT Metric tonne

NGO Non-Governmental Organisation

PICS Purdue Improved Crop Storage

PMU Programme Management Unit

PPI Poverty Probability Index

RUDI Rural Urban Development Initiatives

SACCO Savings and Credit Cooperative Organisation

SADC Southern African Development Community

SFR Strategic Food Reserve

SOPAG Soya Policy Action Group

ToC Theory of Change

USAID United States Agency for International Development

VfM Value for money

WAFM West Africa Food Markets

WFP World Food Programme

WRS Warehouse Receipt System

FINAL EVALUATION

Page | vii

Executive summary

Programme, context and theory of change

FoodTrade East and Southern Africa (FTESA) was a five-year (2013–2018) regional programme funded by the UK Department for International Development (DFID) (£35 million) to support food staples market development and trade by tackling market failures. A Programme Management Unit (PMU), contracted to DAI, was responsible for managing and supporting programme implementation. FTESA aimed to catalyse lasting changes that enable efficient trade in staple foods across the region to improve the lives of farmers, suppliers, service providers, traders, retailers and consumers, contributing to price and market stability for staple foods in the region. FTESA aimed to invest in systems that allow small-scale farmers to access regional grain markets. The programme worked with the private sector and relevant institutions to tackle a constraining set of market failures in the following areas:

• Output 1: Improved post-harvest markets (storage and aggregation; market information; value chain coordination; warehouse receipts and supplier credit; grades and standards)

• Output 2: Improved input markets (including seeds and fertilisers)

• Output 3: Improved trade environment and reduced uncertainty

FTESA’s intended programme scope included nine countries across East and Southern Africa. However, the programme’s operations and grant coverage focused largely on four countries in East Africa (Kenya, Rwanda, Tanzania and Uganda) and one in Southern Africa (Zambia). The regions and countries where FTESA operated are characterised by a vast range of challenges affecting the staple food market. Drawing on the Evaluation Management Unit’s (EMU) baseline and mid-term phases, we summarise some of these below. Annex 8 provides further details on the country context for the main FTESA countries in which the programme operated.

• Significant post-harvest losses and inadequate storage and aggregation systems: Farmers experience significant post-harvest losses largely due to a lack of access to good-quality storage and aggregation systems, reducing volumes for sale. Limited understanding of the benefits of improving quality and aggregating produce, as well as lack of skills and capital to improve quality (including applying grades and standards), leads many smallholder farmers to act as price takers, selling small quantities of grain at the farm gate to traders at low prices.

• Limited access to capital: Lack of functioning inventory credit systems, such as warehouse receipt systems, limit opportunities to use stock as collateral – reducing the ability to store and defer sales until prices rise and reducing access to capital to invest in more productive farming (e.g. improved inputs).

• Limited productivity and production: Limited access to improved inputs (including lack of improved seeds in the market) and advice on good agricultural practices hampers productivity and the quantity and quality of production.

• Lack of access to markets: Small-scale production, limited aggregation and the disparate nature of smallholders increase transaction costs for buyers and sellers. The lack of aggregation of grain that reaches quality and quantities that provide convenience for buyers reduces access to a wider range of buyers willing to compete for higher quantities and quality, as well as limiting the bargaining power of smallholder farmers in a way that again leads them to be price takers.

• Lack of information and transparency: Lack of information, particularly on prices, leads to uncertainty and risk, limits bargaining power and reduces the ability to make informed decisions on when, where and who to sell to, leading to market inefficiency.

• Unpredictable government policies and administrative hurdles: Government interference can lead to unpredictable markets, creating distortions that either inflate or depress prices away from market levels and introducing both supply and demand disincentives into the market. A lack of predictability reduces the incentive to invest in improved practices, including storing grain at harvest in anticipation of higher prices

FINAL EVALUATION

Page | viii

later. Moreover, the persistence of trade barriers, including unpredictable export bans, between countries (despite the regional integration commitments made through the East African Community (EAC), Common Market for Eastern and Southern Africa (COMESA) and Southern African Development Community (SADC)) curtails the potential gains from greater regional integration and trade, including reducing the potential gains to both producers and consumers of moving produce from surplus to deficit areas.

Stated simply, FTESA’s theory of change (ToC) was that by identifying and addressing market and government failures that characterise staple food markets, including improving farmers’ capacity to produce, store and aggregate grain, and market more staple food, this will lead to ‘more staple food traded and more people benefiting from participation in national and cross-border value chains (outcome)’, which in turn will contribute to the ‘improved functioning of national and regional staple food markets systems including more stable food prices across the region’ (impact), thus benefiting both producers and consumers.

Evaluation

Between Q2 and Q4 2018, the EMU, managed by Itad UK, undertook the independent final evaluation. The final evaluation is summative and theory-based and its purpose is to focus on accountability (programme results) and lesson learning. The objectives of the evaluation are to generate information on the performance of the overall programme and accountability for funds spent, assessing whether the programme brought about the change expected, as well as lessons learned, exploring how and why some interventions were successful and others not, to feed in lessons to the next programme and similar programmes, including recommendations on how to improve performance in future. The intended audience is DFID and implementing partners. For DFID, the final evaluation is an important tool to provide lessons and inform its thinking on existing and future similar programmes.

The final evaluation considers the change processes identified in the overall ToC, as well as the more detailed theories for each output area, and the extent to which interventions have affected these change processes. We apply a modular evaluation design that responds to the evaluation questions (EQs):

1. Case synthesis examining FTESA’s progress against EQs 1–6, centred on a sample of grants.

2. Thematic studies examining FTESA’s progress against EQs 1–6, with a focus on questions related to systemic change and sustainability, centred on two grants.

3. Policy dialogue and influencing studies exploring FTESA’s policy-influencing efforts, focusing on specific issues.

4. Portfolio review drawing on secondary information across the full portfolio of grants against the EQs.

5. A value for money (VfM) assessment exploring the VfM of the programme.

Together these examine:

• Different levels of the FTESA programme, exploring individual grants, interconnected and complementary grants, the overall grant portfolio and the overall programme.

• Different levels in the market system, exploring the role of different market actors and their interactions with FTESA-funded interventions.

Main findings and conclusions

To what extent is FTESA a collection of individual interventions or a coherent portfolio (EQ1)?

The FTESA portfolio of grants offered opportunities for projects to benefit from complementarities and synergies between them. There is some evidence of interlinkages and complementarities materialising between FTESA grantees that have generated results more than those achievable through an individual grant, where some grantees were able to tap into support provided by others to increase the impact of their interventions (including combining support at different points along the value chain). Partnerships have enabled grantees to capitalise on each other’s services. However, there are concerns regarding the

FINAL EVALUATION

Page | ix

sustainability of some of these interactions, particularly those reliant on development fund interventions rather than commercial entities.

The most frequently cited connection is between grantees and the East African Grain Council (EAGC). The PMU focused much of its efforts on building interlinkages and complementarities across the portfolio around the EAGC grant, especially warehouse certification and the G-Soko platform, with the intention that several different FTESA-funded grantees would use the G-Soko platform and create a group of early adopters. However, the failure of G-Soko to take off successfully and demonstrate consistent results through early adopters significantly reduced the opportunity for synergies and seriously hampered FTESA’s ability to create a portfolio that delivered more than the sum of its parts through grantees using the platform and accessing larger markets, with others copying, thus limiting sustainability and curtailing the impact of the FTESA programme in the wider market.

Linkages between FTESA grantees were in part facilitated by the PMU but were also due to a grantee’s knowledge of the other grantees, existing relationships and/or searching out synergies themselves. The PMU’s model restricted the programme’s ability to design strong interlinkages and build a coherent, comprehensive portfolio. In several cases, the PMU attempted to build linkages into the design of projects after awarding the grants. With relatively few rounds and lack of real-time monitoring and learning built into the programme design and decision-making, there were limited opportunities to quickly learn from previous rounds and tailor future rounds to attempt to generate interlinkages, building on existing projects to improve the performance of the overall portfolio. Moreover, the PMU itself was not set up with the level of technical assistance required to provide substantive design inputs.

Strength of evidence: medium

To what extent have improved trade support systems increased production and trade (EQ3)?

Overall, evidence of FTESA-led improvements in trade support systems shows mixed success, partly due to delays in implementation. This includes instances where some interventions were reliant on the completion of other activities (e.g. a warehouse receipt system (WRS) requires the warehouse to be in place to the required standards). There were also short timeframes in which results were expected to transpire. There was good progress and achievements on activities that set the foundations for improvements in trade support systems. Despite differences in context and implementation models, several of the grants helped farmers reduce post-harvest losses, increase the volumes and quality of produce stored and aggregated, as well as improving farmers’ position in the market, but the results fall short of expectations. Training on post-harvest handling, in combination with the incentive of higher prices for aggregated better-quality produce, helped farmers understand the value of improving post-harvest handling and aggregating produce, as well as motivated farmers to improve post-harvest handling and store and aggregate their produce with others, reaching required standards, and marketing collectively to access better markets, given the right incentives and demonstrated benefits – namely, better prices and market access.

However, in many cases the aggregation volumes fell short of the volumes expected. Barriers to storage and aggregation have curtailed the achievement of expected results, including construction delays, smallholder farmers’ preference for immediate payment, issues of trust, transport distances and costs, and lack of better markets. The case study evidence shows that some of the grants were unable to deliver key output milestones within the relatively short timeframes. Also, access to finance remains a major challenge, with considerable institutional barriers existing in the financial markets, where in many cases banks remain risk averse about lending to smallholder farmers, with the existence of collateral alone not sufficient to engender confidence.

Strength of evidence: strong

To what extent have improved availability and use of inputs and application of good agricultural practices increased production and trade (EQ4)?

Several cases present good evidence of improvements in the availability and use of inputs and farmers simultaneously applying good agricultural practices due to FTESA-funded activities, although the numbers

FINAL EVALUATION

Page | x

reached are low, with supplies of inputs and numbers trained lower than expected. Most progress is in training to support the application of good agricultural practices. Where farmers have applied good agricultural practices and used improved inputs, productivity and quality has improved. There is strong evidence across the cases that farmers are willing to adopt new/improved inputs and practices where the benefits are clear (demonstration effects and proof of concept) and are in line with a specific farmer’s own risk appetite.

The use of known and respected institutions (grantees, research institutions, input suppliers and local savings and credit cooperative organisations (SACCOs)) to mobilise farmers enhanced the credibility of the intervention and generated trust, especially in contexts where farmers were risk averse and had lower levels of education. This led to greater participation rates and adoption of new and/or improved inputs and practices. The use of the lead farmer approach worked well when farmers considered the lead farmer to be experienced and successful, increasing their credibility, with farmers more likely to apply lessons from training provided by such farmers.

In some cases, gaps in service provision, such as lack of finance, reduced the uptake of new and/or improved inputs. Also, despite efforts to improve access to quality inputs, delays in accessing inputs at the required quantity and quality limited the benefits. Factors outside of the direct control of the programme, such as fake seeds, government policy leading to delays in accessing improved seed and weather-related issues, reduced the benefits of applying better practices. There is limited evidence across the grants that these interventions have led to higher-level results (prices, sales, incomes), which is partly due to the stage of implementation and lack of success in finding markets (see EQ5).


To what extent and how has FTESA brought (or facilitated) smallholder farmers into structured regional markets (EQ5)?

Increased smallholder farmer participation in structured regional markets was a central focus for FTESA and grantees. Most of the grantees worked with smallholder farmers with existing or potential tradeable surpluses, with women accounting for approximately 40% of participants. Several of the grants helped smallholder farmers improve yields, production and quality, as well as store and aggregate greater volumes, making them ‘market-ready’ and able to sell to a wider range of buyers. Unfortunately, lack of data and information on specific groups limited the analysis on these different groups.

The evidence strongly suggests that lack of storage, aggregation and collective marketing (e.g. selling as a group) form a fundamental barrier to improving farmers’ position in the market and their ability to command higher prices. Several cases show that the interventions helped farmers understand that storing produce and deferring sales can lead to higher prices and aggregating good-quality produce and marketing collectively can lead to access to better markets as buyers prefer the convenience of purchasing aggregated produce that reaches required standards. The cases provide evidence of changes in the mindset of participating farmers toward aggregation and selling as a group, and examples of farmers’ bargaining power and ability to negotiate better prices increasing when farmers increased and improved aggregation and sold as a group, creating competition among buyers for their produce. In some cases, the programme facilitated connections between farmer groups and buyers, leading to deals, increased sales and better prices.

However, the integration of smallholder farmers into structured regional markets was not widespread. The programme was unable to reach the scale and levels of volume and trade required to have an impact at the regional level, including influencing market prices, partly given underperformance but largely due to unrealistic expectations. Access to new and better markets was a challenge for many farmers during the programme’s timeframe. Nevertheless, FTESA helped ‘lay the foundations’ for greater integration in future.


FINAL EVALUATION

Page | xi

To what extent is FTESA likely to improve the functioning of national and regional staple food markets and generate systemic change (EQ2)?

The grants show limited evidence of systemic change and only early signs of the potential for spreading new behaviours to others. This is largely due to the stage of implementation at the time of the evaluation, where several activities are yet to demonstrate consistent and enduring benefits to participants and therefore these have not yet been observed and spread to non-participants. While there is evidence that farmers have adopted new ways of doing business and accessed new markets, in many cases there was over-reliance on support from the grantee for inputs, services and market access, limiting sustainability and impact in the future. Some grantees did not provide support for interventions for long enough to deliver the ‘critical mass’ of consistent success required to build buy-in for the intervention and encourage others to crowd in. However, there is anecdotal evidence that FTESA has generated systemic change in some cases, as indicated by examples of behaviour change in smallholder farmers (e.g. adoption of new methods and some copying by others) and buyers offering better prices. Again, enablers of behaviour change include transparency and trust between market actors. The most frequently cited barriers were the absence of supporting rules and limited capital.


To what extent has FTESA benefited consumers (EQ6)?

Given the limited scale of most of the interventions, including in terms of their geographical reach (several projects only had a limited footprint across the region), the programme has not generated the substantial volumes required to pass through the market to lead to price smoothing at a regional level, partly due to the underperformance of G-Soko. While there is no systematic reporting on the benefits to the end consumer, there is anecdotal evidence that grantees are producing improved quality and value-added products.

Strength of evidence: low

To what extent have FTESA approaches to supporting reform to relevant policies, regulations, etc. contributed to change (EQ7)?

Except for one case, the intended results of policy-influencing efforts are yet to materialise – which is partly due to long drawn-out political and bureaucratic processes outside of the control of the programme. However, even where they did materialise, there are significant risks of policy reversal.

FTESA’s support to policy-influencing efforts has helped remove export bans and set processes in motion. There is strong evidence that taking an evidence-based approach and working through partners were critical factors for successful influencing of policies and regulations in both Kenyan and Zambian policy case studies. Across the cases, no one activity type was better than another. Rather, each policy issue required a number of activities that combine and interact, building on each other. Examples include producing technical analyses that fed into position papers by key partners, to be shared with policy-makers, and presented and discussed in public–private dialogues.

In Kenya, through its persistent engagement with constituents who are seeking change and those who the programme sought to influence, FTESA set in motion processes that may lead to changes with positive repercussions for the market, particularly where EAGC continues its policy-influencing efforts. However, in Zambia the inactivity of the Soya Policy Action Group (SOPAG) since the programme closure, and the exclusion of FTESA partners in policy processes, threatens the sustainability of efforts.


Does FTESA offer VfM in the results it achieves, compared with possible alternatives (EQ8)?

Based on the VfM metrics analysed and benchmarking conducted, there are several areas where the programme performed well in comparison to other similar programmes, as well as improvements in some of the metrics since 2016. VfM metrics on economy improved after the mid-term evaluation in 2016, and FTESA performs better than comparable programmes such as West Africa Food Markets (WAFM). The smallholder engagement cost (efficiency) increased between 2016 and 2018, although it remains within the range of other

FINAL EVALUATION

Page | xii

comparable programmes. Portfolio leverage rates (efficiency) improved and are higher than similar programmes. The volume of sales contributed per beneficiary farmer (effectiveness) increased between 2016 and 2018, most likely due to progress in implementation, with results transpiring further along the value chain, as well as the addition of new grants in 2016 that generated substantial sales volumes (e.g. Farm Africa and the World Food Programme (WFP)). The cost of female outreach (equity) was significantly higher than overall reach, suggesting that achieving equity is a cost driver, but it is within the range of other comparable programmes.

The PMU was slow to develop VfM indicators; it only started collecting data in 2016 and did not set any targets. There was no evidence to suggest that the PMU was using VfM data to feed into ongoing analysis and learning by programme teams and at programme close the PMU provided limited assessment of the VfM data.


Lessons and recommendations

The following are lessons and subsequent recommendations drawn from the findings and conclusions of the evaluation, including those on how market development programmes can promote structured trade and systemic change. They are ordered according to the top three priority areas DFID has identified, followed by additional recommendations from the evaluators for similar, future programmes. Some priority areas contain multiple lessons and recommendations.

1. Bringing smallholder farmers into structured regional markets

a) Improved post-harvest markets: access to finance

Lesson: While collateral can help farmers to access credit (one of the original programme assumptions), the experience across the grants shows a wider range of conditions necessary to improve the bankability of smallholder farmers and their access to credit, including: (i) lending to a registered farmer-based organisation (not individuals) that can provide a group guarantee; (ii) trusted suppliers providing inputs and evidence of good quality training conducted; (iii) farmer-based organisations’ exposure to formal sales demonstrated through contracts with buyers; (iv) evidence of high loan repayment rates.

Recommendation: Similar programmes should take a more comprehensive, multi-faceted approach to improving access to finance, focusing on improving the credit worthiness of farmers.

b) Systemic Change: Transparency and trust

Lesson: Farmers altered and improved their practices when there was transparency and trust between the farmers and FTESA grantees, and confidence in the viability of the intervention. Conversely, farmers’ negative experiences with buyers and agro-dealers in the past led to entrenched negative perceptions of some actors in the value chain.

Recommendation: Similar programmes should give adequate attention to the demand-side and to facilitating relationships and contractual arrangements that are open, transparent and based on regular communication which incentivises both parties – the buyer and seller – to fully honour their commitments in a timely manner. They should also allow enough time for interventions to get up-and-running and broker relationships (through trusted partners) across the value chain.

c) Systemic Change: Positive demonstration effects

Lesson: Demonstration effects were important catalysts to generating trust in the intervention and increasing uptake of unproven methods and crops. Changes in practices endure where farmers experienced ‘proof of concept’ (e.g. where new practices led to increased yields) including securing better markets and associated benefits consistently.

Farmers typically require follow-up demonstrations and repeated interactions with trainers to build confidence in adopting new crops, inputs and methods, as well as to help adapt practices as challenges arise over time.

FINAL EVALUATION

Page | xiii

Recommendation: Similar programmes must allow enough time for early adopters to demonstrate benefits to others for wide-spread adoption to occur.

2. Reform to food trade policies and regulations

Lesson: The FTESA experience shows that working through local actors is most effective, particularly those embedded in the local context who can build coalitions and search out like-minded actors and who know who to target and how, and who have strong incentives to reduce barriers to improving the market system. This also helps to ensure accountability, effectiveness and sustainability. This is applicable to other areas of the programme, where developing strategic partnerships with those strongly motivated to change the market system (e.g. partners already established and embedded in the local political context affected by the detrimental effects of market inefficiencies) may lead to long-term sustainable impact.

Recommendation: Similar programmes should ensure that they work with local partners who have the incentive and therefore motivation to continue policy influencing engagements and activities, and those who already have influence, particularly those representing constituents most affected by the policy and regulatory constraints.

3. Portfolio inter-linkages and complementarities

Lesson: Award-based mechanisms, by design, can limit the ability to directly build in strong inter-linkages and complementarities across the portfolio.

Recommendation: DFID should ensure that the design of future portfolio-approach programmes, which rely on inter-linkages and complementarities to generate expected results, includes more active hands-on support from PMUs (or similar) in designing projects when needed. This requires a greater investment in technical assistance.

It also requires mechanisms to generate real-time learning and foster coordination and collaboration between implementing partners. This may require alternative models to award-based mechanisms, with PMUs taking a more active role in shaping and designing the portfolio of projects to improve coherence and complementarities through active and ongoing learning.

4. Additional recommendations

a) FTESA’s Value for Money (VfM)

Lesson: The existing VfM metrics are suitable for the assessment but we were unable to find comparable indicators on effectiveness across DFID programmes, which suggests that in future there needs to be more careful selection of appropriate indicators that allow for comparison (e.g. jobs created, etc.). Moreover, future programmes need to assign targets to indicators to track progress and provide incentives to achieve these, as we recommended in the mid-term evaluation.

Recommendation: At the outset, programmes need complete VfM frameworks with metrics that: are comparable with similar programmes; provide adequate coverage of equity; align to the logframe and targets; and outline clear definitions and plans on how and who will collect and analyse data. Such plans need consistent implementation and should feed into ongoing learning and decision-making by programme teams.

b) Systemic Change: Crowding in and sustainability

Lesson: Sustainability hinges on a continuous supply of improved inputs from permanent market actors (e.g. agro-dealers) and a continued demand for them from farmers. It also relies on permanent market actors (e.g. agro-input suppliers, buyers) repeating training and consistently demonstrating benefits, with farmers seeing the benefit. Ultimately farmers should be willing and able to pay for inputs and services if they experience consistent benefits, and market actors therefore have a commercial incentive to continue these functions. As a result, farmers adopt new behaviours and these spread to others as other farmers observe the benefits, spreading the new behaviours further afield through demonstration effects.

FINAL EVALUATION

Page | xiv

Recommendation: The assumption that the benefits generated through interventions catalyse longer term, widespread changes that are sustainable is only likely where programmes give adequate attention to the following:

• Facilitating mechanisms for continual updating of knowledge and learning (for farmers and businesses) to ensure better practices continue and can adapt to the external environment (e.g. new technologies, threats, etc.), alongside consistent positive demonstration effects that lead to wider adoption rates.

• Crowding in other commercial actors to the system who provide incentives for farmers to maintain and continually improve changed practices.

• Ensuring grantees (or similar) develop effective exit strategies, and permanent market actors have the incentive to provide, scale and adapt services.

c) Systemic Change: Scaling up, timeframes and targets

Lesson: Rolling out and attempting to scale up interventions quickly often leads to insufficient time to engage users, experiment, pilot, learn and adapt interventions, as was the case with the FTESA-funded G-Soko electronic trading platform.

Recommendation: Similar programmes should take a more cautious approach to scaling up, by extending timeframes and reducing targets, giving time for experimenting, piloting, learning, adaptation etc. and demonstrating results before scaling up. Projects should not (and cannot successfully) attempt to pilot and scale-up simultaneously.

Programmes should also give more attention to the time required to roll out activities and deliver results (e.g. some interventions covered only one or two harvests) and the sequencing of different elements of an intervention (e.g. constructing a warehouse and then establishing a warehouse receipt system).

Moreover, similar programmes attempting to generate change in the wider market system should identify strategic partnerships and leverage a wider network of players already working in the same fields.

FINAL EVALUATION

Page | 1

Introduction1

Between Q2 and Q4 2018, the EMU, managed by Itad UK, undertook the independent final evaluation of the FTESA programme managed by the PMU set up by DAI and funded by DFID. The FTESA programme closed in April 2018. This report presents the findings of several different evaluation modules undertaken.

Purpose, objectives, audience and communications

The purpose of the final evaluation is accountability (programme results) and lesson learning, as detailed in the terms of reference (see Annex 13 for further details). The objectives of the evaluation are to generate information on the performance of the overall programme and accountability for funds spent, assessing whether the programme brought about the change expected, as well as lessons learned, exploring how and why some interventions were successful (in terms of design and implementation) and others not, to feed in lessons to the next programme and similar programmes, including recommendations on how to improve performance in future.

The evaluation is summative, following a theory-based approach, exploring what change materialised (or may materialise in the future), the extent of that change and FTESA’s contribution, and how and why, for whom and in what circumstances, and whether any changes will endure after programme closure (sustainability). The theory-based approach provides evidence (where available) of ‘proof of concept’, i.e. whether the underlying theories behind such programmes hold true. The final evaluation considers the change processes identified in the ToC, as well as the more detailed theories, and the extent to which interventions have affected these change processes or may do so in future.

In this report, we combine, compare and contrast the evidence from the different evaluation modules against the expected results, EQs and underlying theories. The report identifies and explores emerging patterns and themes to produce findings, conclusions, lessons and recommendations against eight EQs.

The main audience is DFID and the FTESA implementing partners. The evaluation generates wider learning for DFID (and potentially others) on market development programmes, including informing the scaling up and/or rolling out of similar programmes, such as FTESA’s successor programme (FoodTrade and Resilience Africa) and FTESA’s sister WAFM programme.

To ensure that the evaluation is useful, and therefore used by the intended audience, the evaluation team liaised closely with DFID to revise the final evaluation terms of reference (see section 1.2). Also, during the data collection and analysis process, the evaluation team engaged with both the PMU and DFID, checking understanding and presenting preliminary findings, without compromising the independence of the evaluation (see section 3.6). However, the PMU’s contract ended in April 2018, limiting the EMU’s ability to share preliminary findings with the PMU. We shared with DFID various reports across the modules before drafting this final evaluation report.

The team presented the initial final evaluation report findings to DFID in November 2018, having shared earlier drafts of the various reports for comments. This will be followed by a workshop in January 2019. The workshop will provide an opportunity to discuss findings, conclusions, lessons learned and recommendations, with the aim of co-creating the final conclusions, lessons learned and recommendations with DFID. The team will also produce products tailored to a wider audience, after the approval of the contents of this report.

1 Liz Turner (overall Team Leader; EAGC case and thematic study) prepared the report, with inputs from Jen Leavy (case studies; synthesis), Jessica Rust

Smith (portfolio review), Daniel Joloba (case studies; Kenya policy dialogue and influencing study), Phil Compernolle (Zambia policy dialogue and influencing study), Mark Winters (Farm Africa thematic study) and Gulden Bayaz (value for money assessment), as well as additional inputs on market system change from Gordon Freer (Team Leader, EMU, WAFM).

FINAL EVALUATION

Page | 2

Revisions to the terms of reference and any departures

The EMU discussed and agreed with DFID all the revisions and departures in the terms of reference outlined below.

Mid-term: In April/May 2016, the EMU updated the original EQs to reflect the needs of DFID, based on inputs from DFID and the evaluation team. As outlined in the EMU Inception Report and original terms of reference, the mid-term evaluation intended to draw on a sample of baseline surveys and studies (at the grant level), as well as an organisational review and VfM assessment conducted at mid-term. After reviewing the original evaluation design and its usefulness (including whether it addresses the needs of the client), the EMU team and DFID identified the following gaps: the absence of any portfolio-wide assessment in the mid-term and final evaluations; data collection at mid-term being limited to a PMU organisational review and VfM assessment; and no provision to assess the complementarity and synergies across the portfolio. This led to the addition of the thematic study (including fieldwork) to explore core interlinked grants (centred around the G-Soko platform) and a portfolio review to provide an assessment of the full portfolio. Working closely with DFID, we revised the terms of reference in May 2016.2

Final evaluation: In December 2017, the EMU updated the EQs in consultation with DFID to reflect the summative nature of the final evaluation, updating the final evaluation terms of reference accordingly (approved in March 2018). The questions provided similar coverage to the mid-term, but the wording of some was changed and some became sub-questions. The questions reflect those in the original terms of reference and inception report but provide a lot more detail and additional questions. Compared to the timing in the original terms of reference, the EMU conducted the final evaluation after the programme closed, rather than before, although some of the data collection commenced before the programme closed in April 2018. There was an expectation that the PMU would receive a six-month no-cost extension, as had been the case with the EMU’s no-cost contract extension to November 2018. However, this did not happen, which is why the evaluation took place post-closure. Nevertheless, both the EMU and DFID saw advantages in doing the evaluation after the programme closed since more time had elapsed to assess the results and explore the sustainability of the interventions.

Departures from the final evaluation terms of reference: DFID requested that the EMU change the scope and focus of the thematic study. Rather than focus only on EAGC and G-Soko and interlinkages between a core group of complementary grants, DFID requested an additional study to cover Farm Africa. The EAGC and G-Soko grant underperformed and DFID had already undertaken extensive studies of the grant, with the effect that DFID requested that we continue with the EAGC/G-Soko case study but dedicate fewer resources to EAGC/G-Soko as part of the thematic study and free up resources for a specific study of Farm Africa. Moreover, the reporting on the Farm Africa grant demonstrated more success than other grants, hence DFID expressed a desire to explore what was working, how and why, to provide lessons for similar and future programmes. Both the EAGC/G-Soko and Farm Africa studies still looked at the interlinkages between the ‘anchor’ grant and other FTESA grants. The EAGC/G-Soko case study explored interlinkages with other grants such as Shalem and Classic, but the Farm Africa data collection and analysis showed little linkages with other grantees. Also, it is worth noting that all case studies and the portfolio review explored any interlinkages between grantees and grants under EQ1.

Other departures include the downgrading of the Kaderes quantitative survey at endline to a monitoring survey (i.e. excluding the control group surveyed at baseline), and undertaking a before-and-after analysis instead of a full impact assessment, given that the main intervention was not yet operational. DFID and the EMU discussed how the lack of the main intervention significantly reduced the usefulness of the survey and did not provide VfM. However, the EMU and DFID agreed that a pared-down version of the survey would still provide some useful information that the qualitative case study could build on.

2 Itad/DFID (2016) FTESA Mid-term Evaluation Terms of Reference; Itad (2017) FTESA Mid-term Evaluation Report.

FINAL EVALUATION

Page | 3

Structure of the report

The report is organised in seven sections and has several annexes:

• Section 1 provides the evaluation purpose, objectives, audience and communications

• Section 2 presents the programme background, challenges the programme sought to address, ToC and results areas, as well as a brief analysis of the grant portfolio

• Section 3 outlines the evaluation design and methods, including limitations and qualifications

• Section 4 presents the findings

• Section 5 provides main findings and conclusions

• Section 6 provides lessons and recommendations

• The annexes are as follows: 1. Common and output ToCs and detailed theories; 2. Evaluation matrix; 3. Methodology; 4. Evaluation questions and main modules; 5. Generic data collection templates; 6. Module evidence assessment frameworks; 7. Strength of evidence and extent of triangulation; 8. Programme context; 9. DFID logframe; 10. Portfolio data; 11. Grantee summaries; 12. Documents consulted; and 13. Final Evaluation Terms of Reference.

FINAL EVALUATION

Page | 4

Programme background and ToC

Background

Overview of the programme

FTESA was a five-year (2013–2018) trade enhancement and promotion programme funded by DFID (£35 million) to support food staples market development and tackle market failures. It chiefly covered four countries in East Africa (Kenya, Rwanda, Tanzania and Uganda) and one in Southern Africa (Zambia), although it originally planned to operate in nine countries (Burundi, Kenya, Malawi, Mozambique, Rwanda, Tanzania, Uganda, Zambia and Zimbabwe). It focused on staple food crops, especially maize, rice and beans (including soya). The programme aimed to improve the functioning of national and regional staple food market systems and generate benefits for households and consumers by ensuring staple foods are more widely available at affordable, more stable prices. It operated through two main instruments:3

• The challenge fund (CF), which aimed to stimulate innovative private sector investment in staple food value chains; and

• The development fund (DF), which aimed to unblock systemic constraints within staples supply chains.

FTESA used a market-based approach to working with enterprises. It sought to increase production and trade in staple foods by addressing market imperfections and stimulating innovation. The programme aimed to improve market access for producers and suppliers within staples value chains and link producers to a wider customer base. FTESA focused on improving outreach and market access for smallholder farmers in national and regional markets.

A PMU, contracted to DAI, was responsible for managing and supporting programme implementation. The PMU’s main roles were:4

• Grants management: awarding and managing grants (grant selection, due diligence, contracting, disbursing funds, monitoring and verifying, etc.) under two main funding instruments.

• Technical resource: serving as a leading centre of thinking, providing technical assistance and learning (including monitoring and evaluation (M&E)).

• Broker: brokering relationships and influencing with the aim of delivering achievable policy and regulatory change.

The PMU awarded grants based on funding rounds, relying on promising applications surfacing, and using successive rounds of grant awards to complement earlier grants, connecting and complementing CF with DF grants as far as possible. The rationale for doing so was to develop greater coherence across the grants and therefore the programme, rather than having a portfolio of separate projects.

Regional context

The FTESA programme operated in the context of a growing appetite for greater regional integration across East and Southern Africa, with several regional trading arrangements to increase integration in EAC, COMESA and SADC, each at different stages of regional integration (e.g. free trade area, customs union and common

3 Through the challenge fund, FTESA supported for-profit companies to develop new business models that provide solutions to food staple market

failures while delivering commercial benefits as well as jobs, income and market access for smallholder farmers. The challenge fund aimed to lower investment risks by providing matching grants and concessional loans (between £150,000 and £1 million per grant). It also aimed to encourage the private sector to take on greater risk to invest. The requirement was that all innovations had both national and cross-border impact, supporting staple food trade in at least two of the nine focus countries. FTESA provided partial funding of up to 49% of an intervention. The development fund worked in parallel to the challenge fund, investing in governmental or non-governmental not-for-profit entities with innovative ideas that connect smallholder farmers to regional markets. It also invested in policy dialogue between private sector actors and policymakers. 4 Itad (2017) FTESA Mid-term Evaluation Report.

FINAL EVALUATION

Page | 5

market). In addition, many countries have signed the African Continental Free Trade Agreement, demonstrating a continued commitment to the integration agenda. Regional trade is steadily increasing in intensity over time: since the 1980s the share of regional exports in total exports has more than tripled. However, although the direction is favourable regional trade levels nonetheless remain relatively low.5

Across East and Southern Africa, several donors are actively engaged in facilitating regional integration, including through multi-donor regional integration programmes such as TradeMark East Africa (TMEA), which focuses on physical access to markets (e.g. improving border posts) and improving the enabling environment for regional trade.6 FTESA aimed to complement other programmes, such as TMEA, by focusing on food staples and specific markets and the constraints faced, supporting increased sales volumes of staple food nationally and regionally, moving produce from surplus to deficit areas (including across borders).

Challenges facing the staples food market

The regions and countries where FTESA operated are characterised by a vast range of challenges affecting the staple food market. Drawing on the EMU’s baseline and mid-term phases, we summarise some of these below. Annex 8 provides further details on the country context for the main FTESA countries where the programme operated.

• Significant post-harvest losses and inadequate storage and aggregation systems: Farmers experience significant post-harvest losses largely due to a lack of access to good-quality storage and aggregation systems, reducing volumes for sale. Limited understanding of the benefits of improving quality and aggregating produce, as well as lack of skills and capital to improve quality (including applying grades and standards), lead many smallholder farmers to act as price takers, selling small quantities of grain at the farm gate to traders at low prices.

• Limited access to capital: Lack of functioning inventory credit systems, such as WRSs, limit opportunities to use stock as collateral – thus reducing the ability to store and defer sales until prices rise in order to access capital to invest in more productive farming (e.g. improved inputs).

• Limited productivity and production: Limited access to improved inputs (including lack of improved seeds in the market) and advice on good agricultural practices hampers productivity and the quantity and quality of production.

• Lack of access to markets: Small-scale production, limited aggregation and the disparate nature of smallholders increases transaction costs for buyers and sellers. The lack of aggregation of grain that reaches quality and quantities that provide convenience for buyers reduces access to a wider range of buyers willing to compete for higher quantities and quality, also limiting the bargaining power of smallholder farmers and leading them to be price takers.

• Lack of information and transparency: Lack of information, particularly on prices, leads to uncertainty and risk, limits bargaining power and reduces the ability to make informed decisions on when, where and who to sell to, leading to market inefficiency.

• Unpredictable government policies and administrative hurdles: Government interference can lead to unpredictable markets, creating distortions that either inflate or depress prices away from market levels as well as both supply and demand disincentives in the market. A lack of predictability reduces the incentive to invest in improved practices, including storing grain at harvest in anticipation of higher prices later. Moreover, the persistence of trade barriers (including unpredictable export bans) between countries, despite the regional integration commitments made through the EAC, COMESA and SADC, curtails the potential gains from greater regional integration and trade, including reducing the potential gains to both producers and consumers of moving produce from surplus to deficit areas.

5 https://www.imf.org/external/pubs/ft/fandd/2018/09/sub-saharan-africa-regional-integration-arizala.htm 6 https://www.trademarkea.com

https://www.imf.org/external/pubs/ft/fandd/2018/09/sub-saharan-africa-regional-integration-arizala.htm

https://www.trademarkea.com/

FINAL EVALUATION

Page | 6

ToC and expected results

Overview

Figure 1 presents the FTESA Common ToC (CToC). It sets out the programme’s pathways of change around three interconnected output areas: 1) improved post-harvest markets; 2) improved input markets; and 3) improved policy and regulatory environment. The CToC and output ToCs (see Annex 1 for the latter) set out the hypothesised causal pathways between activities and expected results, including the long-term change that the programme sought to influence to deliver programme outputs, outcomes and impacts.

FTESA’s main target beneficiary group were smallholder farmers, with consumers expected to be secondary beneficiaries of more stable food prices. At first, the programme did not target specific groups of smallholder farmers; however, in the last 18 months the programme increased its focus on women.7 The intended beneficiaries are not the ‘poorest of the poor’ but rather those smallholder farmers with an existing tradeable surplus, or the potential to generate such a surplus, and who can increase that surplus.

Annex 2 presents the underlying detailed programme theories on how change happens, linked to the EQs, drawing on theory development work conducted as part of the baseline, mid-term and final evaluation phases. We developed these theories applying realist evaluation principles.8

Stated simply, FTESA’s CToC was as follows: by identifying and addressing market and government failures that characterise staple food markets, including improving farmers’ capacity to produce, store and aggregate grain, and market more staple food, this will lead to ‘more staple food traded and more people benefiting from participation in national and cross-border value chains (outcome)’, which in turn will contribute to the ‘improved functioning of national and regional staple food markets systems including more stable food prices across the region’ (impact). In particular:

• By connecting farmers to more efficient input and output markets and by improving the enabling environment for trade, this will help farmers produce, store and aggregate greater volumes and sell more produce at better (higher and/or more stable) prices. For instance, better storage and aggregation facilities, alongside better access to markets, will enable and incentivise farmers to increase production and storage, sell more, and receive better (higher or more stable) prices, enhancing the incentive to invest. Increased production and trade, and better prices, will lead to higher incomes for farmers. In contexts where farmers do not have access to good-quality facilities, are risk averse, face restricted markets and are price takers selling to middlemen, important underlying assumptions include, for example, that:

▪ Farmers will choose to store produce in facilities;

▪ Farmers find training credible and can and will put it into practice;

▪ Suppliers (e.g. inputs) have the confidence to provide inputs in advance of payment;

▪ Market actors trust WRSs linked to storage and aggregation facilities; and

▪ Farmers feel motivated to organise themselves to aggregate their produce.

7 Itad (2017) FTESA Mid-term Evaluation Report. 8 Westhorp (2014) Realist Impact Evaluation: An Introduction.

FINAL EVALUATION

Page | 7

Figure 1: The FTESA CToC

• By enabling buyers and sellers to make decisions with less risk, this will encourage farmers to increase the amounts produced and traded, increasing competition in the market system. Markets will become less prone to sudden shocks as they will work at a regional level, rather than as a series of disconnected national or sub-national systems, with produce moving from surplus to deficit areas. A well-functioning market will help mitigate the price fluctuations caused by poor harvests, bumper harvests and other local variables. This will enable farmers, traders and other market players to operate with less price risk, thus encouraging further investment in the market (a virtuous circle). In contexts with restricted market access and linkages between businesses (namely buyers and sellers) in the value chain and disjointed small-scale and individual farming operations, important underlying assumptions are, for example, that:

▪ By creating contractual obligations that guarantee purchases and provide prompt payments to farmers, farmers will have the confidence and incentive to invest in increasing production, quality and storage and aggregation;

▪ A critical mass of farmers aggregate quantities that satisfy the minimum quantity delivery requirements of many buyers;

▪ Buyers pay more for aggregated quantities and quality;

▪ WRSs deter farmers from quick sales of produce at the farm gate, encouraging storage and reducing post-harvest losses, leading to increased sales and better prices, as well as smoothed and/or increased incomes.

• The larger the pool of potential buyers and sellers and fewer barriers to production and trade, the more opportunity there is for food to flow from areas of surplus production (and low prices) to areas of demand (and high prices), thus mitigating price differences between them. An increase in staple food bought and sold in national markets will encourage more cross-border trade between areas of surplus and deficit. This will contribute, in turn, to the better functioning of regional markets, with more stable food prices across the region, benefiting both producers and consumers. As the performance of the market system continues to improve, this will manifest itself through increased competitiveness, higher trade volumes and improved provision of inputs and services, all adding further to the virtuous circle.

FINAL EVALUATION

Page | 8

• By identifying critical policy and regulatory challenges hindering market functioning and the interests of key players, key partners embedded in the local context with clout/influence take forward targeted policy engagements ‘nudging’ achievable policy and regulatory change in the right direction, reducing the policy and regulatory barriers to staple food markets, improving predictability and increasing market efficiency, and facilitating the achievement of expected results under outputs 1 and 2.

Impact, outcomes and outputs

As stated in the logframe, the goal (impact) of the FTESA programme was: improved functioning of national and regional staple food market systems. The overall outcome of the FTESA programme was: more staple food traded and more people benefit from participation in national and cross-border value chains.

Impact indicators

1. Percentage differential between hungry and harvest season prices for key food staples.

2. (a) Number of consumer households in areas with more stable intra-annual food prices; (b) Number of consumers in areas with more stable intra-annual food prices (includes all household members of consumer households).

3. Volume of regional food trade between programme countries.

4. Number of traders not supported by FTESA or similar initiatives trading through the G-Soko platform.

Outcome indicators

1. Net additional farm-gate price received by FTESA beneficiaries relative to local comparator.

2. Volume of staple food sold by FTESA farmer beneficiaries (metric tonne (MT)).

3. (a) Number of additional farmers benefiting from national and cross-border value chains; (b) Number of additional individuals benefiting from national and cross-border value chains (including household members).

4. Number of improvements to regional trade policy.

To achieve the above, FTESA targeted constraining sets of market failures in the following areas:

• Output 1: Improved post-harvest markets (storage and aggregation; market information; value chain coordination; warehouse receipts and credit; grades and standards).

• Output 2: Improved input markets (including seeds and fertilisers).

• Output 3: Improved policy and regulatory environment.

Output 1 indicators:

1.1. Number of male/female farmers accessing new/improved storage/aggregation services/facilities as a result of FTESA

1.2. Number of male/female farmers accessing improved market information system as a result of FTESA

1.3. Number of male/female farmers accessing improved value chain coordination (e.g. application of grades and standards to their products, improved logistics and virtual marketplace) as a result of FTESA

1.4. Number of male/female farmers accessing warehouse receipts and supplier credit as a result of FTESA

1.5. Number of private sector entities that adopt common grades and standards as a result of FTESA


2.1. Volume (MT) of new or improved inputs (seeds, fertiliser) traded by programme partners as a result of FTESA

2.2. Number of male/female farmers using improved inputs as a result of the activities of programme beneficiary input suppliers


3.1. Number of achievable regulatory and policy changes identified for which a dedicated influencing strategy is developed

3.2. Number of achievable regulatory and policy changes for which a dedicated influencing strategy is being implemented

3.3. Number of identified regulatory or policy changes for which public–private dialogue platform functioning as outlined in each influencing strategy

FINAL EVALUATION

Page | 9

Grant portfolio

Overview

This section provides a summary of the overall grant portfolio (see Annexes 10 and 11 for further details), based on the Portfolio Review Report.9 The total portfolio includes six DF and 16 CF grants, with two grants cancelled in 2015/16. There were four funding rounds for the CF:

• 2013 round 1: Early bird window

• 2014 round 2: Inputs and related services

• 2015 round 3: Farmer aggregation mechanisms

• 2016 round 4: Soybean value chain

Between 2014 and 2016, the following received awards under the DF:

• 2014: EAGC and Alliance for Commodity Trade in East and Southern Africa (ACTESA)

• 2015: Kilimo and WFP

• 2016: Farm Africa non-governmental organisation (NGO) consortium

The DF grants comprise 59% of the total value of grants while the CF accounts for 41%. The WFP grant is the largest and Sosoma the smallest. The average value for CF grants is £0.54 million and £1.8 million for DF grants.

Output areas

Grants typically cover more than one of FTESA’s three output areas:

• Output 1: improved post-harvest markets (storage and aggregation, market information, value chain coordination, credit, standards and grades)

• Output 2: improved input markets (seeds and fertiliser)

• Output 3: improved policy and regulatory framework

Three grants fall under output 1 only; two grants under output 2 only; one grant under output 3 only; 13 grants under both output 1 and output 2; and one grant falls under both output 1 and 3. Most grantees are active in improving value chain coordination, supporting farmers to use improved inputs and improving farmers’ access to storage and aggregation. Only a few grantees are working on improving farmers’ access to market information systems and credit, and with private sector entities on adopting common grades and standards. Only two grantees (EAGC and ACTESA) are working explicitly on policy and regulatory issues.

Geographical areas

FTESA’s intended programme scope covered nine countries across East and Southern Africa.10 The programme’s operations and grant coverage, however, focused largely on four countries in East Africa (Kenya, Rwanda, Tanzania and Uganda) and one in Southern Africa (Zambia). Tanzania is the country with the greatest grant coverage/activity, followed by Kenya and Uganda.

9 Itad (2018) Portfolio Review. 10 Burundi, Kenya, Malawi, Mozambique, Rwanda, Tanzania, Uganda, Zambia and Zimbabwe.

FINAL EVALUATION

Page | 10

Evaluation design and methods

This section outlines the updated EQs, design, modules, data collection methods, approach to analysis and synthesis, limitations and qualifications. The following sequence guides the final evaluation:

Figure 2: Evaluation sequence/structure

EQs

There are eight main EQs and associated sub-questions and theories (see Annex 2 for the evaluation matrix). The EQs test theories about the programme and encompass the following evaluation criteria: relevance, effectiveness, impact, efficiency, replicability, synergies, cross-cutting and sustainability (Table 1).

In April/May 2016, the EMU updated the original questions for the mid-term evaluation based on inputs from DFID. In December 2017, we updated the questions again to reflect the summative nature of the final evaluation. These questions provide the same coverage but the wording of some has changed and some are now sub-questions. The evaluation matrix includes theories we tested and approaches to assessment. The theories draw on the theory development undertaken as part of the baseline and mid-term phases. The EQs feed into the design of each of the evaluation modules (e.g. case studies).

The questions explore: what change has materialised and the contribution of the programme; how, why and who benefits, and in what circumstances; and whether these changes are likely to endure after programme closure.

Portfolio level (complementarity/synergies)

5. To what extent is FTESA a collection of individual interventions or a coherent portfolio?

a. What: To what extent has the combination of interventions generated results in excess of the programme’s component parts (i.e. generated complementarities/synergies)?

b. How, why and for whom and in what circumstances: How and why have these complementarities/synergies materialised? What were the mechanisms at play? Who has benefited from the complementarities/synergies? What circumstances (conditions, enabling/constraining factors) are conducive (or not) to generating benefits in excess of the programme’s component parts?

c. What indications are there of sustainability? What is the likelihood these will be sustained after direct support has ended?

Market level (systemic change/sustainability)

6. To what extent is FTESA likely to improve the functioning of national and regional staple food markets and generate systemic change?

a. What: To what extent is FTESA likely to improve the functioning of national and regional staple food markets and generate systemic change? Is there any evidence it has done so, so far?

b. How, why and for whom and in what circumstances: How and why have changes materialised, or are likely to materialise? What are the likely mechanisms for the spread of behaviour changes across

Purpose Questions DesignData

collection methods

Analysis and

synthesisFindings

Conclusions, lessons learned

and recommendations

FINAL EVALUATION

Page | 11

networks of actors? Which actors are pivotal to the spread of new behaviours? Who is likely to benefit? What circumstances (conditions, enabling/constraining factors) are conducive (or not) to generating systemic change?


Individual level (producer/farmer/trader/firm/consumer)

7. To what extent have improved trade support systems (output 1: storage, aggregation, information, value chain coordination, grades and standards, credit) increased production and trade?

a. What: To what extent has FTESA improved trade support systems? To what extent has production and trade increased as a result? Where there has been an increase in trade, to what extent has this trade been cross-border or within national boundaries?

b. How, why, for whom and in what circumstances: How and why have these changes materialised? What were the mechanisms at play? Who has benefited? What circumstances (conditions, enabling/constraining factors) were conducive (or not) to generating benefits for producers, farmers, traders and firms?


8. To what extent have improved availability and use of inputs (output 2: inputs) and application of good agricultural practices increased production and trade?

a. What: To what extent has FTESA improved the availability and use of inputs? To what extent has production and trade increased as a result?

b. How, why, for whom and in what circumstances: How and why have these changes materialised? What were the mechanisms at play? Who has benefited? What circumstances (conditions, enabling/constraining factors) were conducive (or not) to generating benefits for producers, farmers, traders and firms?


9. To what extent and how has FTESA brought (or facilitated) smallholder farmers into structured regional markets?

a. What: To what extent has FTESA brought smallholder farmers into structured regional markets?

b. How, why and for whom and in what circumstances: How and why have these changes materialised? What were the mechanisms at play creating these changes? How have smallholder farmers participated in these markets? Who has benefited (poor people, women)? What circumstances (conditions, enabling/constraining factors) are conducive (or not) to bringing in smallholder farmers?


10. To what extent has FTESA benefited consumers?

a. What: To what extent has FTESA delivered benefits for consumers?

b. How, why, for whom and in what circumstances: How and why have these changes materialised? What were the mechanisms at play creating these changes? Who has benefited? What circumstances (conditions, enabling/constraining factors) are conducive (or not) to generating benefits for consumers?


FINAL EVALUATION

Page | 12

Regulatory/policy level

11. To what extent have FTESA approaches to supporting reform to relevant policies, regulations, etc. contributed to change?

a. What: To what extent has FTESA delivered policy change?

b. How, why, for whom and in what circumstances: How and why have these changes materialised? What were the mechanisms at play creating these changes? Who has benefited? What circumstances (conditions, enabling/constraining factors) are conducive (or not) to generating policy change?


VfM

12. Does FTESA offer VfM in the results it achieves, compared with possible alternatives?

a. What: To what extent has FTESA delivered VfM overall? Which approaches provide more/less VfM (within the programme and compared to other programmes)?

b. How and why: How and why has FTESA delivered VfM (or not)?

Table 1: Evaluation criteria

EQ Relevance Effectiveness Impact Efficiency Replicability Synergies Cross-cutting Sustainability

1 x x x x x x

2 x x x x x x

3 x x x x x x x

4 x x x x x x x

5 x x x x x x x

6 x x x

7 x x x x x x

8 x x

Design

The EQs and underlying theories behind the programme informed the evaluation design. The final evaluation is a summative evaluation, following a theory-based approach, that allows for multiple lines of enquiry:

• Simply put, a summative evaluation focuses on the achievements of the programme at programme close, including how it contributed to expected results.

• A theory-based approach allows the exploration of the underlying theories behind the programme. Theory-based evaluations have two components: conceptual and empirical.11 Conceptually, theory-based evaluations spell out the theory behind the programme; empirically, theory-based evaluations explore how programmes cause intended or observed outcomes. The value of such an approach is in generating knowledge – not only knowing that a programme is effective (i.e. that a causal relationship exists between A and B) but also explaining a programme's underlying causal mechanisms (i.e. how and why A causes B). We focus on testing the underlying theories and the likelihood that the programme caused the intended results.

We apply a modular evaluation design that responds to the EQs. We summarise the modules below. Together the modules explore:

11 Rogers et al. (2000) ‘Program theory evaluation: Practice, promise, and problems’. New Directions for Evaluation, 87, 5–13.

FINAL EVALUATION

Page | 13

• Different levels of the FTESA programme: exploring individual grants, interconnected and complementary grants, and the overall portfolio.

• Different levels in the market system: exploring the role of different actors and the interactions between FTESA-funded interventions and these actors.

Evaluation modules

The evaluation matrix identifies the evaluation modules used to answer each EQ. We prepared separate reports, all of which we submitted to DFID ahead of producing the final evaluation report. For each module, the findings, conclusions, lessons learned and recommendations are a major input into this report. We briefly describe each module below (see Figure 3) with further details in Annex 3.

Figure 3: Modular evaluation design

1. The case synthesis explores what, how, why, for whom and in what circumstances the programme worked,

or not, for a purposeful sample of grants. The focus is on EQs 1–6. It draws on six qualitative case studies and two quantitative surveys. We revisited grants studied at baseline (July 2015 to June 2016), except for one cancelled grant. For the case studies, we employed realist enquiry to explore the underlying causal mechanisms that generate change. The key question under this approach is ‘how and why does this programme work, or not work, for whom, and in what circumstances?’ Realist enquiry allowed us to uncover evidence on how, why, for whom and in what contexts interventions contribute to change. Contribution analysis helped explain why the observed results occurred (or not) and the roles played by the interventions and other internal and external factors.

2. The thematic studies are very similar to the qualitative case studies (covering the same EQs) but we provide more analysis of the market system and FTESA’s progress toward fostering a more structured food market and stimulating systemic change (EQ2), including through purposeful and unintended collaboration between ‘lynchpin’ grantees and other grantees (EQ1). In answering the EQ on systemic change and the sustainability aspects of other questions, we applied the systems-level Adopt-Adapt-Expand-Respond (AAER) framework.12 In addition, we undertook an additional synthesis across the qualitative case studies and portfolio review to deepen the analysis pertinent to market systems using the framework, synthesising the findings in this final evaluation report.

3. The policy dialogue and influencing studies explore FTESA’s policy-influencing work, focusing on selected issues based on FTESA and DFID priorities (past and future), updating studies conducted between July and September 2017 for Kenya and Zambia.

12 The AAER framework examines the behaviours of actors in a market system to identify the degree to which they have adopted behaviours, adapted behaviours to suit their own purposes, or crowded in to a new market segment by exhibiting behaviours they observed. Springfield Centre (2014) Adopt-Adapt-Expand-Respond: a framework for managing and measuring systemic change processes.

FINAL EVALUATION

Page | 14

4. The portfolio review attempts to answer as many EQs as possible across the full portfolio of grants, collecting evidence along the theories in the evaluation matrix and drawing on secondary information including monitoring data and reporting prepared by the PMU and grantees.

5. The VfM assessment explores the VfM of the programme by exploring the VfM metrics and data collected by the PMU (against the 4E framework of economy, efficiency, effectiveness and equity).13 It focuses on EQ8 and updates an earlier assessment conducted between December 2017 and January 2018.

We include the sampling criteria in Annex 3. The case and thematic studies covered seven out of 20 grants in detail, accounting for 51% of the total value of the portfolio, and 56% of the portfolio when we include the additional grants linked to EAGC/G-Soko and Farm Africa interventions. Annex 3 provides details on grant coverage of each module.

Data collection methods

The evaluation matrix guided the team’s data collection and helped to ensure the team followed a coherent and comprehensive approach in answering the EQs. The team developed data collection tools (e.g. semi-structured interview and document/data review guides) for each of the evaluation modules based on the matrix, and gathered evidence against the questions and theories in the matrix. The team shared the evaluation matrix and some of the data collection tools with DFID. The matrix and tools used also allowed any member of the evaluation team to trace back all findings to original source data, ensuring transparency and robustness in data collection and analysis.

Each evaluation module combined secondary and primary data, including systematic document and data review and key informant interviews. The use of different evaluation modules and several data collection tools has helped mitigate against methodological biases and allowed triangulation of findings across different sources and modules.

Secondary data collection

The lead expert for each module undertook a systematic documentation and data review exploring a range of relevant programme and grant-specific documents. To ensure consistency and a degree of standardisation, each lead expert developed a template for the document review derived from the evaluation matrix, and recorded findings against the relevant EQs (see Annex 5).

Primary data collection

Key informant interviews

Developing interview guides/protocols and questions: The lead expert for each module developed interview guides and templates based on the EQs and documentation review. The guides ensured systematic coverage of questions across interviews, while retaining the flexibility to explore relevant questions for specific stakeholders and pursuing unforeseen avenues of enquiry. The guides helped ensure that data collection was relevant, consistent and comparable. Each expert tested the interview guides in the first few interviews, after which he/she reviewed the questions and made any necessary revisions based on issues emerging when using the tools, and shared these experiences, lessons and adjustments with other team members (where applicable).

Engagement of stakeholders in evaluation design and implementation: We consulted and actively engaged both the PMU (until it closed) and DFID in the final evaluation design and implementation, as well as all earlier stages of the evaluation. Throughout the five-year contract, the EMU kept the PMU and DFID regularly updated with evaluation plans, designs and findings (as they emerged through various reports), including who

13 www.itad.com/knowledge-products/better-value-for-money

http://www.itad.com/knowledge-products/better-value-for-money

FINAL EVALUATION

Page | 15

was in the team and their roles and plans for fieldwork, ensuring the process was transparent and well understood, providing the PMU with the opportunity to ask questions and raise any concerns. The EMU Team Leader frequently discussed with the PMU the evaluation process, including aims and objectives, and protocols for engaging grantees and agreed timings for fieldwork.

Such discussions provided the opportunity to explore DFID’s priorities, as well as suggestions from the PMU in earlier stages of the evaluation (e.g. mid-term) where they identified areas where learning could usefully feed into programme improvements.

During the final evaluation, the team shared reports from each of the different modules with DFID for any preliminary comments and met with DFID to discuss and strengthen the conclusions, lessons learned and recommendations in the final evaluation report, with DFID providing only suggestions for improvements. DFID did not dispute any of the material presented.

Evaluation team members evaluating specific grants involved grantee management in the early stages of fieldwork planning, fully informing them of the objectives of the evaluation and discussing and agreeing the plan to undertake the fieldwork, including timing. The team discussed with grantees when best to undertake the fieldwork to, for instance, avoid requesting time from farmers during planting and harvesting times.

Throughout the evaluation design, all interviewees actively engaged in the process and there were no reported challenges interacting with grantees and the target beneficiaries. The Team Leader consulted with each expert ahead, during and after fieldwork to discuss and check they were conducting engagements appropriately, and the experts copied the Team Leader into emails with interviewees.

Identifying interviewees and avoiding bias: We selected interviewees based on the purpose of each evaluation module, the questions and our document reviews, attempting to ensure as broad a range of relevant stakeholders as possible given the available time and resources. We requested stakeholder lists from the PMU and grantees and asked DFID for suggestions of interviewees. The evaluation team selected who to interview (groups and individuals) and which sites to visit to avoid any bias if the PMU and/or grantees selected sites and interviewees. In most cases, the evaluation team arranged the interviews. Once we started interviewing, we identified other interviewees based on emerging information from the interviews. In some instances, interviews took place in locations where grantee staff were present. In such cases, we respectfully requested that they not attend the interviews and all agreed and understood that their presence may bias an interviewee’s response.

Informed consent: Before interviews, each expert ensured they secured the interviewee’s informed consent, using the advice included in the guidance documents. Interviewers considered ethical considerations ahead of interviewing specific groups – for example, checking whether it was appropriate for a male interviewer to interview a female farmer.

Interviewers introduced themselves and outlined the purpose of the meeting and its likely duration. The team emphasised that the evaluation was an independent undertaking by Itad (a UK-based consultancy company hired by the funder DFID) to assess progress made and help DFID improve the design of similar programmes in future, stressing that ‘everything you tell us will be confidential and we will not use your name in any of our reports’. The team asked interviewees whether they had any questions or concerns that they wanted to raise before the interviewer proceeded. In addition, Itad securely stored all information from interviews.

1. Case studies and surveys:

a. Qualitative case studies: From March to July 2018, we carried out more than 140 in-person and telephone interviews as well as group discussions with over 370 farmers across six grants: EAGC, Joseph, Kaderes, Kilimo, Mount Meru and Virtual City. A detailed case study design document included protocols for all interviews, which experts tailored for each case study. At endline:

i. EAGC: We spent one week in Kenya in July 2018 meeting stakeholders in Nairobi and visiting project sites in Meru, interviewing buyers/traders, grantee staff, farmer groups and a bank representative.

FINAL EVALUATION

Page | 16

ii. Joseph: We conducted one week of field work in Masindi, Uganda in May 2018, interviewing key Joseph staff involved in the project, traders and a representative from a financial institution. The consultant visited four Joseph centres and interviewed village procurement officers and interviewed farmers at five different sites.

iii. Kaderes: We conducted data collection for one week in March 2018 in the Kagera region of Tanzania. We interviewed grantee staff, farmers, labourers, and SACCOs.

iv. Kilimo: We conducted data collection for one week in March 2018 in Kenya. The consultant visited six towns/counties (Nairobi, Makueni, Nakuru, Siaya, Eldoret and Kisumu), interviewing lead firms, aggregators, input dealers, extension service providers and farmer groups.

v. Mount Meru: We conducted data collection for one week in March 2018 in the Arusha and Singida regions of Tanzania, interviewing grantee staff, extension staff, farmers and a partner bank.

vi. Virtual City: We conducted data collection for one week in March 2018 in Nairobi and in Murang’a. We interviewed grantee staff, input suppliers, distributors, dealers (agrovets) and financial partners.

b. Quantitative surveys: We conducted baseline and endline quantitative endline household surveys for two grants – Joseph and Kaderes. We developed the endline survey questionnaires based on the baseline questionnaires, and requested any feedback from DFID and the PMU. We piloted and tested the questionnaires with farmers who were part of the Joseph and Kaderes target populations but who were not respondents in the final survey. We tested and amended the questionnaires extensively during the piloting phase, through mock interviews and consultation with the enumeration teams. We consulted and informed each grantee, as well as the PMU, at all stages of data collection.

i. Joseph: We conducted surveys in the Masindi district of Uganda at baseline (Q1 2016) and endline (Q1 2018), interviewing the same respondents in a longitudinal panel design, including the intervention group and a control group (counterfactual). We interviewed 586 respondents at baseline (20% female) and re-interviewed 418 of these respondents at endline (17% female).

ii. Kaderes: The team undertook surveys at baseline (Q4 2015) and endline (Q4 2017) in the Kagera region of Tanzania. The baseline panel consisted of an intervention group made up of farmers enrolled with Kaderes and a control group (counterfactual), consisting of a total of 596 respondents (18% female). The original intention was to compare the groups to assess the effects of the FTESA-funded warehouse at endline. However, the warehouse was not yet operational by Q4 2017; therefore, in agreement with DFID, we downscaled the endline survey to a monitoring survey, interviewing only Kaderes beneficiary farmers in the intervention group (210 respondents, 15% female) and not the control group.

2. Thematic studies: In July 2018, the team conducted fieldwork (focused on the EAGC and Farm Africa and complementary grants):

a. EAGC: We combined the fieldwork for the EAGC thematic study and case study (see details above). We visited the warehouses and village aggregation centres that received support through the FTESA grant to both EAGC and Shalem.

b. Farm Africa: We spent one week in Tanzania during June 2018. We visited Dar es Salaam and the Mbeya region, interviewing representatives from Farm Africa and Rural Urban Development Initiatives (RUDI), farmer groups, traders, Raphael Group and Kitalika Milling.

3. Policy dialogue and influencing studies:

a. Kenya: During September 2018, we conducted interviews in Nairobi with ex-PMU staff and key FTESA partners and intended targets for influencing: PMU staff; africapractice; Advisory Forum members;

FINAL EVALUATION

Page | 17

EAGC; Ministry of Agriculture; DFID; the United States Agency for International Development (USAID) East Africa Trade and Investment Hub (EATIH); and AGRA.

b. Zambia: During August 2018, we conducted interviews in Lusaka with ex-PMU staff and key FTESA partners and intended targets for influencing: PMU staff; Advisory Forum members; Indaba Agricultural Policy Research Institute (IAPRI); Consumer Unity & Trust Society (CUTS); SOPAG members; Ministry of Trade; DFID; and Food and Agriculture Organization (FAO).

4. Portfolio review: We relied on documents and data provided by the PMU and grantees, as well as a stakeholder survey (using an online survey tool: www.surveymonkey.com). We did not conduct any interviews.

5. VfM assessment: We carried out consultations (by phone) with PMU staff in December 2017 and followed up subsequently via email to update the February 2018 study.

As with the document and data review, we recorded the findings from each interview against the relevant EQs and theories for each module (see Annex 5).

In addition, we conducted an online stakeholder survey (August/September 2018) sent to 20 grantees to allow them to provide anonymous responses to questions related to the evaluation matrix. The EMU team developed the questions. We planned the survey at the end of the data collection so that we could use the survey to also fill in any gaps in data collection across the modules. We based the questions on the EQs. We used Survey Monkey, which is user-friendly and supports real-time analysis, including cross-tabulation. Of the 20 grantees, 10 responded. The survey included two open-ended questions while the remaining 23 questions were either multiple choice or required the input of a single number or date. Several of the questions offered the option to add additional comments.

Analysis and synthesis

Analysis

The lead expert for each module analysed information and presented findings using the evidence assessment frameworks (Annex 6), triangulating findings across the data collection methods and assessing the quality of evidence. Content analysis and thematic coding were the main analytical tools employed to undertake qualitative analysis across evidence sources, allowing the team to reduce large amounts of content into manageable evidence relevant to the EQs. We conducted the following steps:

• For each module, the lead expert read the material generated by the data collection tools to identify certain trends, themes and patterns emerging from the data, as well as flag diverging views and opposing trends that might require further data collection.14

• We categorised and coded text according to the EQs and theories. This combined deductive and inductive approaches, allowing the team to evolve the categorisation and coding as trends, themes and patterns became clearer.

• In a coding table, each column was a unit of data collected and each row a dimension (e.g. theme) analysed according to the EQs and theories.

• The lead expert analysed occurrences of trends, themes and patterns, including similarities and differences in the text, to generate insights and inferences.15

14 Trends, themes and patterns inductively reveal themselves to the researchers in the data's interaction with the empirical tools. 15 http://betterevaluation.org/evaluation-options/content_analysis

http://betterevaluation.org/evaluation-options/content_analysis

FINAL EVALUATION

Page | 18

Synthesis

Once each expert had produced their reports, the team discussed the emerging findings at the synthesis level. This approach helps ensure that evidence from the different levels feeds into the overall findings, conclusions and recommendations. To guide a systematic approach to synthesis, the Team Leader reviewed the evidence from the reports against each EQ. In some cases, only one report provided evidence to answer the EQ (e.g. on VfM). However, for others, more than one report fed into answering the EQ. The Team Leader triangulated and synthesised the findings as follows:

• Extracted the findings from each evaluation module, alongside reviewing the quality of the evidence gathered. In some cases, crosschecked findings with data collected and team members to check interpretation and ensure rigour and completeness.

• Assembled findings from the analysis and compiled these against the EQs and theories.

• Analysed assembled findings, extracting key trends, themes and patterns.

• Tracked back to ensure the logical source of the trends, themes and patterns from the evidence.

• After verifying trends, themes and patterns, drew these together in narrative form.

• Once the findings were synthesised in narrative form, the Team Leader logically derived conclusions, lessons learned and recommendations from the findings and shared, discussed, refined and agreed the final narrative with the team.

The Team Leader led the synthesis stage across the modules. To avoid potential bias associated with one team member undertaking the synthesis and interpreting the findings, each lead expert reviewed the final evaluation report and highlighted any points of digression and the Team Leader refined the narrative. The synthesised interpretation forms the basis for the main findings, conclusions, lessons learned and recommendations in this report.

Independence

As mentioned in section 3.4.2, the EMU took several steps to ensure independence. There were no conflicts of interest within the EMU team or with the PMU, DFID and grantees. Moreover, as mentioned in section 3.4.2, the team was able to work freely and without interference. Where the PMU disagreed with any of our findings and conclusions, we went back to our source material and requested any additional available information to double-check and triangulate in case of any misunderstanding on our part. The Team Leader requested a detailed explanation from the respective experts in any cases where disputes arose regarding the content of their reports. We revised material where our findings and conclusions reflected gaps in information and our misunderstanding and where we agreed with the comments. Where we had differences in judgements compared to the PMU, but did not agree with the PMU, as happened with one case study, we discussed the reasons for this, documented these discussions and provided an account to DFID, following the procedure agreed with DFID, to ensure the evaluation team’s independence.

While the PMU was involved at different steps (e.g. commenting on the evaluation design, providing suggestions on key interviewees and discussing preliminary findings in earlier stages of the evaluation), we operated independently of the PMU, with the evaluation team selecting stakeholders to interview and without PMU or grantee staff in attendance. This helped ensure the evaluation was a credible exercise, with stakeholders willing and comfortable to provide unbiased judgements on performance. This increased the credibility and usefulness of our reports for both accountability and learning.

The evaluation team frequently ‘checked-in’ to discuss preliminary findings and conclusions and openly discussed any differences in opinion, in most cases going back to the source material to check understanding, before reaching agreement. The case study experts had several discussions on the synthesis of the different findings and conclusions across the cases to derive main findings, conclusions and recommendations. The Team Leader also shared with the evaluation team the main findings, conclusions and recommendations from the different modules triangulated in the final evaluation report. Across the team, no one disputed any of the

FINAL EVALUATION

Page | 19

findings; however, in some cases team members reached slightly different conclusions, which the Team Leader addressed where there was consensus across the team. Also, some team members highlighted gaps in information presented, which the Team Leader added to the report.

Limitations and qualifications

Limitations

We faced several limitations, including the following:

• Stage of implementation of grants: Some of the grants only started in 2016, while others faced delays in implementation, limiting evidence at higher levels of the ToC (including on outcomes and impacts).

• Lack of detailed information on beneficiaries: Grantees report that most beneficiaries are smallholder farmers, but we were unable to explore beneficiary profiles in detail. Apart from gender-disaggregated data reported against the logframe and the EMU’s surveys of two grants and their beneficiaries, where the EMU surveys assessed respondents’ levels of poverty using the Poverty Probability Index (PPI), there is limited data on the poverty profiles of beneficiaries across all grants. Neither grantees nor the PMU systematically collected data on incomes and poverty levels. In addition, given the small sample sizes of female farmers represented in the two EMU quantitative surveys (where we surveyed ‘main farmers’, most of whom were male), it was sometimes not possible to disaggregate findings by gender in a way that would yield significant and meaningful results.16

• Strength of evidence: The case studies and thematic studies interviewed key market actors, including beneficiaries (namely, farmers), and triangulated data sources (EMU, PMU and grantee). We were unable to conduct primary data collection for all grants. This introduces potential bias when reporting results based on secondary sources, which we take account of in our strength of evidence assessment against each EQ (see Section 4 and Annex 7). Also, in several cases where we had concerns about the quality of quantifiable data (e.g. a lack of field/mission verification reports), we reported the direction of change (e.g. increase) but not the magnitude of those increases. Moreover, the PMU conducted data verification, which may introduce potential bias. To avoid the potential of bias, we recommend an independent entity verifies the data (either the equivalent of the EMU or another entity) in any future programme.

• Synthesis: Lack of good-quality information across all grants constrained our ability to synthesise results across the entire portfolio. Hence, in many cases the team relied on presenting examples within the narrative to substantiate findings.

• Closure of the PMU: The PMU closed in April 2018, limiting our ability to check understanding and preliminary findings with key staff. However, we were able to follow up and seek clarification with the grantees, where needed, and some of the ex-PMU staff were willing and able to engage with the team.

• Lack of data on cross-border trade: It is difficult – and in many cases impossible – to assess the extent to which direct beneficiaries’ (farmers) produce crosses the border. Traders and other buyers usually take the produce across borders, not farmers. However, we provide anecdotal examples where we found evidence.

Qualifications

We also include a few qualifications:

• We provide detailed evidence in the different reports for each of the modules (cross-referenced throughout this report) that we then analysed and synthesised in this report to assess FTESA’s performance, rather than include all the details in this report.

16 Itad (2018) Kaderes 2017 Monitoring Study Report. Itad (2018) Joseph Initiative Quantitative Impact Assessment.

FINAL EVALUATION

Page | 20

• While we interviewed a range of market actors along the value chain, most of the data collected for the case studies and surveys was at the farmer level, as agreed with DFID when we developed the final evaluation terms of reference.

• The CToC and output ToC diagrams and FTESA indicators do not explicitly refer to production and focus on volumes sold and traded, but increased productivity and production are objectives of the programme and therefore we explore these against the relevant questions.

• EQs 2 and 4 both refer to production and trade, but we focus on the effects of inputs on productivity and production under EQ4 (focusing on supply-side) and the effects of improved storage and aggregation etc. (supporting functions) on post-harvest losses and volumes stored and aggregated under EQ2. Then, under EQ5, we explore supply and demand, and therefore volumes sold and traded. We focus on the sub-questions related to ‘whom’ under EQs 3, 4 and 5.

• The focus of the final evaluation was summative, with no formative elements, and there was no intention from the start of the evaluation to undertake a review of the PMU at the end of the programme. However, reviews of an earlier version of this final evaluation report highlighted that this reduces the usefulness of this report. Hence, we have attempted to integrate some of the findings from the mid-term evaluation that we consider still relevant, as well as any information that emerged about the PMU through the different evaluation modules.

FINAL EVALUATION

Page | 21

Findings

This section provides the findings of the evaluation, drawing on the individual evaluation modules and synthesising findings across these modules. The sub-section structure follows the EQs in the evaluation matrix:

4.1. To what extent is FTESA a collection of individual interventions or a coherent portfolio (EQ1)?

4.2. To what extent have improved trade support systems increased production and trade (EQ3)?

4.3. To what extent have improved availability and use of inputs and application of good agricultural practices increased production and trade (EQ4)?

4.4. To what extent and how has FTESA brought (or facilitated) smallholder farmers into structured regional markets (EQ5)?

4.5. To what extent is FTESA likely to improve the functioning of national and regional staple food markets and generate systemic change (EQ2)?

4.6. To what extent has FTESA benefited consumers (EQ6)?

4.7. To what extent have FTESA approaches to supporting reform to relevant policies, regulations, etc. contributed to change (EQ7)?

4.8. Does FTESA offer VfM in the results it achieves, compared with possible alternatives (EQ8)?

This section highlights the performance of the FTESA programme in strengthening and supporting staple food trade. Success across grantees is variable. Overall, the programme has helped set important foundations for changes further along the CToC (outcomes and impact) but falls short of original expectations, partly due to delays and often unrealistic timeframes and targets to deliver outcome- and impact-level results (volumes sold, prices, trade).

To what extent is FTESA a collection of individual interventions or a coherent

portfolio (EQ1)?

Main findings and conclusions [strength of evidence – medium]: The FTESA portfolio of grants offered opportunities for projects to benefit from complementarities and synergies between them. There is some evidence of interlinkages and complementarities materialising between FTESA grantees that generated results that were more than those achievable through an individual grant, where some grantees were able to tap into support provided by others to increase the impact of their interventions (including combining support at different points along the value chain). Partnerships have enabled grantees to capitalise on each other’s services. However, there are concerns regarding the sustainability of some of these interactions, particularly those reliant on DF interventions rather than commercial entities.

The most frequently cited connection is between grantees and EAGC. The PMU focused much of its efforts on building interlinkages and complementarities across the portfolio around the EAGC grant, with the intention that several different FTESA-funded grantees would use the G-Soko platform and create a group of early adopters. However, the failure of G-Soko to take off successfully and demonstrate consistent results through early adopters significantly reduced the opportunity for synergies and seriously hampered FTESA’s ability to create a portfolio that delivered more than the sum of its parts through grantees using the platform and accessing larger markets, with others copying, thus limiting sustainability and curtailing the impact of the FTESA programme in the wider market.

Linkages between FTESA grantees were in part facilitated by the PMU but were also due to a grantee’s knowledge of the other grantees, existing relationships and/or searching out synergies themselves. The PMU’s model restricted the programme’s ability to design strong interlinkages and build a coherent, comprehensive portfolio. In several cases, the PMU attempted to build linkages into the design of projects after awarding the grants. With relatively few rounds and lack of real-time monitoring and learning built into the programme design and decision-making, there were limited opportunities to quickly learn from previous rounds and tailor future rounds to attempt to generate interlinkages, building on existing projects to improve the performance of the overall portfolio. Moreover, the PMU itself was not set up with the level of technical assistance required to provide substantive design inputs.

FINAL EVALUATION

Page | 22

What?

Synergies between grantees

In this section, we seek to understand whether the programme took advantage of intended and unintended synergies and complementarities and whether the programme delivered results more than the sum of its component parts. Complementarity of grants was central to the programme’s strategy, providing the opportunity to fund complementary interventions designed to support each other’s success. The PMU awarded grants based on funding rounds, relying on promising applications surfacing, and using successive rounds of grant awards to complement earlier grants, connecting and complementing CF with DF grants as far as possible. The rationale for doing so was to develop greater coherence across the grants and therefore the programme, rather than having a portfolio of separate projects.

The PMU focused much of its effort on building interlinkages and complementarities across the portfolio around the EAGC grant, especially warehouse certification and the G-Soko platform, with the intention that several different FTESA-funded grantees (including farmers they support) would use the platform, as buyers or sellers, and create a group of early adopters. Connecting grantees with the platform offered the potential to create a group of early adopters who would successfully use the platform and generate demonstration effects, leading to others users using the platform and other market actors crowding in to support the system. In addition, FTESA supported EAGC to inspect and certify warehouses and supported EAGC and several other grantees to work with other grantees’ warehouse operators, village aggregation centres and farmer groups to help farmers (e.g. through training) to improve post-harvest handling, aggregate and meet quality standards (required by G-Soko), enabling them to aggregate sufficient quantities at the required quality to store in certified warehouses and sell on the platform.

• There is some evidence of interlinkages and complementarities materialising between FTESA grantees that have generated results more than would have been achievable through an individual grant, where some grantees (especially CF grantees) were able to tap into support provided by others to increase the impact of their interventions, including combining support at different points along the value chain (e.g. access to inputs, storage and aggregation, and brokering relationships with buyers). Partnerships enabled grantees to capitalise on each other’s services.

• Linking grantees: Eight out of 10 survey respondents stated that they interacted with other grantees when implementing activities funded by FTESA: eight stated they interacted with EAGC, five with WFP, four with Kilimo, three with Farm Africa, three with Mount Meru, three with Musoma and three with Raphael.17 This included linking grantees, but not necessarily grants per se.

• Complementary activities: In some cases, FTESA funded more than one grant working together to support the same groups of farmers (e.g. where one grantee supported training on storage and aggregation and another helped farmers directly access markets). In some cases, DF grantees collaborated with CF grantees:18

▪ For example, WFP collaborated with Joseph to train smallholder farmers on post-harvest handling, with Joseph providing the aggregation facilities.

▪ Classic partnered with WFP and Syngenta to train farmers on good agricultural practices.

▪ EAGC and Shalem collaborated to provide training to smallholder farmers in Meru on post-harvest handling and collective marketing.

▪ WFP provided training on good agricultural practices to farmers working with Shalem and helped them access input loans.

17 Itad (2018) Stakeholder Survey. 18 Itad (2018) Portfolio Review.

FINAL EVALUATION

Page | 23

▪ WFP helped farmers secure deals with grantees to purchase grain from their farmers (e.g. Musoma, Shalem).

▪ Farm Africa facilitated relationships between farmer groups and Raphael, one of the largest buyers in Southern Tanzania, who bought aggregated quantities of rice from the groups.19

• EAGC warehouse certification: There is a concentration of connections to EAGC through inspection and certification of warehouses and the G-Soko platform. The certification process occurred in conjunction with connecting certified warehouses to the G-Soko system and providing the necessary equipment to use G-Soko, making them ‘G-Soko ready’. EAGC inspected grantees’ warehouses (Classic, Farm Africa, Joseph, Kaderes, Musoma, Raphael, Shalem and Yak) and certified all but Kaderes (pending improvements).20

• G-Soko: Several grantees (Classic, Farm Africa, Joseph, Kaderes, Kilimo, Musoma, Raphael, Shalem, Sosoma, WFP and Yak) intended to feed grains onto the EAGC’s G-Soko platform or purchase through the platform. While most grantees have certified their warehouses, and are ready to use G-Soko, only a few grantees have used G-Soko (e.g. Farm Africa, Raphael and Shalem as sellers and/or buyers) and most did not use its full functionality. Those grantees that traded (or attempted to trade) via the platform indicated that they are unlikely to do so in future given several challenges using the platform and a lack of demonstrated benefits (see Box 2, section 4.4).21

• Lead firm partnerships: Other examples of synergies include partnerships between Kilimo and lead firms Kaderes, Musoma, Raphael and Shalem. The lead firms each received their own FTESA grant but were also part of Kilimo’s lead firm consortia model, where Kilimo linked farmers to lead firms (buyers), securing farmers a market and bringing together value chain actors (e.g. aggregators, input and finance providers) through structured arrangements (consortia) guided by memoranda of understanding (MoUs). For example, Kilimo facilitated connections and sales between farmer groups and Kaderes, helped farmer groups connect to G-Soko, assisted Kaderes in the EAGC warehouse certification process and provided technical assistance on good agricultural practices and post-harvest handling. Kilimo also linked Kaderes in Tanzania with Cheptarit Star Women’s Group in Kenya, who bought beans from Kaderes, resulting in cross-border trade.22

Synergies with other actors and relevant programmes

Several grantees formed partnerships with other actors (e.g. banks, research centres, government actors and other donors). In some cases, this improved smallholder farmers’ access to inputs and services. We provide examples below.23

• Financial service providers: Several grantees formed partnerships with financial service providers, most of them banks. In a few cases, this resulted in improved access to finance for smallholder farmers (see EQ3 for a further discussion of this). Classic partnered with Kenya Commercial Bank to provide smallholder farmers with access to input financing, resulting in 1,700 farmers signing up for loans. The WFP partnered with microfinance institutions that facilitated access to finance for farmer-based organisations. Shalem partnered with an insurance provider, thus enabling smallholder farmers access to crop insurance.

• Universities and research centres: Grantees (e.g. Kaderes, Kilimo, Raphael and Shalem) connected with research institutions and universities to enable access to improved inputs and training.

• Government actors: There are several cases of grantees working with government. For example, Farm Africa (RUDI) approached local government to request support for introducing G-Soko to local warehouses, Kaderes sought local government help to access seeds and Shalem partnered with the Ministry of

19 Itad (2018) Farm Africa Thematic Study. NB: Raphael’s FTESA grant is specifically for beans, not rice. 20 Itad (2018) EAGC Endline Qualitative Case Study. 21 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. 22 Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Portfolio Review. 23 Itad (2018) Portfolio Review.

FINAL EVALUATION

Page | 24

Agriculture in Kenya to provide advice to farmers on tackling pests, which led to application of pesticides and reduced spread of pests.

• Other donors and their programmes working with the PMU: The PMU and several grantees collaborated with other donors and their programmes. The PMU collaborated mostly with EATIH and AGRA, providing similar support at the regional level. Both AGRA and EATIH worked with several of the grantees at an individual level (e.g. EAGC) and with the PMU. EATIH’s support to grades and standards and reducing non-tariff barriers complemented activities undertaken by FTESA. There were cases of duplication and double-dipping by grantees, but good relationships between the PMU, Alliance for a Green Revolution in Africa (AGRA) and EATIH, as well as DFID, helped identify and mitigate against this risk. In most cases, collaboration focused on improving coordination and keeping each party up to date on what activities they were funding to avoid duplication and any double-dipping by grantees where grantees were seeking funding for the same activities from other donors, especially in the case of EAGC. The mid-term evaluation and earlier DFID annual reviews recommended better coordination with other donor programmes, such as TMEA and EATIH. During 2015/16 and after the mid-term evaluation, the PMU made significant efforts to coordinate the activities of different organisations working in similar fields and the PMU was instrumental in coordinating the donors and programmes (AGRA and EATIH) funding EAGC, which in some cases led to the reallocation of funds to reduce duplication. However, the PMU made less progress in coordinating with programmes such as TMEA. Other donors also acknowledged lack of coordination and potential overlap between their programmes and TMEA, concluding that this was due to TMEA expanding its agreed scope and neglecting coordination with others.24 In terms of other donors and their programmes working with other FTESA grantees (apart from EAGC), AGRA worked with several grantees including ACTESA, Esoko, Kilimo, Pee Pee and WFP. For example, AGRA provided WFP farmers in Tanzania with post-harvest handling training and equipment, working with other partners including RUDI (part of the Farm Africa consortium). In terms of complementarities between grants and the PMU’s other roles (e.g. on policy influencing), we discuss this further in section 4.7.

How, why, for whom and in what circumstances?

According to the stakeholder survey, eight (of 10) grantees stated that the PMU facilitated interactions with other grantees, which represents an improvement since the mid-term when five (out of 14) grantees connected with other FTESA grantees and attributed these linkages to the programme. According to the responses, the main factors that facilitated these linkages were grantee meetings held by the PMU, which only began after the mid-term, and the PMU actively connecting grantees including to the G-Soko platform.25 Across the case studies, linkages between FTESA grantees were in part facilitated by the PMU but were also due to the grantee’s knowledge of the other grantees, existing relationships and/or searching out synergies themselves. For instance, all grantees know EAGC, and are members, and are aware of its role in inspecting and certifying warehouses. Several grantees learnt about the G-Soko trading platform through their EAGC membership but also through their involvement with FTESA. EAGC’s reputation on certification, as well as capacity building and awareness raising on grades and standards, encouraged grantees to certify their warehouses. Similarly, EAGC’s reputation led several grantees to explore and use G-Soko. The PMU encouraged grantees to use G-Soko, sometimes directly by including linkages to G-Soko in grant agreements (e.g. Farm Africa).26

As discussed during the mid-term, one of the main intended benefits of the network of relationships was through increased aggregation and sales through G-Soko.27 However, challenges rolling out and implementing G-Soko (see Box 2, section 4.4), highlighted by several grantees including Farm Africa, Raphael and Shalem,

24 Itad (2016) Mid-term Evaluation Report. Interviews. 25 Itad (2018) Stakeholder Survey. 26 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. Itad (2018) Case Evaluation Synthesis. 27 Itad (2016) Mid-term Evaluation Report.

FINAL EVALUATION

Page | 25

significantly reduced the opportunity for synergies and seriously hampered FTESA’s ability to create a portfolio that delivered more than the sum of its parts through grantees using the platform and accessing larger markets, with others copying, thus limiting sustainability and curtailing the impact of the FTESA programme in the wider market.28

Successful connections between grantees emerged largely due to grantees’ buy-in, where they understood that their interventions complement one another and therefore contribute to each other’s success, so have the incentive to work together, and where each actor is clear on their roles and responsibilities and committed to the partnership. For example, Kilimo’s relationship with lead firms in its consortium, especially Kaderes, resulted in synergies since both parties were ‘bought-in’ and committed to the partnership, cemented by an MoU. Conversely, Kilimo reported ‘dormant partners’, undermining its lead firm consortium model, despite having an MoU in place with these partners.29

Other conditions, including overlapping geographies, have aided in achieving complementarities and synergies (e.g. Raphael, Farm Africa and WFP all working in Mbeya, Tanzania). However, this can also have a downside, as the PMU raised concerns regarding some duplication of efforts. Similarly, Seba raised concerns it was duplicating efforts by working with the same farmers as Mount Meru in Zambia.30

The PMU’s model, where applicants designed the projects and applied through rounds of awards (four CF rounds and three DF rounds), albeit with support from the PMU especially for the DF projects, restricted the programme’s ability to design strong interlinkages and build a coherent, comprehensive portfolio. In several cases, the PMU attempted to build linkages into the design of projects after awarding the grants. Moreover, a lack of real-time monitoring and learning built into programme design and decision-making, and technical assistance available from PMU staff, limited opportunities to quickly learn from previous rounds and tailor future rounds to attempt to generate interlinkages (albeit reliant on applications received), building on existing projects to improve the performance of the overall portfolio.

Sustainability

The main ‘hubs’ are DF grantees (EAGC, Kilimo and WFP) rather than CF (private sector) grantees, raising concerns that synergies between grantees may not last without continued funding. Moreover, the G-Soko platform failed to demonstrate its intended benefits.31

In a few cases, grantees connected private sector actors supported by different grants (e.g. Kilimo linked Kaderes with an off-taker, while WFP collaborated with a financial institute to get Shalem’s farmers access to input loans) and, where each party has the commercial incentive to continue working together, these relationships are likely to endure. 32

28 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. Itad (2018) Case Evaluation Synthesis. 29 Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Portfolio Review. 30 Itad (2018) Portfolio Review. 31 Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Portfolio Review. 32 Itad (2018) Portfolio Review.

FINAL EVALUATION

Page | 26


Main findings and conclusions [strength of evidence – strong]: Overall, evidence of FTESA-led improvements in trade support systems shows mixed success, partly due to delays in implementation including where some interventions are reliant on the completion of other activities (e.g. a WRS requires the warehouse to be in place to the required standards) and the short timeframes for results to transpire. There was good progress and achievements on activities that set the foundations for improvements in trade support systems. Despite differences in context and implementation models, several of the grants helped farmers reduce post-harvest losses, increase volumes and quality of produce stored and aggregated, as well as improving farmers’ position in the market. However, the results fall short of expectations. Training on post-harvest handling, in combination with the incentive of higher prices for aggregated better-quality produce, helped farmers understand the value of improving post-harvest handling and aggregating produce, as well as motivated farmers to improve post-harvest handling and store and aggregate their produce with others, reaching required standards, and marketing collectively to access better markets, given the right incentives and demonstrated benefits – namely, better prices and market access.

However, in many cases the aggregation volumes fell short of the volumes expected. Barriers to storage and aggregation have curtailed the achievement of expected results, including construction delays, smallholder farmers’ preference for immediate payment, issues of trust, transport distances and costs, and lack of better markets. The case study evidence shows that some of the grants were unable to deliver key output milestones within the relatively short timeframes. Also, access to finance remains a major challenge, with considerable institutional barriers existing in the financial markets, where in many cases banks remain risk averse about lending to smallholder farmers, with the existence of collateral alone not sufficient to engender confidence.

What?

Overall, evidence of FTESA-led improvements in trade support systems (storage and aggregation, grades and standards, credit, market information and value chain coordination) shows mixed success, partly due to delays in implementation as well as the short timeframes for results to transpire. However, there was some good progress and achievements on activities that set the foundations for improvements in trade support systems.

Improved access and use of storage and aggregation facilities

Several of the grantees received funding to improve storage and aggregation facilities (e.g. EAGC, Farm Africa, Joseph, Kaderes, Mount Meru, Musoma, Pee Pee, Raphael Group, Shalem, Sosoma, WFP and Yak) or connect farmers to facilities (e.g. Kilimo).33 Across these cases, there was progress on activities that set the foundations for improvements in trade support systems: for example, farmers registered; training on post-harvest handling (including grades and standards); construction and/or upgrading of warehouse and aggregation facilities; linking farmers with aggregators and warehouses; forming partnerships between buyers and sellers; etc. There are examples of these activities contributing to increased storage, aggregation and reduced post-harvest losses, as well as contracts between farmers and buyers and sales realised at higher prices.

Joseph constructed over 70 aggregation centres (called Joseph centres) that were well utilised with significant volumes aggregated, but most of the centres closed once Joseph reduced its purchases from farmers.34 EAGC, through its certification of warehouses and support to aggregation centres, helped increase volumes aggregated. However, the number of warehouses certified and aggregation centres supported falls short of the original target, while storage capacity is underutilised in many certified warehouses and the flows of commodities from village aggregation centres to certified warehouses are lower than envisaged.35 Farm Africa

33 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Mount Meru Endline Qualitative Case Study. Itad (2018) Joseph Initiative Endline Qualitative Case Study. Itad (2018) Case Evaluation Synthesis. Itad (2018) Portfolio Review. 34 Itad (2018) Joseph Initiative Endline Qualitative Case Study. 35 Itad (2018) EAGC Endline Qualitative Case Study.

FINAL EVALUATION

Page | 27

renovated 46 aggregation centres to meet EAC standards, with significant increases in volumes aggregated, exceeding original targets.36

Delays in implementation, especially warehouse and aggregation centre construction or refurbishment, limited potential progress for several grantees. Kaderes and Mount Meru did not fully complete the construction of storage facilities before the end of the FTESA programme. The facilities, while very close to completion, were not yet operational and farmers continued to store on the farm or sell at harvest.37 Environmental challenges beyond Kaderes’ control, namely an earthquake, impacted the construction of the warehouse. Also, Kaderes dropped the planned aggregation centres due to the rising costs of warehouse construction. Kilimo, through its lead firm approach, relied on other firms (e.g. Raphael and Kaderes) to provide additional storage and aggregation, but Raphael and Kaderes faced delays in constructing facilities, with the effect that many farmers continue to store as before at home.38

Delays in completing the establishment and upgrading of storage and aggregation facilities (e.g. Kaderes) led to knock-on delays for other planned activities (e.g. WRSs and access to credit).39 However, in the case of EAGC, where warehouses and aggregation centres were available, and many improved, the WRS did not take off due to implementation challenges that were partly due to delays establishing and rolling out G-Soko.40

Improved access to training on post-harvest handling, including grades and standards

Several cases provide strong evidence of the benefits of FTESA-funded activities on post-harvest handling, storage and aggregation, especially training of farmers on post-harvest handling (including grades and standards). For example, for Kaderes, even though farmers were not using the warehouse, there is strong evidence that post-harvest losses reduced and quality increased due to training.41 For EAGC, there is evidence that certifying warehouses, improving village aggregation centres and training warehouse operators and farmers on post-harvest handling (including grades and standards) contributed to increased storage, aggregation, improvements in post-harvest handling and reduced post-harvest losses (of between 10 and 20%), as well as better-quality produce. There is also strong evidence that Farm Africa farmers reduced post-harvest losses considerably (by 18%, on average) and increased quality and volumes aggregated.42 Other grantees (e.g. Classic, Kilimo, Musoma, Raphael, Seba, Shalem, Sosoma, WFP and Yak) trained farmers in post-harvest handling, with several reporting reductions in post-harvest losses, improved quality and increased volumes aggregated.43 By contrast, Joseph farmers experienced increased post-harvest losses, despite receiving training on post-harvest handling, when Joseph significantly reduced its purchases from farmers.44

Improved access to finance

Many of the grantees (e.g. Classic, EAGC, Farm Africa, Joseph, Kaderes, Kilimo, Mount Meru, Virtual City, WFP and Yak) working on improving smallholder farmers’ access to credit focused on facilitating access to commercial credit through formal banking institutions.45 Across most of the case studies, there is limited evidence of tangible improvements in access to credit by smallholder farmers as a direct result of FTESA, due to delays in implementation (e.g. Kaderes’ WRSs) and the risks involved with dealing with smallholder

36 Itad (2018) Farm Africa Thematic Study. 37 Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Mount Meru Endline Qualitative Case Study. 38 Itad (2018) Kilimo Trust Endline Qualitative Case Study. 39 Itad (2018) Kaderes Endline Qualitative Case Study. 40 Itad (2018) EAGC Endline Qualitative Case Study. 41 Itad (2018) Kaderes Endline Qualitative Case Study and Itad (2018) Kaderes 2017 Monitoring Study Report. 42 Itad (2018) Farm Africa Thematic Study. 43 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Portfolio Review. 44 Itad (2018) Joseph Initiative Endline Qualitative Case Study. Itad (2018) Joseph Initiative Quantitative Impact Assessment. 45 Itad (2018) Portfolio Review.

FINAL EVALUATION

Page | 28

farmers.46 However, there are some examples that provide evidence of successful consortia-type models (e.g. Farm Africa and WFP).47

For Kaderes, the WRS is not yet operational.48 Similarly, there were delays in the roll-out of Virtual City’s voucher system and EAGC’s WRS linked to G-Soko.49 However, some of the grantees have made the necessary partnerships with financial institutions to provide credit and payment vehicles. Mount Meru formed a partnership with NMB Bank to roll out a Mastercard Foundation-funded programme called e-Kilimo, where farmers can trade their agricultural goods and receive payments electronically, advancing credit to farmers. However, no farmer has accessed credit facilities, even with guaranteed market, price and farming contracts, due to the perceived and real risk involved in dealing with individual farmers.50 Similarly, for Kilimo, while some farmers accessed credit, banks were reluctant to provide farmers with credit facilities in the absence of solid proof of their ability to repay or a bank guarantee from Kilimo.51 Despite an initial partnership with one bank, farmers supported by the Joseph grant also struggled to access finance.52

In Tanzania, RUDI (under the Farm Africa grant) promoted the expansion of a financial model established under the Southern Highlands Rice Consortium. The consortium included a buyer (Raphael), input suppliers (Yara, Agroseed Techs and Obo Investment) and a bank (NMB). Members signed a tripartite agreement with farmer organisations to use crops aggregated in warehouses as collateral to access production loans and inputs on credit, on condition of there being a buyer (in this case, Raphael). For 10 of the 11 farmer-based organisations that participated in the model, payback was 100% (and for the other, 93%).53 Also, in Tanzania WFP built a network of buyers, input providers and banks working with farmer-based organisations, increasing the number of farmer-based organisations accessing input loans from 21 (2015/16 season) to 69 (2017/18 season).54

Market information

The case studies and grantee reports offer limited systematic evidence of improvements in sharing relevant market and agricultural information. However, there are some examples of improvements:55

• Esoko reported more than 100,000 farmers in Malawi, Tanzania and Zimbabwe subscribing and receiving messages with information on markets and good agricultural practices.

• The G-Soko platform shares market information from the Regional Agricultural Trade Intelligence Network and allows warehouses to communicate with farmers by sending bulk SMSs.

• Kilimo Trust used WhatsApp groups for each of its consortium members so that members can pose trade and production-related questions and arrange trades.

• Some of the grantees (e.g. Farm Africa) also report using village aggregation centres as hubs to share information on markets and good agricultural practices (via noticeboards and mobile phones).

46 Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Joseph Initiative Endline Qualitative Case Study. 47 Itad (2018) Farm Africa Thematic Study. Itad (2018) Portfolio Review. 48 Itad (2018) Kaderes Endline Qualitative Case Study. 49 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Virtual City Endline Qualitative Case Study. 50 Itad (2018) Mount Meru Endline Qualitative Case Study. 51 Itad (2018) Kilimo Trust Endline Qualitative Case Study. 52 Itad (2018) Joseph Initiative Endline Qualitative Case Study. 53 Itad (2018) Farm Africa Thematic Study. 54 Itad (2018) Portfolio Review. 55 Itad (2018) Farm Africa Thematic Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Portfolio Review.

FINAL EVALUATION

Page | 29


Improved access and use of storage and aggregation facilities

In some cases (e.g. Kaderes and Mount Meru), delays in implementation have meant that participants are not yet using storage and aggregation facilities and there are knock-on effects elsewhere, for instance regarding access to credit through WRSs and deferred sales through storage.56 Both Kaderes and Mount Meru faced environmental constraints, including an earthquake (Kaderes) and rains (Mount Meru), which delayed construction and raised costs. For Kaderes, a group of women farmers became involved with the project in 2015, incentivised to grow beans as Kaderes assured them of a market. However, Kaderes did not buy the beans given delays with the warehouse construction, so the women farmers sold crops to other buyers and dropped out of the project.57 EAGC also reported high costs of warehouse improvements to meet the required standards, deterring many warehouse operators from making the upgrades.58

The results reported in some cases (e.g. Kaderes) were not due to storage and aggregation facilities per se but rather due to training that increased knowledge and changed the behaviour of farmers (e.g. understanding the benefits of aggregation and deferred sales leading to improvements in farmers’ bargaining power) and/or because farmers responded to the incentive provided by the promise of a market. In the case of Kaderes, for example, farmers responded to the visual signal of a huge warehouse situated within their neighbourhood. Some farmers dropped out as the market they anticipated did not materialise, but many remained since they had high levels of trust in Kaderes and expected purchases in future.59

The closure of the Joseph centres and the fact that the minimum quantity requirements (20 MT) for a ‘home pick-up’ by Joseph are out of the reach of most farmers sent a negative signal to farmers about the long-term prospects of dealing with Joseph as well as other off-takers, resulting in a loss of confidence and low levels of trust in the relationship with Joseph and others.60

Both Farm Africa and EAGC cite the upgrading of storage and aggregation facilities as an enabler to farmers using the facilities due to increased confidence. However, storage rates at EAGC-certified warehouses were lower than anticipated for a variety of reasons, with several of these related to trust and incentives. For example, perceived conflict of interest, where warehouse operators are also traders, including instances of switching stocks and co-mingling, and lack of a ‘tried and tested’ dispute resolution system to counter such behaviour, discouraged farmers from storing in, and buyers from sourcing from, warehouses. Also, practical constraints deterred use of warehouses by farmers, such as long distances to travel to reach the warehouse (with many aggregation centres outside of the catchment areas) and associated high transport costs (EAGC, Farm Africa and Joseph).61 Moreover, many farmers have a longstanding mistrust of third parties handling their stocks, particularly given their produce usually accounts for most of their livelihood. Coupled with a lack of progress with G-Soko and the WRS, limiting any positive demonstration effects of using certified warehouses, farmers’ incentive to move their produce to a certified warehouse was lower than anticipated.62

Some cases (e.g. EAGC and Farm Africa) show that many farmers still prefer immediate cash payments, rather than storing and aggregating their crop in anticipation of higher prices later, which contributed to the slow uptake of the WRS for EAGC.63 Similarly, Shalem reported low levels of interest in a potential WRS, which they decided not to establish.64 Farmers’ immediate cash needs contributed to side-selling in several cases (e.g.

56 Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Mount Meru Endline Qualitative Case Study. 57 Itad (2018) Kaderes Endline Qualitative Case. 58 Itad (2018) EAGC Endline Qualitative Case Study. 59 Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Kaderes 2017 Monitoring Study Report. 60 Itad (2018) Joseph Initiative Endline Qualitative Case Study. 61 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Joseph Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. Itad (2018) Portfolio Review. 62 Itad (2018) EAGC Endline Qualitative Case Study. 63 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. Itad (2018) Portfolio Review. 64 Itad (2018) Portfolio Review.

FINAL EVALUATION

Page | 30

Kilimo, Mount Meru and WFP). Other factors also contributed to side-selling; for instance, where off-takers delayed payments or market prices increased and other buyers offered higher prices and farmers’ rational response was to sell to others.65

For whom: With respect to access to storage and aggregation by gender, 44% (on average) of all FTESA farmers accessing new and/or improved storage and aggregation services were female (logframe output 1.1). This ranged from 27% for Kaderes to 71% for Shalem, with EAGC, Kilimo, Raphael, Yak, Seba, Shalem, Sosoma and WFP above the average of 44%.66 By age, the Kaderes monitoring survey found that there was no significant difference in storage rates between farmers under 40 years of age and those 40 and over.67 Female farmers made up 25% and 47% of FTESA farmers accessing improved market information systems for Esoko and EAGC respectively (output 1.2).68

Improved access to training on post-harvest handling, including grades and standards

Training on post-harvest handling (including grades and standards) and good agricultural practices, in combination with the incentive of better prices, enabled and motivated many farmers to improve the quality of their grain produced and stored (e.g. Classic, EAGC and Shalem) and also contributed to building trust among farmers concerned about combining their grain with others, leading to increased aggregation and storage rates and reduced post-harvest losses.69 However, there were a few cases of co-mingling, undermining trust in storage and aggregation facilities (e.g. EAGC).70 Also, the adoption and adherence to grades and standards for some farmers (e.g. Joseph farmers) remains a challenge, where the level of effort required to meet the standards in some cases acts as a disincentive, especially when there are no assurances that buyers will purchase their grain at higher prices to reflect better quality.71

For whom: Female farmers made up 45% (on average) of all FTESA farmers accessing improved value chain coordination, including the application of grades and standards (output 1.3). This ranged from 17% for Esoko to 70% for Shalem, with Afritec, EAGC, Kilimo, Raphael, Seba, Shalem, Sosoma and WFP above the average of 45%.72 By age, the Kaderes monitoring survey found that differences in the proportion of farmers suffering post-harvest losses are small and not significant between those aged 40 or more years and those who are younger, with slightly lower losses among the older age group.73

Improved access to finance

Smallholder farmers remain a high-risk borrower for most banks and there was little evidence that this longstanding perception was changing. Across the cases (e.g. EAGC, Kilimo and Mount Meru), there is strong evidence that there remains a lack of trust and a perception that smallholder farmers, particularly at an individual level, are high-risk customers.74 Many farmers lack financial literacy and this leads to high default rates (e.g. Kilimo), although some grantees (e.g. Shalem) worked with farmers to improve their financial literacy, thus reducing default rates.75 Mount Meru’s attempts at facilitating farmers’ access to credit failed, principally given the model was set up to lend to individual farmers.76

65 Itad (2018) Mount Meru Endline Qualitative Case Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Portfolio Review. 66 FTESA (2018) ‘PMU end-of-project grantee data’. 67 Itad (2018) Kaderes 2017 Monitoring Study Report. 68 FTESA (2018) ‘PMU end-of-project grantee data’. Only two grants contribute to this output. 69 Itad (2018) Portfolio Review. Itad (2018) EAGC Endline Qualitative Case Study. 70 Itad (2018) Portfolio Review. Itad (2018) EAGC Endline Qualitative Case Study. 71 Itad (2018) Joseph Initiative Endline Qualitative Case Study. 72 FTESA (2018) ‘PMU end-of-project grantee data’. 73 Itad (2018) Kaderes 2017 Monitoring Study Report. 74 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Mount Meru Endline Qualitative Case Study. Itad (2018) Portfolio Review. 75 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Portfolio Review. 76 Itad (2018) Mount Meru Endline Qualitative Case Study.

FINAL EVALUATION

Page | 31

Farmers typically lack credit/banking history and bank guarantees to reduce their risk profile and there is often insufficient time to create ‘creditworthiness’ within the lifetime of a project (3–5 years). They also continue to face challenges in raising collateral. In many cases, banks remain reluctant to consider stocks as collateral, partly due to collateral management issues at warehouses (e.g. co-mingling) (e.g. EAGC).77 Also, in some cases (e.g. EAGC, Kilimo and Mount Meru), the presence of formal contracts with guaranteed prices and markets did not provide enough comfort for banks.78

Where farmers successfully accessed credit, this typically depended on a set of conditions being present: for example, (i) lending to a registered farmer-based organisation (not an individual) that can provide a group guarantee; (ii) where trusted suppliers provided inputs; and (iii) where a contract was in place with a buyer, especially where the buyer repaid the loan and paid the difference to the farmer-based organisation, creating an additional incentive for farmers to aggregate and sell via the farmer-based organisation (e.g. Farm Africa).79 WFP presents another success story, in which the high loan repayment rate by its farmers in one season encouraged other financial institutions to also provide loans to WFP farmers in subsequent seasons. Through farmer-based organisations’ exposure to formal aggregated sales, quality input utilisation and training opportunities, this enhanced their bankability and, along with good repayment rates, raised the confidence of financial institutions in doing business with smallholder farmers.80 Despite this evidence of success in improving access to finance, there is no data reported against logframe output 1.4 for Farm Africa and WFP.81

For whom: Female farmers made up 37% of all FTESA farmers accessing warehouse receipts and supplier credit (output 1.4). This ranged from 27% for Kaderes and 84% for Shalem.82 Shalem reported that female smallholder farmers have access to funds through village savings and loans (which Shalem provided training on) and they have opened bank accounts.83

What indications are there of sustainability?

Signs of behaviour change (on the part of both farmers and buyers) indicate sustainability. There are signs of enduring behavioural change among smallholder farmers, where grantees report the adoption of new practices (e.g. improved post-harvest handling). Grantees, such as Pee Pee, which sells PICS bags to farmers, report that healthy sales indicate smallholder farmers have adopted the use of their bags.84 There is evidence across several grantees (e.g. EAGC, Farm Africa and Shalem) that storage and aggregation increased, post-harvest losses reduced and farmers’ bargaining power increased, improving power relations between farmers and buyers (see section 4.4.2), as well as that these changes will continue as farmers see the benefit (i.e. increased sales through better market access).85 However, for some grantees, delays in implementation reduced the opportunity to test, adopt and adapt interventions, reducing the potential sustainability of such interventions.

The Kilimo and Kaderes grants demonstrate a change in mindset on the part of farmers who have come to accept beans as an important new crop that can improve household incomes.86 Kilimo has several success stories of participating farmers making large investments in bean production, but there are also examples of smallholder farmers ‘dis-adopting’ beans in cases where their crops failed and they saw no return on their investments. Other grantees report that farmers have been resistant to adopting a new crop, especially soy

77 Itad (2018) EAGC Endline Qualitative Case Study. 78 Itad (2018) Mount Meru Endline Qualitative Case Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) EAGC Endline Qualitative Case Study. 79 Itad (2018) Farm Africa Thematic Study. 80 Itad (2018) Portfolio Review. 81 FTESA (2018) ‘PMU end-of-project grantee data’. Farm Africa (2018) Final Report. WFP (2018) Final Report. 82 FTESA (2018) ‘PMU end-of-project grantee data’. 83 Itad (2018) Portfolio Review. 84 Itad (2018) Portfolio Review. 85 Itad (2018) Farm Africa Thematic Study. Itad (2018) EAGC Endline Qualitative Case Study. 86 Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study.

FINAL EVALUATION

Page | 32

(e.g. Seba and Mount Meru), partly due to the short timeframe in which to encourage adoption and lack of demonstration effects.87

In some cases, it appears buyers’ adopted behaviour change will also continue. Lead firms Raphael and Shalem both express their commitment to new business models adopted and providing farmers with a guaranteed market. In contrast, Joseph’s smallholder-focused business model was not financially sustainable and ceased to function.88

Some grantees (e.g. EAGC, Farm Africa and Sosoma) have raised concerns around the sustainability of maintaining warehouses and aggregation centres, including the equipment.89 To mitigate these risks, Farm Africa worked with warehouses and aggregation centres to help them develop business plans and identified local officials to take on aggregator roles. However, Farm Africa expressed concern that warehouse operators may not maintain best practice after the project ends, when there is no one to follow up to check. That said, EAGC can help mitigate this risk for EAGC-certified warehouses through their checks/audits.

To what extent have improved availability and use of inputs and application

of good agricultural practices increased production and trade (EQ4)?

Main findings and conclusions [strength of evidence – strong]: Several cases present good evidence of improvements in the availability and use of inputs and farmers simultaneously applying good agricultural practices due to FTESA-funded activities, although the numbers reached are low, with supplies of inputs and numbers trained lower than expected. Most progress is in training to support the application of good agricultural practices. Where farmers have applied good agricultural practices and used improved inputs, productivity and quality have improved. There is strong evidence across the cases that farmers are willing to adopt new/improved inputs and practices where the benefits are clear (demonstration effects and proof of concept) and are in line with a farmer’s own risk appetite.

The use of known and respected institutions (grantees, research institutions, input suppliers and local SACCOs) to mobilise farmers enhanced the credibility of the intervention and generated trust, especially in contexts where farmers were risk averse and had lower levels of education, leading to greater participation rates and adoption of new and/or improved inputs and practices. The use of the lead farmer approach worked well when farmers considered the lead farmer as experienced and successful, increasing their credibility, with farmers more likely to apply lessons from training provided by such farmers.

In some cases, gaps in service provision, such as lack of finance, reduced the uptake of new and/or improved inputs. Also, despite efforts to improve access to quality inputs, delays in accessing inputs at the required quantity and quality limited the benefits. Factors outside of the direct control of the programme (e.g. fake seeds, government policy leading to delays in accessing improved seed and weather-related impacts) reduced the benefits of applying better practices. There is limited evidence across the grants that these interventions have led to higher-level results (prices, sales and incomes), partly due to the stage of implementation and lack of success in finding markets (see EQ5).

What?

Several cases provide strong evidence that facilitating access to improved inputs and training in good agricultural practices has led to increased productivity and quality. However, in many cases the gains are not widespread in terms of number of farmers reached. There is limited evidence that this, alone, resulted in greater sales and higher prices.

87 Itad (2018) Mount Meru Endline Qualitative Case Study. 88 Itad (2018) Joseph Endline Qualitative Case Study. Itad (2018) Portfolio Review. 89 Itad (2018) Farm Africa Thematic Study. Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Portfolio Review.

FINAL EVALUATION

Page | 33

Access to quality inputs and training

Improved inputs: Many grantees were involved in helping farmers access improved inputs, including through input loans and other forms of financing and/or facilitating linkages with input companies, as well as assisting farmers to produce seeds. For example, Afritec produced quality-declared seeds, Mount Meru, Raphael and Yak provided improved certified seeds to farmers, Kaderes and Mount Meru supported farmers to grow seeds, WFP worked with banks to help farmers access input loans, and Musoma and Seba provided inputs on credit.90 Under its lead firm consortia model, Kilimo partnered with agricultural research institutions to produce improved seeds and recruit farmers to produce the seed, improving the availability of certified seeds to farmers.91 Kilimo linked lead firms in their consortia to input providers who provided quality-declared seeds to the farmer groups (e.g. Shalem). A partnership between Kilimo and Kaderes significantly improved access to, and use of, improved seeds. However, volumes were low so the benefits were not widespread.92 Similarly, Joseph directly procured certified seeds and fertilisers and delivered them to farmers but the quantities supplied were very small. Mount Meru reported high usage of improved seeds (estimated at 80%), but they distributed less seed than anticipated and there were significant delays in distribution.93

Numerous grantees report smallholder farmers accessing inputs through farmer-based organisations, especially lead firms (e.g. Kaderes farmers accessed improved seed through SACCOs), and often at village aggregation centres (e.g. Joseph and Farm Africa). For some (e.g. Farm Africa), aggregation centres are becoming service hubs attracting different service providers (e.g. for inputs), which is making access to inputs easier for farmers.94

Good agricultural practices: Many of the grantees offered farmers training on good agricultural practices (Afritec, Classic, Ets Nkubili Alfred & Sons (ENAS), Joseph, Kaderes, Mount Meru, Musoma, Seba, Shalem, Sosoma and WFP). Some grantees (Afritec, Kaderes, Seba, Shalem and Sosoma) trained lead farmers or engaged in training of trainers, with the assumption that they would pass on learning to other farmers. Some grantees also established demonstration farms (Afritec, Classic, ENAS, Mount Meru and Musoma) and training days (Afritec, Joseph and Seba). Joseph and Mount Meru promoted good agricultural practices through radio, while Mount Meru and Sosoma used leaflets to disseminate messages on good practices.95 For Joseph, training on good agricultural practices, coupled with demonstration plots, allowed farmers to see the productivity benefits of using certified seed as opposed to local seed.96 There is evidence from other case studies, however, that training fell short of expectations in terms of number of farmers reached: for example, for Mount Meru there were too few agronomists to deliver training services effectively across large numbers of farmers and geographical areas.97 This contrasts with the lead farmer approach taken by Kaderes, with each lead farmer responsible for training a manageable number of smallholder farmers.98

Yields and production

Several grantees reported increases in productivity (e.g. Kaderes, Kilimo, Musoma, Raphael, Shalem, Sosoma, WFP and Yak).99 In most cases, farmers simultaneously adopted increased usage of improved seeds and applied good agricultural practices. Farmers reported improved yields due to using improved seeds and

90 Itad (2018) Portfolio Review. Itad (2018) Mount Meru Millers Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study. 91 Itad (2018) Kilimo Trust Endline Qualitative Case Study. 92 Itad (2018) Kaderes 2017 Monitoring Study Report and Itad (2018) Kaderes Endline Qualitative Case Study. 93 Itad (2018) Portfolio Review and Itad (2018) Mount Meru Millers Endline Qualitative Case Study. 94 Itad (2018) Portfolio Review. Itad (2018) Farm Africa Thematic Study. Itad (2018) Joseph Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study. 95 Itad (2018) Portfolio Review. Itad (2018) Joseph Endline Qualitative Case Study. Itad (2018) Mount Meru Endline Qualitative Case Study. 96 Itad (2018) Joseph Initiative Endline Qualitative Case Study. 97 Itad (2018) Mount Meru Millers Endline Qualitative Case Study. 98 Itad (2018) Kaderes Endline Qualitative Case Study. 99 Itad (2018) Portfolio Review.

FINAL EVALUATION

Page | 34

applying good agricultural practices.100 Kaderes and Kilimo farmers attributed the improved yields largely to the use of improved seeds, coupled with the application of good practices. One farmer group working with Kaderes reported productivity increases of 15–20%, with other groups reporting increases of up to 100% compared to traditional seeds.101 For Kilimo, increased use of improved seeds by farmers led to productivity increases of 30% on average, with production per acre more than tripling in some cases.102

Lead firms Raphael and Musoma reported that farmers nearly tripled (Raphael) or doubled (Musoma) their productivity.103 Off-takers Yak and Sosoma reported yield increases of between 25% and 200%.104 WFP reported a 21% increase in yields for its farmers in Rwanda due to training and the subsequent adoption of good agricultural practices. Some farmers also reported improved quality of outputs (e.g. Kaderes and WFP).105


Known, respected actors: There are several factors supported by case study evidence that contributed to activities supporting inputs and good agricultural practices working well. For example, where the interventions work through known institutions that the farmers trust, this enhances the credibility of the intervention, especially in contexts where farmers often have low levels of education and are risk averse. For example, Mount Meru targeted farmers for the soybean enterprise who had previously supplied them with sunflowers and farmers noted this as a key factor influencing their decision to take up the new riskier crop: they already had experience with the grantee and knew and trusted them.106 This was also the case for Kaderes, where registered farmers were already producing and selling coffee supported by Kaderes. Kaderes reached farmers through existing institutional structures that farmers trusted (local SACCOs), helping mobilise farmers to join the project.107 Women traditionally grew beans as a food crop, and continued to do so as a cash crop, but the male farmers, who were growing coffee for Kaderes, also started to grow beans.108

Some cases suggest that the lead farmer approach worked well, especially when farmers consider the lead farmer as experienced and successful, increasing their credibility (e.g. Kaderes), with farmers more likely to apply lessons from training provided by such farmers.109 The perceived credibility of the lead farmer and the grantee directly influenced participation rates (e.g. Kaderes).

Mutually reinforcing partnerships: Partnerships with credible research institutions, extension agencies and well-known suppliers was an important factor in getting farmers to embrace new and/or improved seed varieties (e.g. Kaderes and Kilimo).110

The success of grants employing consortia-type models (e.g. Farm Africa, Kilimo and WFP) in establishing structured relationships providing support and facilitating access to services along the entire value chain – including access to buyers, inputs, finance and training on good agricultural practices – relies on effective partnerships and collaborations.111 Evidence from the grants suggests that the key factors of success include ensuring each actor is clear on their roles and responsibilities and understands that interventions complement

100 Itad (2018) Joseph Endline Qualitative Case Study (August 2018); Itad (2018) Kilimo Endline Qualitative Case Study; Itad (2018) EAGC Endline Qualitative Case Study; Itad (2018) Kaderes Endline Qualitative Case Study; Itad (2018) Portfolio Review. 101 Itad (2018) Kaderes Endline Qualitative Case Study. 102 Itad (2018) Portfolio Review and Itad (2018) Kilimo Trust Endline Qualitative Case Study. 103 Itad (2018) Portfolio Review. 104 Itad (2018) Portfolio Review. 105 Itad (2018) Kaderes Endline Qualitative Case Study and Itad (2018) Portfolio Review. 106 Itad (2018) Mount Meru Millers Endline Qualitative Case Study. 107 Itad (2018) Kaderes Endline Qualitative Case Study. 108 Itad (2018) Kaderes Endline Qualitative Case Study. 109 Itad (2018) Kaderes Endline Qualitative Case Study. 110 Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study. 111 Itad (2018) Portfolio Review. Itad (2018) Farm Africa Thematic Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study.

FINAL EVALUATION

Page | 35

one another and thereby contribute to each other’s success. MoUs, as in the case of Kilimo, have helped contribute to the success of such models.

Demonstration effects: There is strong evidence across the cases that demonstration effects reinforce training and generate trust, increasing uptake and effective application of interventions (e.g. Kaderes, Joseph and Virtual City).112 Trainings and demonstrations by trusted actors helped farmers understand what they needed to do and the requirements to use new/improved inputs and methods effectively and, importantly, the benefits that accrue from adopting new approaches. For Kaderes, many farmers described a ‘lightbulb moment’ through demonstration effects when they witnessed the success of farmers who had put the training into practice.113 Virtual City provided evidence of the importance of demonstrating that interventions work and deliver benefits beyond doing ‘business as usual’. Some shopkeepers were reluctant to use the app to order inputs as they did not see anything wrong with the existing system and, more importantly, were not aware or convinced of the benefits of changing their ways of doing business, so lacked the incentive and motivation to participate. It also suggests that the intervention may not be responding to a real need.114

Gaps in service provision: Some of the grantees that did not take a comprehensive approach to providing, or facilitating, support along the value chain provide examples where a lack of access to a specific service hampered farmers’ ability to participate successfully in the intervention. For instance, a lack of initial capital to purchase inputs (e.g. seeds, tarpaulins, etc.) hampered efforts by Kaderes farmers to participate in the intervention.115 Other grantees also highlighted farmers’ inability (Musoma and Seba) to pay for inputs.116

Poor-quality inputs and delays: Moreover, some grantees and farmers continued to face challenges accessing good-quality inputs (even where the grantee took a ‘whole of the value chain’ approach to support), with several citing factors outside their control – such as the proliferation of fake seeds – as a major constraint to productivity and quality and, consequently, sales (e.g. Farm Africa, Kilimo and WFP).117 In some cases, when there was access to quality inputs, there were several cases of late delivery of inputs delaying planting, as well as insufficient quantities available, limiting farmers’ ability to increase yields and production (Afritec, Kilimo, Mount Meru, Seba and WFP).118 Government policy created the delays in some cases; for example, the Rwandan government’s subsidy scheme delayed access to inputs for farmers under the WFP grant and the export ban in Zambia delayed farmers in Tanzania working with Mount Meru accessing seed, significantly affecting distribution and uptake.119 However, FTESA’s policy-influencing efforts helped facilitate the removal of export bans in Zambia (see Section 4.7).

Frequency of training and demonstrations: Some of the cases also raised concerns regarding the frequency and sustainability of training and demonstrations (e.g. Farm Africa and Joseph).120 Repeated exposure of large numbers of farmers to a new method is more likely to generate behavioural change that others adopt, leading to widespread changes in behaviour and norms. Adoption and replication occur where farmers can see what is happening. Farmers require repeated interactions with extension officers and follow-up demonstrations and training to build the confidence that new methods are worth adopting. Moreover, training needs change over time as farmers face new challenges and opportunities, requiring farmers to adapt their methods. However, for both Farm Africa and Joseph training was a one-off activity organised by the grantee with no indication that other permanent market actors, with an incentive to scale and adapt the training (e.g. buyers, input suppliers, etc.), would step in to continue this function.

112 Itad (2018) Portfolio Review. Itad (2018) Kaderes Qualitative Case Study. Itad (2018) Joseph Initiative Qualitative Case Study. Itad (2018) Virtual City Endline Qualitative Case Study. 113 Itad (2018) Kaderes Endline Qualitative Case Study. 114 Itad (2018) Virtual City Endline Qualitative Case Study. 115 Itad (2018) Kaderes Endline Qualitative Case Study. 116 Itad (2018) Portfolio Review. 117 Itad (2018) Farm Africa Thematic Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Portfolio Review. 118 Itad (2018) Mount Meru Endline Qualitative Case Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Portfolio Review. 119 Itad (2018) Portfolio Review. 120 Itad (2018) Farm Africa Thematic Study and Itad (2018) Joseph Initiative Endline Qualitative Case Study.

FINAL EVALUATION

Page | 36

Climate/weather: The most commonly cited constraint to production, irrespective of the application of improved methods, was the weather. Almost half of the grantees (Afritec, EAGC, Farm Africa, Kaderes, Kilimo, Mount Meru, Yak, Musoma and WFP) cited late rains or drought as a significant constraint, as well as flooding (Kaderes). Pests, especially outbreaks of fall armyworm in East Africa, have also taken their toll on crops recently.121

For whom: Female farmers made up 44% (on average) of all FTESA farmers using improved inputs (output 2.2), ranging from 27% for Kaderes to 71% for Shalem, with Afritec, EAGC, Kilimo, Raphael, Seba, Shalem, Sosoma and WFP above the average of 44%.122 The Kaderes monitoring survey found only 26% of female, compared to 43% of male, farmers used improved inputs.123 The PMU and grantees, such as Farm Africa, reported the importance of accessing improved inputs through farmer collectives for female smallholder farmers.124 The Joseph quantitative survey found female main farmers harvested less maize than male farmers on average, but the effect was not significant and nor was there a statistically significant difference between Joseph farmers and the counterfactual group.125 The Kaderes survey also found that on average female farmers harvested less than male farmers.126

By age, for Kaderes farmers, there was no significant difference in the size of a farmer’s bean harvest based on age (either under 40 years of age or 40 plus); younger farmers (under 40 years) obtained slightly higher prices for beans than farmers aged 40 or older, but the difference was not significant.127 The Joseph quantitative survey found older farmers have higher maize harvest volumes on average, but again this effect was not significant and nor was there a statistically significant difference between Joseph farmers and the counterfactual group.128

What indications are there of sustainability?

Sustainability hinges on continued supply and demand of improved inputs and repeated training and demonstration by permanent market actors who have the incentive to continue these functions, with farmers seeing the benefit and willing to pay for inputs and services, and spreading new behaviours to others. Positive demonstration effects have the potential to crowd in other farmers and deliver sustainable and systemic change. In several cases, farmers have internalised good agricultural practices as interventions delivered ‘proof of concept’, demonstrated by farmers’ continued demand for improved inputs (e.g. ENAS, Kilimo, Shalem and Yak).129 However, widespread changes are yet to materialise, in some cases due to the early stage of implementation. There were also examples where farmers understood the benefits of using improved inputs and wanted to use them but were unable to pay for such inputs (e.g. Afritec, Kaderes, Musoma and Seba).130 In some cases (Kilimo and Mount Meru), farmer behaviour limited the adoption of new crops (e.g. soya) and therefore use of improved inputs.131

Additionally, securing market access and making better deals, which in turn deliver benefits for the farmers (e.g. increased incomes), provide an incentive to continue applying improved inputs and practices. However, in several cases (discussed in the next section), better market access has not yet materialised, reducing the incentive to apply improved methods.

121 Itad (2018) Portfolio Review. 122 FTESA (2018) ‘PMU end-of-project grantee data’. 123 Itad (2018) Kaderes 2017 Monitoring Study Report. 124 Itad (2018) Portfolio Review. 125 Itad (2018) Joseph Initiative Quantitative Impact Assessment. 126 Itad (2018) Kaderes 2017 Monitoring Study Report. 127 Itad (2018) Kaderes 2017 Monitoring Study Report. 128 Itad (2018) Joseph Initiative Quantitative Impact Assessment. 129 Itad (2018) Portfolio Review. Itad (2018) Kilimo Trust Endline Qualitative Case Study. 130 Itad (2018) Portfolio Review. Itad (2018) Kaderes Endline Qualitative Case Study. 131 Itad (2018) Mount Meru Endline Qualitative Case Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Portfolio Review.

FINAL EVALUATION

Page | 37


This section builds on sections 4.2 (EQ3) and 4.3 (EQ4), which focus on the supply side (e.g. inputs) and supporting functions (e.g. storage and aggregation), to explore how FTESA supported market integration (essentially bringing together supply and demand) and whether and how farmers are benefiting from increased integration in structured markets through better and more deals, greater volumes sold, higher prices and higher incomes (where evidence is available). It also explores who is benefiting.

Main findings and conclusions [strength of evidence – strong]: Increased smallholder farmer participation in structured regional markets was a central focus for FTESA and grantees. Most of the grantees worked with smallholder farmers with existing or potential tradeable surpluses, with women accounting for approximately 40% of participants. Several of the grants helped smallholder farmers improve yields, production and quality, as well as store and aggregate greater volumes, making them ‘market-ready’ and able to sell to a wider range of buyers. Unfortunately, lack of data and information on specific groups limited analysis on these different groups.

The evidence strongly suggests that lack of storage, aggregation and collective marketing is a fundamental barrier to improving farmers’ position in the market and their ability to command higher prices. Several cases show that the interventions helped farmers understand that storing produce and deferring sales can lead to higher prices and aggregating good-quality produce and marketing collectively can lead to access to better markets as buyers prefer the convenience of purchasing aggregated produce that reaches required standards. The cases provide evidence of changes in the mindset of participating farmers toward aggregation and selling as a group, and examples of farmers’ bargaining power and ability to negotiate better prices increasing when farmers increased and improved aggregation and sold as a group, creating competition among buyers for their produce. In some cases, the programme facilitated connections between farmer groups and buyers leading to deals, increased sales and better prices.


What?

Improved market integration, higher volumes sold and better prices

As discussed under section 4.3, several cases provide strong evidence that facilitating access to improved inputs and training led to increased productivity and quality. Similarly, section 4.2 demonstrates several cases where post-harvest losses reduced. However, in many cases, the gains are not widespread in terms of number of farmers reached. This has translated into more sales at better prices for some farmers, but market access remains a key constraint for many of the farmers interviewed. Farmers often relied on the grantee to provide the market, which in some cases fell short of expectations.

Linking buyers and sellers: Several grantees facilitated linkages between farmers and buyers (e.g. EAGC) or directly linked them through contracts (e.g. Kilimo and WFP), while some grantees intended to buy directly from farmers (e.g. Joseph, Kaderes, Mount Meru and Shalem).132

Volumes sold and traded: For five grants (Farm Africa, Musoma, Raphael, Shalem and WFP), sales volumes significantly exceeded expected results.133 Farm Africa and WFP recorded the highest volumes sold. However,

132 Itad (2018) Portfolio Review. Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Joseph Initiative Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Mount Meru Endline Qualitative Case Study. 133 Itad (2018) Portfolio Review. Itad (2018) Farm Africa Thematic Study.

FINAL EVALUATION

Page | 38

seven grants (EAGC, Joseph, Kaderes, Kilimo, Mount Meru, Seba and Yak) fell short of expectations.134 EAGC, Kilimo and Seba were the worst performers: Kilimo achieved 25% of the target, EAGC achieved less than 5% of the target (sales through the G-Soko trading platform) and Seba achieved only 1% of the target.135 However, the targets may have been unrealistic in many cases. Most relevant grantees reported cross-border trade, especially off-takers (Classic, Joseph, Mount Meru, Seba and Yak), but also lead firms (Kaderes, Musoma, Raphael and Shalem) and Farm Africa.136 WFP only sells to local buyers and off-takers but, like with other grants, some of these buyers may tap into export markets in neighbouring countries for onward sales.137

Prices received: Twelve grantees (EAGC, Farm Africa, Joseph, Kaderes, Kilimo, Mount Meru, Musoma, Raphael, Seba, Shalem, Sosoma and WFP) reported higher prices, reporting prices either above farm-gate/market prices or increases in prices due to the FTESA-funded interventions.138

Incomes: The PMU did not require grantees to report on farmer incomes, but several reported increased incomes (Classic, Farm Africa, Kaderes, Kilimo, Raphael, Shalem and WFP). Some quantified these increases while others suggested incomes increased because of higher prices and increased sales due to greater yields, reduced post-harvest losses, better quality and more deals secured. In terms of quantifiable increases, Classic reported incomes increasing by 30%, Kaderes farmers’ gross margins increased by over 100%,139 Kilimo reported farmers receiving a 44% increase in incomes, Raphael reported average increases in income of £691 per farmer and WFP reported farmers engaged in their interventions earning incomes 58% higher than non-participating farmers.140 For Joseph, however, the quantitative survey found that farmers supported by their grant experienced a fall in their incomes (see Box 1). The Joseph survey showed that ‘less poor’ households harvest more maize, sell larger volumes of maize, and have higher maize gross margins, and that households with more dependent members are more likely to be impoverished and food insecure.141

Box 1: Joseph Initiative142

For Joseph, there is no indication of a difference in farm performance between the counterfactual and intervention groups. Maize harvest volumes declined more for the Joseph group compared to the counterfactual, and maize gross margins decreased in both groups, but the results were not significant. However, at baseline, the Joseph group had already received support and therefore we did not capture a true baseline (due to delays in commencing data collection). The decline for both groups was due to increased post-harvest losses, largely caused by an increase in pests and diseases and poor handling at harvest.

Joseph farmers reported deteriorating qualitative measures of well-being, while counterfactual farmers experienced improvements in the same indicators, resulting in approximately similar endline values for both groups. The better well-being indicators reported by the Joseph group were likely a result of the services and training already received at that time. Therefore, we cannot say with confidence whether the decline in the Joseph well-being indicators is a response of a deteriorating situation compared to the pre-project state, or just the erosion of the initial benefits of the services provided at the late baseline. However, the survey results indicate that the well-being of target farmers did not improve between baseline to endline, raising questions about the sustainability of benefits for target farmers.

134 Itad (2018) Portfolio Review. Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Joseph Initiative Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Mount Meru Endline Qualitative Case Study 135 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Portfolio Review. 136 Itad (2018) Portfolio Review. Itad (2018) Joseph Initiative Endline Qualitative Case Study. Itad (2018) Mount Meru Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. 137 Itad (2018) Portfolio Review. 138 Percentage price increases: Farm Africa (up to 21%). Price differential (compared to market/farm-gate prices): EAGC (15–30% higher than market prices); Farm Africa ($0.40/kg more than farm-gate prices); Joseph (10Ush/kg above market price); Kaderes (26% higher than competitors); Kilimo (up to 25% higher than market prices); Mount Meru (100Tsh/kg and 1.5ZMW/kg more than market price); Musoma (£0.05/kg more than market price); Raphael (£54/MT more than farm-gate price); Seba (0.4ZMW/kg more than market price); Shalem (2–3Ksh/kg more than market price); Sosoma (15Rwf/kg more than market price); WFP ($12/kg more than market price in Rwanda). Itad (2018) Portfolio Review. 139 Itad (2018) Kaderes 2017 Monitoring Study Report. 140 Itad (2018) Portfolio Review. 141 Proxied using gross margins. Source: Itad (2018) Joseph Quantitative Impact Assessment. 142 Itad (2018) Joseph Quantitative Impact Assessment.

FINAL EVALUATION

Page | 39


Improved market integration, higher volumes sold and better prices

Deferring sales: In several cases, farmers’ knowledge about the market improved and access to storage and aggregation facilities increased, enabling them to make better decisions regarding selling versus storage, encouraging them to store until prices increased, while applying the knowledge acquired through training to ensure they maintained the quality of produce (e.g. Farm Africa and Kaderes).143 However, lack of progress in establishing storage and aggregation facilities and WRSs for some grantees (Kaderes and Mount Meru) reduced the opportunity to improve returns by deferring sales through storing.144

Improved bargaining power: There is evidence that farmers’ bargaining power and negotiating skills increased (e.g. EAGC, Farm Africa and Shalem), affecting power relations in the value chain and improving the position of farmers, as well as reducing the extent to which farmers are price takers (a key objective of the FTESA programme).145 Through training and advice, farmers realised the benefits of collective marketing and that they can negotiate and command a higher price with buyers for larger, aggregated quantities that reach required quality standards that provide convenience for buyers wanting to source larger quantities.146 Kaderes and Shalem provide evidence of female farmers benefiting from training, with female farmers reporting that they have better knowledge about how markets and traders work, how to operate as a business and when to store/sell.147 Also, competition among buyers for aggregated quality produce helped increase prices in several cases (e.g. EAGC, Farm Africa, Joseph and Kaderes).148

Market access: At present, the integration of smallholder farmers into structured regional markets is not widespread because access to new and better markets remains a challenge. However, FTESA has helped ‘lay the foundations’ for greater integration in future. Farmers often still need assistance to find and reach out to a wider range of buyers, including those willing to compete and pay more for aggregated quantities that reach required quality standards. For example, farmers supported by the Joseph grant expected Joseph to buy their produce, despite there being no guarantee that Joseph would. Initially, farmers sold to Joseph at a higher price (8% premium) and other buyers also raised their prices; however, farmers that we interviewed had not sold to Joseph in the last two years and prices fell to pre-intervention levels, indicating that reduced competition between buyers for their produce led to lower prices.149 Kaderes farmers reported selling beans via their ‘usual’ routes to traders at the farm gate when Kaderes only purchased small quantities of beans as they were waiting to complete the warehouse.150

For EAGC, few trades took place via the G-Soko trading platform due to various challenges associated with its development and roll-out (see Box 2), which reduced the opportunity to channel quality produce (from across relevant FTESA grantee interventions) through the platform and for sellers to access a wider range of buyers willing to compete for quality aggregated quantities.151

143 Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. Itad (2018) Portfolio Review. 144 Itad (2018) Mount Meru Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study. 145 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. Itad (2018) Portfolio Review. 146 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. 147 Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) EAGC Endline Qualitative Case Study. 148 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Joseph Initiative Endline Qualitative Case Study. 149 Itad (2018) Joseph Initiative Endline Qualitative Case Study. 150 Itad (2018) Kaderes Endline Qualitative Case Study. 151 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study.

FINAL EVALUATION

Page | 40

Box 2: Challenges developing and rolling out G-Soko152

While G-Soko linked some buyers and sellers, numbers using the platform and volumes traded fall significantly short of targets. The critical mass of participation to achieve market liquidity and price discovery is a long way from materialising. Where trades took place, these often used only some of the G-Soko functions (e.g. clearing and settlement). In several cases, buyers (e.g. Raphael) accessed G-Soko to find suppliers and then did the transaction ‘offline’, contacting the seller directly. In other cases, EAGC linked buyers with sellers offline, who then made a deal and used the clearing and settlement function to do the payment. In these cases, the ‘deal’ itself did not take place online. Using the clearing and settlement account reduces payment default and provides confidence for both the buyer and seller, which is particularly useful when dealing with new buyers/sellers, and was the most successful aspect of G-Soko.

Reasons for low utilisation rates:

• Delays in rolling out: The design, development and roll-out took longer than expected, partly due to technical difficulties during roll-out (see below).

• Slow change in mindset: Most buyers/sellers prefer using existing relationships to buy/sell grain, largely since G-Soko is yet to demonstrate itself as a trusted trading platform.

• Lack of full functionality: EAGC spent considerable time striking deals offline, with inadequate effort spent improving the efficiency of the platform which itself is meant to facilitate deals. More effort is necessary to make the G-Soko platform and all its modules operational, efficient and desirable to users, which, if self-sustaining, would reduce the need for EAGC to themselves establish business-to-business linkages. As one interviewee said “if all of this worked, G-Soko would be a very useful platform for trade”.

• Perception of unnecessary charges and better deals off platform: The fee/commission charged for trading via G-Soko is a disincentive for several potential users. Many of the buyers can get better deals off the platform, with several interviewees saying that prices on the platform are too high.

• IT challenges: Several users of G-Soko had difficulties accessing the software, discouraging buyers and sellers from using the platform. Examples include: software crashing; limited capacity at village aggregation centres and warehouses to use the software; users not using the software correctly; poor internet access; power outages; lack of access to smartphones; and lack of access to all modules. These issues were partly due to the rapid roll-out of G-Soko, with inadequate testing.

• Slow process: For most buyers, while they see the potential value in the G-Soko platform, the priority is to use whichever route is easiest and quickest when doing spot trades. Several users considered the G-Soko process too slow, with auctions taking place only once a week and turnaround times of one week. These buyers typically purchase on the open market with turnaround times of 1–2 days. Buyers interested in using spot platforms are typically ‘just in time’ buyers, so the platform does not serve them well. Some potential users suggested the platform could add value by facilitating forward trades, but this is not yet possible.

• Low volumes: Currently, the level of consolidation (i.e. volumes and scale) and the regularity of flows on the platform are too low to make using the platform appealing, especially for larger buyers. Larger buyers find it less efficient and more expensive to participate on a platform which brings to the market fragmented and irregular flows from farmers, rather than consolidated regular volumes of commodity that flow through wholesale markets or that come to their own depots from traders.

• Lack of transparency: Several buyers and sellers complained that they face limitations on what they can view (e.g. trades completed, prices, etc.) through the G-Soko platform, leading to comments about lack of transparency on market dynamics and potential conflict of interest, especially regarding prices.

• Inadequate oversight and management: Several interviewees raised concerns regarding the concentration of decision-making authority within EAGC and the lack of adequate and credible regulatory oversight, reducing transparency and trust in the system and therefore its use.

• Delays in rolling out the WRS and price uncertainty: The roll-out of the WRS stalled and banks have concerns regarding the current collateral management systems that discourage them from lending against commodities. Moreover, uncertainty on price make it risky for financiers to offer finance for stored commodities. Financiers are more likely to finance against offtake agreements (forward contracts) with price commitments issued by credible buyers.

• Government interference and unpredictable markets: On-and-off export bans (e.g. in Tanzania) and uncertain market conditions for maize in Kenya partially contributed to lower than expected trading volumes on the platform. Moreover, some of the interviews questioned the underlying theory behind storage and WRS – i.e. that deferred sales would always lead to higher prices. The anticipated level of price uplift later in the season is often uncertain or insufficient to justify deferring sales, undermining the theory underpinning G-Soko and other FTESA interventions. For example, government involvement in the maize market in Kenya has led to unpredictable prices (e.g. recent imports from Uganda flooding the market due to government intervention and prices plummeting at a time when they should be increasing); and where there are two harvests in one year (e.g. Uganda) or bumper harvests (as in 2017), potential price increases after storing are diminished.

152 Itad (2018) EAGC Endline Qualitative Case Study.

FINAL EVALUATION

Page | 41

In addition, some farmers remained reluctant to try a crop that was quite new to them (e.g. soya), despite the grantee (Mount Meru) addressing several important contextual factors that might usually act as constraints to farmer participation (e.g. making sure farmers did not face costs to participate, working with existing growers who trusted the grantee, providing inputs on credit and guaranteeing prices and markets).153 For Kilimo and WFP, some farmers found the payment terms not acceptable (e.g. due to farmers preferring immediate cash payments rather than on credit, the prices offered being below local market prices and delays in collection and therefore payment). This led to side-selling, with farmers selling to other buyers who could offer prompt payments, despite farmers having signed contracts with buyers.154 With respect to cross-border trade, the main challenges faced included transport costs and export bans (e.g. in Tanzania and Zambia) (Mount Meru, Farm Africa, Kilimo and EAGC).155

Reaching scale: Above we discuss evidence at the individual grant level. However, FTESA’s expected impact and outcome went beyond individual projects, anticipating that the whole programme would lead to stabilised prices and smaller price differences between hungry and harvest seasons and deficit and surplus areas at the regional level, benefiting both producers and consumers across nine countries. Similarly, the programme aimed to increase regional trade across all countries. Given the scope of these interventions, the impact is more localised as the programme did not reach the level of scale required for such effects to materialise. As discussed at mid-term, the successful rolling out of the G-Soko platform, working with complementary grants, had the greatest potential in generating any substantial scale effects and region-wide impact but the platform failed to take off.

For whom

Smallholder engagement: Findings from across the evaluation modules show that the involvement of smallholder farmers is central to the programme. Almost all grantees refer to the farmers they work with as smallholder farmers, but few report the size of landholdings.156 According to stakeholder (grantee) survey responses, smallholder farmers are the most common beneficiaries of grantee FTESA-funded projects.157 Smallholder farmers are the most commonly cited beneficiary (90% of respondents) of FTESA-funded projects, followed by aggregators (70%), their own firm (50%) and input suppliers (50%).158

Many of the grantees (EAGC, Farm Africa, Joseph, Kaderes, Kilimo, Musoma, Sosoma, WFP and Yak) provided evidence indicating that the farmers they work with are smallholders, with average landholdings ranging from one to 20 acres.159 Three reported that their interventions enabled farmers to increase their acreage for growing staple food crops. A few grantees either struggled to target smallholder farmers (Mount Meru) or offered examples of working with farmers with larger landholdings (Shalem and Yak).160

Many smallholder farmers are in remote areas that are far from warehouses, which usually have minimum delivery quantities that smallholders cannot attain. In addition, most do not have the capacity to produce the volumes and quality required in the market.161 During the early stage of programme implementation, FTESA became aware of the acute challenge of reaching remote or low-capacity smallholder farmers. As a result, FTESA scaled up efforts to support farmer-based organisations and village aggregation centres to bring

153 Itad (2018) Mount Meru Millers Ltd Endline Qualitative Case Study. 154 Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Portfolio Review. 155 Itad (2018) Portfolio Review. Itad (2018) Mount Meru Millers Ltd Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) EAGC Endline Qualitative Case Study. 156 Itad (2018) Portfolio Review. 157 Itad (2018) Stakeholder Survey. 158 Itad (2018) Stakeholder Survey. 159 Itad (2018) Portfolio Review. Itad (2018) Farm Africa Thematic Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Joseph Initiative Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study. 160 Itad (2018) Portfolio Review. Itad (2018) Mount Meru Millers Ltd Endline Qualitative Case Study. 161 FTESA (2016) Enhancing the Poverty Impact of FTESA: Mainstreaming Poverty within Operations and Systems. Itad (2016) FTESA Mid-term Evaluation.

FINAL EVALUATION

Page | 42

together quantities from smallholder farmers, marketing collectively, as well as supporting a range of FTESA grantees to source crops through contract farming.162

While the programme is set up to support smallholder farmers, the intended beneficiaries are not the ‘poorest of the poor’ but rather those farmers with an existing tradeable surplus or the potential to generate such a surplus, and who can increase that surplus. According to a study commissioned by FTESA, ‘the main beneficiaries are likely to be smallholders producing a surplus that are members of a functioning farmers group or an organised out-growers scheme’.163 As an example of beneficiary farmers being relatively better off, Kilimo used lead farmers to breed seeds and these farmers have land available for seed breeding and possess enough resources that enable them to handle the seed breeding efficiently.164 There was no systematic data collection across all grantees on the poverty profiles of farmers. However, 90% of grantees that responded to the stakeholder survey indicated that they thought impoverished households benefited from their FTESA project, with 50% claiming impoverished households benefited more than wealthier households.165 According to the EMU’s two quantitative surveys, 29% of Kaderes and 35% of Joseph farmers describe themselves as poor. For Joseph, households with a lower risk of falling below the poverty line have on average higher maize gross margins.166

Box 3: Effects on poverty (Joseph and Kaderes) 167

Analysis of quantitative data does not yield evidence that FTESA interventions have reduced farmers’ levels of poverty. The EMU quantitative surveys found the following results for changes in farmers’ poverty status using the PPI, a measure of poverty likelihood:

• Kaderes farmers’ PPI at endline was almost the same as the baseline, indicating very little change in farmers’ poverty levels. The average value of the PPI in 2017 was 46, the same as in 2015, translating to a probability of 10.9% of average households in the sample falling below the national poverty line. Kaderes farmers’ subjective well-being – measured as satisfaction with finances, job and life – in general increased slightly, with 21% of farmers saying they are currently not satisfied with their life.168

• The Joseph survey found the PPI increased significantly in the counterfactual group169 to the same endline average PPI score as for the Joseph cohort, whereas the Joseph cohort displayed almost no change between baseline and endline. Net satisfaction with respondent’s life increased significantly for the counterfactual group and decreased (although not significantly) for the Joseph group between 2016 and 2018. The proportion of respondents who say they have enough to make ends meet, or better, increased significantly for the counterfactual group and increased (not significantly) for the Joseph group.

Female engagement: The PMU reported female farmers made up 41% of all FTESA farmers benefiting from national and cross-border value chains (logframe outcome 3a).170 Grantees report gender-disaggregated data (in terms of numbers accessing, benefiting, etc.), with an average female engagement rate among smallholders of 42% across 17 grants (ranging from 25% to 71%, with Esoko, Kaderes and Musoma at the lower end and Afritec, Sosoma and Shalem at the higher end).171 All respondents to the grantee stakeholder survey indicated that they think women benefited from their FTESA project, with 50% claiming women have benefited as much as men.172 In addition, during the last 18 months of the programme, the PMU supported several grantees (Shalem, Classic, Seba, Kaderes and Kilimo) to develop gender action plans.173 There are several examples of grantees actively promoting the inclusion of women, including targeting female farmers’ groups: 174

162 Itad (2016) FTESA Mid-term Evaluation. 163 FTESA (2016) Enhancing the Poverty Impact of FTESA: Mainstreaming Poverty within Operations and Systems. Itad (2016) FTESA Mid-term Evaluation. 164 Itad (2018) Kilimo Trust Qualitative Case Study. 165 Itad (2018) Stakeholder Survey. 166 Itad (2018) Joseph Initiative Quantitative Impact Assessment. Itad (2018) Kaderes 2017 Monitoring Study Report. 167 Itad (2018) Joseph Initiative Quantitative Impact Assessment. Itad (2018) Kaderes 2017 Monitoring Study Report. 168 Itad (2018) Kaderes 2017 Monitoring Study Report. 169 The higher a household's score on a scale from 0 to 100, the lower the chance that its members are living below the national poverty line. 170 FTESA (2018) ‘PMU end-of-project grantee data’. 171 Itad (2018) VFM report 172 Itad (2018) Stakeholder Survey. 173 FTESA (2018) PMU Final Report. 174 Itad (2018) Portfolio Review.

FINAL EVALUATION

Page | 43

• Kilimo actively recruited female farmer groups into its consortia, reporting women accounting for 52% of farmers brought into structured regional markets.175 One women’s group (Cheptarit Star) transformed the group into a company and engaged in cross-border trade (300 MT) (see Box 4).176 Kilimo also reported members of the Zinduka Women’s Group using proceeds from grain sales to build or improve their houses – an example of women increasing their access to resources as a result of the FTESA-funded project (also detailed in Box 4).177

• Farm Africa actively encouraged women to take up leadership roles in farmer-based groups and aggregation centres since they found that women-led groups are better managed than male-led groups. EAGC also reported examples of women in leadership positions at village aggregation centres.

• Classic allocated leadership positions to both women and youth. It reported that this strategy was intended to engage 70% of women among smallholder farmers, although it did not achieve this.

• Shalem targeted women to improve their business and financial skills (e.g. table-top banking) and identified male ‘gender champions’ to include women in decision-making. Shalem had the highest female inclusion rate at 71%.

• Mount Meru promoted the inclusion of women by providing women with twice as many seeds as men.

Several of the grantees reported barriers to female smallholder farmers benefiting from grantees’ interventions, including cultural norms and asset ownership:178

• Farm Africa reported that some communities ‘have less belief’ in female (and youth) leadership.

• Lead firms Musoma and Raphael reported that men’s ownership and control over most farms and other agricultural resources limited the potential benefits for women.

• Off-taker Joseph reported low levels of female participation (women account for 25% of smallholders supplying Joseph) due to local cultural expectations that women are ‘expected to remain at home’.179 Households with female main farmers and poorer households have significantly lower maize sales volumes (the difference between the Joseph and counterfactual groups is not significant). However, on average, female main farmers obtained higher per kilo prices for maize than male main farmers.180

175 Itad (2018) Case Synthesis. 176 Itad (2018) Kilimo Trust Endline Qualitative Case Study. 177 Itad (2018) Portfolio Review. 178 Itad (2018) Portfolio Review. 179 Itad (2018) Joseph Initiative Endline Qualitative Case Study. 180 Itad (2018) Joseph Initiative Quantitative Impact Assessment.

FINAL EVALUATION

Page | 44

Box 4: Gendered benefits

FTESA results demonstrate that certain grantees were able to engage a substantial percentage of female farmers and contribute to positive results for farmers. However, data limitations (PMU, grantee and EMU) restrict the analysis on gender in this report (as noted under section 3.7), except for a few anecdotal examples including Cheptarit Women’s Group and Zinduka Women’s Group (see below). The original design of FTESA and the grants did not prioritise gender issues, and there was no obligation in contracts to reach out to both male and female farmers in equal measure. So, it is not surprising that monitoring to capture this is lacking, nor that existing data does not reflect significant benefits for women. Only after recommendations emerged at mid-term did the PMU begin to prioritise gender in the portfolio of grants and more broadly through the programme, including hiring a gender adviser and developing a Gender Inclusion Strategy.181 However, since mid-term, the PMU awarded only two further grants, limiting the ability to integrate gender objectives in grant activities. Nevertheless, some of the grantees prioritised gender issues irrespective of their FTESA contracts. For instance, the aims and objectives of several NGOs (e.g. Kilimo and Farm Africa) funded by FTESA target marginalised groups, including women, while companies such as Shalem also specifically target women as part of their business plan. Also, the PMU was able to work with several grantees (Kaderes, Kilimo and Shalem, as well as the two new grantees Classic and Seba) to support the development of gender action plans.

Zinduka and Cheptarit women’s groups:

• Zinduka Women’s Group, Tanzania: Through knowledge and skills gained from training by Kilimo, introducing the concept of beans farming as a business, the group has significantly increased its productivity, from 30–120kg per acre to 600–800kg per acre, and members receive higher prices for their produce due to collective marketing and increased bargaining power.182

• Cheptarit Women’s Group, Kenya: In a context where the women had high illiteracy rates and faced male-dominated structures, the women harvested lower yields and faced additional barriers to market entry compared to men. Kilimo supported the group, who run their own aggregation centres. Kilimo provided training on good agricultural practices and post-harvest handling to the group, including educating the group to specialise in crops with high demand rather than grow multiple crops (which they were previously doing). The project provided them with access to equipment (moisture meters and digitalised weighing machines), which encouraged more farmers to bring their produce to their aggregation centres as they trusted the scales and the moisture meters minimised arguments with farmers on the moisture content. As a result, the aggregation centres received better-quality produce, which led to better market access and prices due to aggregated quantities and improved quality. The groups negotiated a contract to distribute beans and maize to schools and this led to cross-border trade, with the women travelling to buy up beans for selling to schools and other organisations. In addition to increasing their earnings, the women gained social benefits in their communities and at the household level. The project helped women increase access to water in their homes, pay for school fees and construct houses, raising their social standing in the community, giving them credibility as decision-makers and financial contributors in their families, in some cases becoming equal decision-makers with their husbands. Also, the group is now a registered company, giving them better access to credit. They have also earned respect in their communities, who increasingly turn to them for advice on good agricultural practices.183

Other grantees reporting results on gender inclusion include Shalem, the grantee with the highest female engagement rate, a very high rate of women accessing credit, and reports of female farmers’ improved bargaining power. Kaderes also reported improved bargaining power on the part of its female farmers. However, the quantitative data available (from the Joseph and Kaderes surveys) shows that women harvested lower volumes than men, with Joseph’s female farmers selling less and at lower gross margins than male farmers. The Kaderes survey found there was no significant difference between the PPI scores of female versus male main farmer households, while the Joseph survey found that female main farmers had lower PPI scores than male main farmers, although the difference was not significant.

By age: the Kaderes survey found younger farmers under 40 years obtained slightly higher prices for beans than farmers aged 40 or older, but this difference is not significant.184 The Joseph survey found that older main farmers obtained slightly higher maize prices but lower maize gross margins, although again these effects are not significant. For Joseph, the age of the main farmer has no effect on maize sales volumes.185

Farmer aggregation models

Below we describe the contextual factors and mechanisms that either enable or constrain the different models that FTESA tested. The most important enablers are open communication, trust and transparency between smallholder farmers and buyers, and other value chain actors. Support from national and local government is

181 FTESA (2018) PMU Final Report. 182 FTESA (2018) PMU Final Report. 183 FTESA (2018) PMU Final Report. 184 Itad (2018) Kaderes 2017 Monitoring Study Report. 185 Itad (2018) Joseph Initiative Quantitative Impact Assessment.

FINAL EVALUATION

Page | 45

also an important enabling factor. The most commonly cited constraints are, unsurprisingly, the converse of the enablers, such as a lack of trust, which in some cases caused or led to side-selling.

• WFP and Farm Africa: Communication and ICT helped integrate smallholder farmers into WFP’s model, including its digital application linking value chain actors. WFP, like Farm Africa, reports transparent and open communication between value chain actors as being essential to promoting structured trade. However, WFP reported several threats to the model’s success, most of them also applicable to the other farmer aggregation models. In a context where most farmer-based organisations are informal and not legal entities (e.g. self-help groups or community-based organisations), individual farmers must sign contracts rather than groups – a problem that Farm Africa addressed by formalising farmer-based organisations. WFP reported that farmer-based organisations often overestimated their future production volumes and, in some cases, signed too many contracts with off-takers. WFP reported problems with smallholder farmers side-selling; for example, in Rwanda, when off-takers delayed payment at the same time as maize prices increased, smallholder farmers sold to others. Farm Africa credits its relationship with government ministries in Uganda with enabling farmer-based organisations to formally register, allowing them to do business with other formal organisations. Farm Africa also supported farmer-based organisations to register with local government in Northern Tanzania. Farm Africa also invested time in maintaining its partnerships, fostering them through regular communication.186

• Lead firms: Lead firms have attracted smallholder farmers to their model by establishing trust between them, by providing farmers with relevant information and through the promise of a stable market. For example, Musoma signed contracts with farmer-based organisations and reported that trust between farmers and buyers is an important mechanism enabling this model, which depends on transparency, accountability and timely communication. Raphael farmers continue to sign contracts with Raphael because they have successfully used, and therefore trust, their facilities and services, while Raphael consistently provided a stable market. Like WFP, Kilimo reported that side-selling due to price fluctuation was a significant threat to its model working. Kaderes leveraged its existing relationship with smallholder coffee producers who were convinced that Kaderes would provide a stable market given their experience successfully selling coffee to Kaderes.187

• Off-takers: Most off-takers (Classic, Seba, Sosoma and Yak) signed contracts/agreements with farmer-based organisations, whereas Mount Meru signed supply agreements with individual farmers. Both Seba and Mount Meru sought the involvement of local leaders, with Seba asking traditional leaders to encourage farmers to grow soy. Also, government support to farmer-based organisations (e.g. warehouses and equipment) has contributed to successful relationships between farmer-based organisations and off-takers (e.g. for Sosoma and Yak in Rwanda). Joseph did not sign contracts with smallholder farmers. For Joseph, transparency in transactions helped build trust in the early stages of implementation but levels of trust declined when Joseph stopped purchasing directly from most of the farmers without providing an explanation. Reported threats to the off-taker model focused mostly on a lack of trust: Classic reported challenges in developing trust with farmers who had previous negative experiences with middlemen; Seba reported a case where they had to rebuild trust with farmers where an agro-dealer contracted by Seba failed to deliver services to the smallholder farmers; and Mount Meru reported other buyers offering higher prices, thus leading to side-selling.188

Sustainability

There is some evidence that grantees will continue with activities carried out under FTESA beyond the life of the FTESA grant, potentially allowing for a greater maturation effect. There are also some signs of

186 Itad (2018) Portfolio Review. Itad (2018) Farm Africa Thematic Study. 187 Itad (2018) Portfolio Review. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study. 188 Itad (2018) Portfolio Review. Itad (2018) Mount Meru Millers Ltd Endline Qualitative Case Study. Itad (2018) Joseph Initiative Endline Qualitative Case Study.

FINAL EVALUATION

Page | 46

sustainability, including adoption and adaptation of new business models, new investments and changes in market roles (e.g. an EAGC-supported women’s farmer group is constructing its own warehouse while another is becoming an established regional off-taker),189 including formalising the role of farmer-based organisations (e.g. Farm Africa), which enables better access to finance, contracts, etc. The cases also offer evidence of increased production and storage and aggregation by farmers. However, access to new and better markets fell below expectations, which in turn may undermine the potential for sustainable improvements in the future. Nonetheless, several grantees (e.g. Kaderes, Raphael, Seba and Shalem) expressed their commitment to source from smallholder farmers in future, while farmers are keen to sell to buyers who have demonstrated an assured market at competitive prices.190 However, Joseph reported losses in 2017, selling maize below the price of purchase from farmers, indicating their business model is not sustainable.191

To what extent is FTESA likely to improve the functioning of national and

regional staple food markets and generate systemic change (EQ2)?

Main findings and conclusions [strength of evidence – medium]: The grants show limited evidence of systemic change and only early signs of the potential for spreading new behaviours to others, largely due to the stage of implementation where several activities are yet to demonstrate consistent and enduring benefits to participants, and where this are therefore not yet observed and spread to non-participants. While there is evidence that farmers have adopted new ways of doing business and accessed new markets, in many cases there was over-reliance on support from the grantee for inputs, services and market access, limiting sustainability and impact in the future. Some grantees did not provide support for interventions for long enough to deliver the ‘critical mass’ of consistent success required to build buy-in for the intervention and encourage others to crowd in. However, there is anecdotal evidence that FTESA has generated systemic change in some cases, as indicated by examples of behaviour change in smallholder farmers (e.g. adoption of new methods and some copying by others) and buyers offering better prices. Again, enablers of behaviour change include transparency and trust between market actors. The most frequently cited barriers were the absence of supporting rules and limited capital.

What?

We discuss systemic change in terms of behaviour change in pivotal actors – for example, smallholder farmers, farmer-based organisations, lead firms, off-takers and other buyers – as well as changes in the market’s supporting functions and in its rules, using the AAER framework (see section 3.2 and Annex 3).192 Most evidence is at the ‘adopt’ level of the framework. Given the relatively short timeframes and recent completion of the interventions at the time of fieldwork and reporting by grantees, it is ambitious to expect to see widespread evidence of adaptation, expansion and/or a wider market response. However, we do cite examples of early indications and potential future effects. In this section, we also draw together evidence across the EQs, particularly sub-evaluation questions on sustainability.

‘Systemic change’ occurs when there are changes in underlying norms of behaviour that influence individual behaviours. Norms are common beliefs that are culturally specific, with large distinctions in norms of behaviour even across localities with similar rule structures. The purpose of a systems-focused programme is to change norms in a desired direction. Attempting to achieve market-level changes through a programme such as FTESA, which provides grants in the hope that promising interventions surface, requires time to experiment and learn before identifying and scaling up such interventions, with other market actors crowding in.

Overall, the programme shows limited evidence of systemic change and only early signs of the potential for spreading new behaviours to others, partly due to the stage of implementation, where several activities are

189 Itad (2018) Portfolio Review. 190 Itad (2018) Portfolio Review. Itad (2018) Kaderes Endline Qualitative Case Study. 191 Itad (2018) Joseph Initiative Endline Qualitative Case Study. 192 Springfield Centre (2014).

FINAL EVALUATION

Page | 47

yet to demonstrate consistent and enduring benefits to participants, and these are therefore not yet observed and spread to non-participants. However, there are signs that FTESA has already generated some systemic change, evidenced in cases of adoption and adaptation of new practices. For example, demonstration effects have encouraged some farmers to maintain changes in practices, but there is limited evidence that these effects have extended beyond project participants. Also, while market access remains a challenge, in some cases (e.g. EAGC, Farm Africa and Shalem) farmers understand the benefits of aggregating quality grain and marketing collectively, changing their mindset toward marketing grain, increasing their bargaining power (in both existing and new markets) and potential access to new and better markets.193

Behaviour change: smallholder farmers and farmer-based organisations

Across the portfolio, smallholder farmers showed a willingness to adopt new practices in terms of good agricultural practices and post-harvest handling. Several lead firms and off-takers reported that smallholder farmers were willing to swap the crops they were farming and to adopt new crops promoted by the grantee. For instance, off-taker Classic reported farmers changing from tobacco farming to soy. Lead firm Musoma’s interventions led to a farmer-based organisation switching to maize from tobacco. A significant number of farmers in Kenya supplying Shalem switched from tea to soy.194 Also, Kaderes persuaded its farmers to grow beans in addition to coffee.195 There are also examples of farmers making investments in a new crop like beans (e.g. Kilimo).196 However, off-takers Mount Meru, Seba and Joseph reported some farmers showing reluctance to grow certain crops.197 Seba reported that farmers were not confident about investing in soy after a large surplus in Zambia depressed soy prices and Joseph reported farmers’ discouragement regarding planting maize because of risks posed by climate change.198 Preceding EQs provide more examples of smallholder farmers’ adoption of, and investment in, improved inputs, good agricultural practices and post-harvest handling, as well as adoption of other practices, such as aggregation and collective marketing.

Some of the grantees report market actors taking on new roles. For example, Farm Africa enabled farmer-based organisations to take on new roles previously played by middlemen, by either collecting grains on behalf of a buyer or leasing their village aggregation centre to a trader, receiving a commission in both cases.199 There is also evidence of village aggregation centres making investments in their storage infrastructure (e.g. EAGC). However, there are also examples of village aggregation centres lacking resources to maintain their structures and equipment (e.g. EAGC and Sosoma).200

There is some evidence that farmers not participating in training nevertheless replicated the changes (demonstration effects) when they saw the results of new practices by others (e.g. Farm Africa and Kaderes).201

Behaviour change: lead firms, off-takers and other buyers

Among grantees, lead firms and off-takers reported offering smallholder farmers prices above market prices in exchange for supplies of improved-quality grain in greater quantities (see section 4.4, EQ5). Lead firms Raphael and Shalem expressed their commitment to continue sourcing their grains from smallholder farmers, with Raphael reporting that business is flourishing since adopting a smallholder-focused business model, something attributed to support from Kilimo: ‘we never thought working with smallholder farmers based on

193 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. 194 Itad (2018) Portfolio Review. 195 Itad (2018) Kaderes Endline Qualitative Case Study. 196 Itad (2018) Kilimo Trust Endline Qualitative Case Study. 197 Itad (2018) Mount Meru Endline Qualitative Case Study. Itad (2018) Joseph Initiative Endline Qualitative Case Study. 198 Itad (2018) Portfolio Review. 199 Itad (2018) Portfolio Review. Itad (2018) Farm Africa Thematic Study. 200 Itad (2018) Portfolio Review. Itad (2018) EAGC Endline Qualitative Case Study. 201 Itad (2018) Farm Africa Thematic Study. Itad (2018) Kaderes Endline Qualitative Case Study.

FINAL EVALUATION

Page | 48

pre-agreed supply contracts was profitable; ever since Kilimo Trust linked us to the farmers, we have never looked back and our beans business has been growing tremendously since then’.202

Changes in supporting functions and rules

Supporting functions in the staple food market include the inputs and financial markets. Sections 4.2 and 4.3 explore access to inputs and finance. Numerous grantees report smallholder farmers accessing inputs through farmer-based organisations, especially lead firms (e.g. Kaderes) and often at village aggregation centres (e.g. Joseph and Farm Africa). Successful consortia models improved smallholder farmers’ access to finance (e.g. WFP and Farm Africa). In the case of WFP, the farmers’ high loan repayment rate encouraged other financial service providers to ‘crowd in’ to provide finance in subsequent seasons. The means of distributing inputs and equipment is also an important supporting function, and one that is often lacking (for example, for Pee Pee, most village retailers do not have the capital required to buy and stock their PICS bags, despite demand).203

At a regional level, ACTESA and EAGC actively worked to change the rules governing the staple food trade. EAGC promoted the EAC grades and standards, helping achieve consistent and higher-quality grain in many cases (see section 4.2), while work led by FTESA and partners, including EAGC, on policy influencing is contributing to changes in key legislation that should improve the functioning of the staple foods market in Kenya and improve the results achieved through other FTESA-related interventions (see section 4.7). FTESA supported ACTESA and others to domesticate harmonised seed regulations across COMESA countries, leading to the domestication of harmonised regulations in all seven targeted countries and improving the predictability of rules and standards.204 Moreover, FTESA’s policy-influencing efforts through partners contributed to the lifting of export bans in Tanzania and Zambia, which affected the functioning of the wider market (including many FTESA-supported interventions).


Transparency and trust between smallholder farmers and buyers (including off-takers and lead firms) are the most important mechanisms to enable systemic change in localised market systems. At the level of regional markets, the supporting rules (namely regulations allowing for cross-border trade) are significant contextual factors influencing the extent of systemic change.

Mechanisms for behaviour change

Grantees frequently mention the importance of transparency and trust in encouraging market actors to change behaviour. For example, off-takers working with WFP make payments directly to the farmer group’s bank account and if the group owes the off-taker for inputs/services provided up front, the off-taker can first deduct these costs before making payment, assuring the off-taker that the group will settle its debts.205 Similarly, G-Soko’s (EAGC) payment and settlement account provided confidence to both buyers and sellers by showing that the commodity exists and funds are available in the account.206 On receipt and acceptance of a delivery, the seller received the funds, reducing payment default. This function was particularly useful when dealing with new buyers and sellers. Off-takers Mount Meru and Joseph report that the introduction of digital weighing scales has fostered smallholder farmer trust in them as a buyer that will not try to cheat them. In these cases, digital platforms make transactions more transparent.207 Shalem reported more women registering to grow soy and that it has attracted youth to farming maize and beans.208

202 Itad (2018) Portfolio Review. Itad (2018) EAGC Endline Qualitative Case Study. 203 Itad (2018) Portfolio Review. Itad (2018) EAGC Endline Qualitative Case Study. 204 Itad (2018) Portfolio Review 205 Itad (2018) Portfolio Review. 206 Itad (2018) EAGC Endline Qualitative Case Study. 207 Itad (2018) Portfolio Review. Itad (2018) Joseph Endline Qualitative Case Study. Itad (2018) Mount Meru Endline Qualitative Case Study. 208 Itad (2018) Portfolio Review.

FINAL EVALUATION

Page | 49

Barriers to systemic change

The most predominant barrier to systemic change was the absence of supporting rules (for example, export and import bans). Many grantees (e.g. EAGC, Farm Africa, Mount Meru, Musoma, Raphael and WFP) all reported problems stemming from the export bans in Tanzania and Zambia, including limiting the use of G-Soko for cross-border trade and access to inputs (e.g. soya seeds for Mount Meru).209 Government intervention in terms of purchasing grain also posed challenges for FTESA-funded interventions and the wider market (e.g. EAGC and WFP).210 Other issues included slow registration of new seed varieties. As discussed in section 4.2, problems with access to capital were an important barrier to behaviour change for smallholder farmers, farmer-based organisations, village aggregation centres and agro-dealers (e.g. Kaderes).211

G-Soko provides an example where lack of transparency and trust in the system contributed to poor performance. Systemic change requires that the market adopts the intervention and sees value in continuing with these changes without direct support, crowding in other market actors who offer supplementary services (e.g. finance, WRS, etc.) that are viable because of the grant, with users of the platform (e.g. farmers, traders, etc.) changing their behaviour and spreading new behaviours to others. Key to success is the mechanism for spreading behaviour across networks of market actors. Buyers and farmers require constant demonstrations before most of them become willing to adopt a new model. To endure, the G-Soko model needs to demonstrate consistent performance for the spread of behaviour to larger networks of market actors. However, this is yet to materialise. Given that the platform is not fully functioning, low utilisation rates and volumes traded, as well as low appetite to use the platform and move produce from village aggregation centres to certified warehouses, mean positive systemic effects are unlikely to materialise in future without significant improvements generating positive demonstration effects that spread across networks of actors where users and other actors can see constant benefits of participation.

For whom

As already noted in section 4.4.2, according to stakeholder (grantee) survey responses, smallholder farmers are the most common beneficiaries of grantee FTESA-funded projects, with 90% of respondents choosing this option. However, smallholder farmers are far from the only FTESA beneficiary group: 70% of respondents also chose aggregators as benefiting. In addition, 50% of respondents reported their own firm benefited – one respondent said ‘the program design was excellent as it worked on building private sector capacity’ – and a further 50% chose input suppliers as among the main beneficiaries.212

209 Itad (2018) Portfolio Review. Itad (2018) Mount Meru Endline Qualitative Case Study. Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Farm Africa Thematic Study. 210 Itad (2018) Portfolio Review. Itad (2018) EAGC Endline Qualitative Case Study. 211 Itad (2018) Portfolio Review. Itad (2018) Kaderes Endline Qualitative Case Study. 212 Itad (2018) Stakeholder Survey.

FINAL EVALUATION

Page | 50

Box 5: FTESA grantee capacity

Grantee responses to the 2018 stakeholder survey shed some light on how grantees believe their capacity has improved:

• The two most common access-related results delivered by grantees with help from the FTESA grant were increased access to buyers/sellers and increased access to market information, each chosen by six of 10 (60%) respondents.

• The two most common use-related results delivered by grantees with help from the FTESA grant were increased use of training on good agricultural practices and increased use of post-harvest handling advice/training (e.g. on grades and standards), each chosen by seven of nine (77.8%) respondents.

• The most common result for smallholder farmers delivered by FTESA grantees was increased quality of farmer produce, chosen by eight of 10 (80%) respondents.

• When asked how they worked with other FTESA grantees to achieve their objectives, seven of eight (87.5%) respondents chose collaborated on finding markets (buyers/sellers).

• Eight of eight (100%) respondents indicated that they will continue to engage with other FTESA actors now that FTESA has ended.

• One respondent wrote about how the grants brought about sustainable changes: 'The grant enabled us to build a sustainable supply chain to build upon and introduced us to working with partners from the governmental and non-governmental sectors. Furthermore, the results achieved were pivotal in convincing the management of the importance of allocating budgets for market development’.

Sustainability

Here we further explore concepts of systemic change (e.g. adoptions, innovation, copying, crowding, etc.), drawing on the evidence above and across the EQs. There are examples of sustainable relationships (and respective changes in behaviour) due to ‘adoption’ and ‘adaptation’ and in several cases market actors have adopted new practices. However, in many cases, grantees are providng services, effectively building themselves into the system without a clear exit strategy.

Maintaining and spreading new knowledge and skills: Farm Africa and other grantees aimed to change long-term behaviour through training farmers. For Farm Africa, evidence shows that up to 80% of trained farmers adopted new practices due to training. Since the financial benefits of these changed behaviours materialised, this suggests that farmers’ knowledge and skills learned will last, along with their changed behaviour. Several grantees (ENAS, Kaderes, Kilimo, Shalem and Yak) also reported farmers having internalised training and advice about good agricultural practices, evidenced by their willingness to pay for inputs.213

However, as discussed earlier, the one-off nature of the training limits sustainability. First, we found little evidence that farmers have the incentive, or means, to disseminate information beyond their immediate group. We also found little evidence of approaches by grantees facilitating wider adoption by farmers not directly targeted by the intervention. Similarly, there is typically limited information on how the grantees expect other market actors (e.g. traders and service providers) not directly involved in the intervention to change their behaviour. In this sense, many of the grantees failed to take adequate account of the incentives of actors within a system. Second, there is often limited capacity for the adaptation of the learning to new external developments (for example, a new technology). From a systemic perspective, training is not a one-off activity and should be a function of the market system performed by permanent market actors with an incentive to scale the training and update it as new technologies and practices become available. Relevant actors might include, for example, buyers offering private extension services or agro-input suppliers embedding training in their service offering.214

Adopting new or adapted business models: In several cases, it appears grantees’ adopted behaviour change will continue. For example, Raphael and Shalem expressed commitment to changes in their business models

213 Itad (2018) Farm Africa Thematic Study. Itad (2018) Kilimo Endline Qualitative Case Study. Itad (2018) Kaderes Endline Qualitative Case Study. Itad (2018) Portfolio Review. 214 Itad (2018) Farm Africa Thematic Study.

FINAL EVALUATION

Page | 51

as the new approaches delivered benefits greater than previous ways of doing business.215 In contrast, Joseph’s new business model of establishing and operating Joseph centres led to them operating at a loss, with the result that they stopped operating the centres.216

Crowding in other market players: Kilimo’s approach facilitated the involvement of several market actors to address market constraints at different levels of the system, with some actors joining without a binding MoU. Kilimo indicated that these market actors had bought into the approach, contributing to systemic change in a sustainable fashion.217 However, as discussed earlier, like other grants (e.g. EAGC, Joseph and Mount Meru), Kilimo had limited success crowding in financial service providers, given the perceived continued risk of lending to smallholder farmers, thus limiting the sustainability of the interventions.218

Adaptation or innovation by market actors: There are several examples of adaptation or innovation by market actors. When faced with the export ban in Tanzania, Musoma changed to selling flour to off-takers rather than grain. Lead firm Shalem innovated by investing in a new milling plant, creating a new revenue stream for the company and transforming the firm from a trader to a processor. Off-taker Joseph expanded into sales of a new crop (soy bean seed), the piloting of which was very successful.219 Village aggregation centres in the Farm Africa consortium are gradually expanding and adapting to become service hubs, attracting providers of seeds, other inputs and packaging.220


Main findings and conclusions [strength of evidence – low]: Given the limited scale of most of the interventions, including in terms of their geographical reach (with several projects having a limited footprint across the region), the programme has not generated the substantial volumes required to pass through the market to lead to price smoothing at a regional level, partly due to the underperformance of G-Soko. However, while there is no systematic reporting on the benefits to the end consumer, there is anecdotal evidence that grantees are producing improved quality and value-added products.

Price smoothing and availability: The hypothesis underlying the evaluation question is that FTESA interventions will contribute to more stable prices and food availability for consumers across the region, with grain moving from surplus to deficit areas and stored between harvest and shortage, thus smoothing prices. However, this requires substantial quantities to pass through the regional market. At present, FTESA grantees are not operating at sufficient scale to lead to region-wide price smoothing. Moreover, the programme intended G-Soko to play a large role in the grain trade across the region, bringing in suppliers and buyers above-and-beyond FTESA grantees, making it easier and less costly to trade across regional markets. Both the quality and price-smoothing benefits required that G-Soko efficiently handle enough throughput to affect prices. However, few trades took place via the platform. As a result, it did not provide the price discovery mechanism as intended. Its potential benefit to consumers through price smoothing is unlikely to occur given challenges in both its design and implementation. Moreover, even with substantial throughput, it would be difficult to attribute price smoothing to G-Soko as there are many other factors in play that impact trade volumes and consumer prices.221

Quality and health benefits: Grantees did not report systematically on consumer benefits. Few grantees refer to benefits of more stable prices, either actual or potential. The most commonly cited benefit for consumers is improved quality and health benefits, including: improved grain standards and reduced toxins (EAGC and Shalem); more value-added, processed foods available (Musoma and Yak); increased availability of fortified

215 Itad (2018) Portfolio Review. 216 Itad (2018) Joseph Initiative Endline Qualitative Case Study. 217 Itad (2018) Kilimo Trust Endline Qualitative Case Study. 218 Itad (2018) Kilimo Trust Endline Qualitative Case Study. 219 Itad (2018) Portfolio Review. 220 Itad (2018) Portfolio Review. 221 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Case Synthesis.

FINAL EVALUATION

Page | 52

more nutritious products enriched with minerals and vitamins (Kilimo and Shalem); and improved food safety by using PICS bags rather than chemicals when storing grain (Pee Pee).222 Mount Meru reported its intention to fortify soybean oil and powder but it has not yet aggregated enough soy to begin production.

Improved quality of grain is an important and credible benefit, particularly given the prevalence of contaminants, such as aflatoxin, in many grains (particularly maize and groundnuts), which cause long-term organ damage in humans (and occasional death, in cases of acute exposure). Also, given changing demand toward processed, value-added products, especially in urban centres, and a growing market for quality, the demand for such produce is likely to increase.223


Main findings and conclusions [strength of evidence – medium]: Policy and regulatory issues affecting the food market require constant attention and engagement. Policy and regulatory barriers to improving market functioning in staple foods are unlikely to reduce significantly as some actors in the system have the incentive to perpetuate and/or create new barriers, often profiting from such inefficiencies. The implications are that programmes may need to accept that continual engagement is inevitable in these areas (e.g. policy influencing). Across the cases, the FTESA experience shows that no one activity is more successful than another in reducing such barriers. A multifaceted approach is most appropriate, working through local actors embedded in the local context and system who can build coalitions and search out likeminded actors and who know who to target and how – and who have strong incentives to reduce such barriers (e.g. membership organisations representing farmers) to ensure accountability, effectiveness and sustainability of efforts. Furthermore, this is applicable to other areas of the programme, where developing strategic partnerships with those strongly motivated to change the market system may lead to long-term sustainable impact.

This section focuses on four cases of policy influencing supported by FTESA relating to the role of the Strategic Food Reserve (SFR) and Food Reserve Agency (FRA) in the food staples market in Kenya and Zambia, respectively. It also covers policy and regulatory restrictions affecting the development of the soya value chain in Zambia and the potential impact of the Warehouse Receipt Bill in Kenya on the market for grain.

There is strong evidence that the policy issues prioritised by FTESA are very relevant, since all have the potential to interfere with, and distort, market operations and undermine efforts to develop structured markets and foster structured trade in Kenya and Zambia.

Across the cases, FTESA worked largely through partners and existing initiatives to influence decision-makers and policy processes (central and local government, and parliament), capitalising on the knowledge and connections of established and known institutions, all while taking a non-activist approach. In addition, FTESA and partners worked directly with government on the SFR in Kenya.

FTESA used several types of activities to support its policy-influencing efforts. These involved:

• FTESA staff time, external advisers (africapractice and Policy Advisory Forum members), grantees’ staff time (namely the EAGC for the cases presented here) and consultants.

• Studies and meetings, including consultations with key actors and public–private dialogues.

Warehouse Receipt Bill in Kenya224

With FTESA support, EAGC and members engaged with government through dialogues, including public–private dialogues, commenting and inputting into the draft bill, working with africapractice to gain access to relevant committees and track the progress of the bill through the house. The public–private dialogue was an

222 Itad (2018) EAGC Endline Qualitative Case Study. Itad (2018) Kilimo Trust Endline Qualitative Case Study. Itad (2018) Portfolio Review. 223 Itad (2016) FTESA Mid-term Evaluation; Itad (2018) EAGC Endline Qualitative Case Study. 224 Itad (2018) Kenya Policy Dialogue and Influencing Study.

FINAL EVALUATION

Page | 53

effective way to engage with market actors and with government. The revised draft bill incorporated most of the amendments proposed by EAGC and its members, reducing the overbearing role of government and aligning it more closely with market principles, thus delivering changes that reflected the interests of key market actors.

The most important factor in gaining traction with policy-makers was working through established partners (i.e. EAGC) already embedded in the local social, political and economic context who have existing engagement channels and clout/influence with various decision-makers and their own members. Also, EAGC’s involvement in the drafting of the bill since 2010 provided it with historical information on the evolution of the bill and the key changes that would make it implementable by private sector actors.

However, at programme close, Parliament had not yet ratified the bill, due to several delays at different levels of the political system (Parliament, National Assembly and Senate). While the programme was not able to control such delays, africapractice monitored the progress of the bill in the political system and identified delays, while at critical points EAGC followed up on these (‘they knew who to call’), helping ‘move the bill upstairs’. Moreover, new challenges in early 2018 arose, threatening finalisation, with the Ministry of Trade pushing to extend the scope of the bill to include non-agricultural commodities (through the Multi-Commodity Bill), requiring a significant rewrite and consultations. Despite the Ministry of Agriculture rejecting this proposal, it remains a threat to the finalisation of the bill.

Sustainability: Since programme close, EAGC and others continue to track the progress of the bill and push for its finalisation. The International Finance Corporation, which also provided advice on the legislation, is currently designing a new programme, the success of which will depend on the passing of the bill. Hence, the new programme is also likely to find ways of influencing the bill, directly or indirectly through others. It is in the interest of other donors, such as AGRA, which is also supporting WRS, to use its influence to ensure the passing of the bill.

SFR in Kenya225

FTESA funded a consultant to develop the strategic plan with the Ministry of Agriculture for the SFR Trust Fund, as well as associated agency agreements, directly influencing its structure and operations and allowing FTESA and key partners to contribute to its content through rounds of commenting and revisions. This resulted in a plan that is more aligned to market principles and fostering structured markets and trade. The SFR Board briefed the Cabinet Secretary in the Ministry of Agriculture and the SFR Board approved the plan in April 2018 for launch in May 2018.

Critical enabling factors were EAGC’s direct role as a member of the Oversight Board of the SFR Trust Fund, which has oversight of other agencies, and EAGC connecting FTESA and the consultant to key officials in the Ministry of Agriculture, helping create and develop a strong working relationship and access to other relevant actors as well as forums. Also, FTESA’s involvement meant that more analysis and consultation took place than would have been possible without the funding, strengthening the quality of the plan, internal reflection and buy-in from key actors.

As with the Warehouse Receipt Bill, the launch of the plan faced threats outside of FTESA’s control. In July 2018, a presidential directive instructed that the institutional home of the SFR be moved from the Ministry of Agriculture to the Ministry of Devolution, which put the launch on hold and raised the possibility of substantial changes to the plan. However, a reversal of the decision in mid-September meant it moved back to the Ministry of Agriculture, and plans are now in place to prepare for the launch of the plan and associated agency agreements.

Sustainability: Like with the warehouse bill, EAGC’s continued role helps ensure the sustainability of efforts to deliver the end result (i.e. a reformed structure for the SFR). However, the highly politicised nature of food

225 Itad (2018) Kenya Policy Dialogue and Influencing Study.

FINAL EVALUATION

Page | 54

security, as well as the presence of powerful vested interests with incentives to undermine market principles, is a real and present threat. Wider experience shows that, if the incentive is there, powerful actors will often find ways to get around new structures or stop them from materialising in the first place.

Soya policy and regulatory restrictions in Zambia226

FTESA was instrumental in establishing a coalition to influence policy in the soya sector, the SOPAG, including the ‘big guns’ in the sector such as IAPRI (the secretariat of SOPAG). In addition, IAPRI conducted several FTESA-funded studies for use in policy influencing and IAPRI was also a member of FTESA’s Policy Advisory Forum.

In a context where government is often predisposed to make ad hoc policy changes (with respect to food security) based on political expediency, there is reasonably strong evidence that FTESA-funded studies combined with other influencing efforts (e.g. dialogues, public–private dialogues, etc.) were very useful in providing stakeholders with information on the impact of such policy decisions, helping contribute to the reversal of the export ban on soya in May 2017. As with Kenya, there is strong evidence that policy influencing is most effective when led by those embedded in the local context, who have existing engagement channels and clout/influence with various decision-makers, in combination with providing (and articulating effectively) credible evidence of the problem, why it is an issue and what to do.

Sustainability: While FTESA’s intention was to ensure sustainability through the establishment of SOPAG, the group is no longer active. This is largely attributed to the lack of FTESA staff who previously proactively engaged members around common interests, as well as a lack of financial support for attendance at meetings. Unfortunately, working through local stakeholders and establishing a coalition around common interests was not enough to create long-lasting channels for policy influencing in this case. However, IAPRI has the potential to coalesce other SOPAG members into a continued engagement and could rebuild the momentum created.

FRA in Zambia227

With respect to the FRA, FTESA became involved in policy-influencing efforts toward the end of the programme (2017) as a result of concerns about the direction of intended reforms, which had the potential to undermine the role of the private sector. FTESA’s experience on similar issues in other countries helped inform policy-influencing efforts in Zambia.

During the last quarter of 2017, FTESA – working with IAPRI, CUTS and SOPAG – attempted to influence the process of FRA review (e.g. FTESA supporting both IAPRI and CUTS to conduct and disseminate studies reviewing the proposed amendments of the FRA Act). SOPAG members attended a national consultative workshop in December 2017 where CUTS and IAPRI staff presented their views. Strong rebuttals by SOPAG members (among others) of the intended reforms prevented the government going ahead with their plans, which is arguably a successful outcome of this policy-influencing effort. However, the Ministry of Agriculture responded by setting up a technical working group tasked with reviewing the proposed reforms, including private sector, government and research institutions, but excluding all SOPAG members, IAPRI, CUTS and FTESA. Interviewees attributed their exclusion to the government’s dislike of their opposition to the amendments, which delayed the passing of the reforms. In response, FTESA, CUTS and SOPAG members released a joint press statement expressing concerns about the revised act, also producing a study by CUTS demonstrating the negative effects of the proposed amendments. However, at the time of field work in September 2018, the status of the revised act remains unclear.

The exclusion of FTESA and partners demonstrates their potential influence on the process. However, their exclusion has also prevented them from realising their influence from within. On reflection, FTESA’s strategy

226 Itad (2018) Zambia Policy Dialogue and Influencing Study. 227 Itad (2018) Zambia Policy Dialogue and Influencing Study.

FINAL EVALUATION

Page | 55

was possibly too assertive, and arguably activist, and a better approach would have been to create and foster allegiances with those within the political system (i.e. the Ministry of Agriculture), bringing them more closely within their sphere of influence, as had been done in Kenya.

Sustainability: Given that none of FTESA’s influencing partners are involved in the working group it is difficult to assess whether the revised act will take on board the concerns aired. Moreover, SOPAG’s inactivity since the closure of the programme significantly reduces the sustainability of efforts set in motion by FTESA and partners.


Main findings and conclusions [strength of evidence – medium]: Based on the VfM metrics analysed and benchmarking conducted, there are several areas where the programme performed well in comparison to other similar programmes, as well as improvements in some of the metrics since 2016. VfM metrics on economy improved since the mid-term evaluation in 2016 and FTESA performs better than comparable programmes such as WAFM. The smallholder engagement cost (efficiency) increased between 2016 and 2018, although it remains within the range of other comparable programmes. Portfolio leverage rates (efficiency) improved and are higher than similar programmes. The volume of sales contributed per beneficiary farmer (effectiveness) increased between 2016 and 2018, most likely due to progress in implementation, with results transpiring further along the value chain, as well as the addition of new grants in 2016 that generated substantial sales volumes (e.g. Farm Africa and WFP). The cost of female outreach (equity) was significantly higher than overall reach, suggesting that achieving equity is a cost driver, but it too is within the range of other comparable programmes.

The PMU was slow to develop VfM indicators; it only started collecting the relevant data in 2016 and did not set any targets. There was no evidence to suggest that the PMU was using VfM data to feed into ongoing analysis and learning by programme teams and at programme close the PMU provided limited assessment of the VfM data.

This section draws on the EMU’s VfM assessment shared in September 2018, which builds on an earlier VfM assessment conducted in December 2017 and uses data from the self-assessment by the PMU in April 2018.228 The EMU helped the PMU develop six VfM metrics midway (2016) through programme implementation. Since 2016, the PMU has reported data against these metrics, although there were no associated targets. We therefore focus on comparisons within the portfolio of grants and with similar programmes, as well as change over time (2016 and 2018) where reported. This VfM assessment uses this data to explore the VfM of the overall portfolio and at the grant level, as well as groupings by sub-sectors/themes and funding mechanism. We also compare FTESA VfM with other similar programmes (e.g. WAFM and other DFID-funded market development programmes).

The VfM metrics are suitable for the assessment but we were unable to find comparable indicators on effectiveness across DFID programmes, which suggests that in future there needs to be more careful selection of appropriate indicators that allow for comparison (e.g. jobs created, etc.). Moreover, future programmes need to assign targets to indicators to track progress and provide incentives to achieve these, as was recommended at mid-term.

The table below provides a summary of the VfM of the overall portfolio across different indicators at the end of the programme (2018) compared with 2016 (mid-term).

228 Itad (2018) Value for Money Assessment.

FINAL EVALUATION

Page | 56

Table 2: VfM by metric

Economy

Economy assesses whether the programme bought inputs in the right quantities and at the right price. FTESA tracked two metrics: the fund management cost ratio, using management costs and total funds disbursed, and the administrative cost ratio, using administrative costs and total funds disbursed. According to the PMU, fund management costs include salaries and some office overheads, while administrative costs include rent, maintenance and utilities.

Economy improved between 2016 and 2018. The fund management cost ratio shows a downward trend, decreasing from 29% in 2016 to 21.5% in 2018. The administrative cost ratio also shows a downward trend from 32% in 2016 to 23.6% in 2018. This is most likely due to the increase in funds disbursed (from £16 million in 2017 to £27.5 million in 2018). FTESA results indicate better economy performance when compared to WAFM (21.5% compared to 22% fund management cost ratio; and 23.6% compared to 35% administrative cost ratio), also most likely due to levels of disbursement given the respective stages of implementation.

Efficiency

Efficiency assesses performance in terms of converting inputs into outputs. The smallholder engagement rate (ratio of funds disbursed and the number of farmers benefiting from national and cross-border value chains) increased between 2016 and 2018 (from £48 to £55), again most likely due to higher disbursement rates. The smallholder engagement rate is lower for the CF compared to the DF. The most expensive grants in terms of outreach are ENAS, Mount Meru, Kilimo, EAGC, Seba and Musoma (in declining order). In terms of different models, Farm Africa’s NGO consortium model and WFP’s forward delivery contract appear to be more efficient than the off-taker and lead firm models. The lead firm model performs better than the off-taker model. Several other market development programmes studied for comparison also use this metric and the cost rate lies within the range of other comparable programmes.229

The portfolio-wide leverage ratio (leveraging of additional investment) improved between 2016 and 2017 (from 1.52 to 2.73) and rates are higher than similar programmes. These represent high ratios for this metric, indicating good performance, especially when compared to other programmes (including WAFM).230

The cost per MT capacity of certified warehouses constructed or refurbished is another efficiency metric and includes the cost of materials, trainings and overheads in the numerator and volume of capacity created or improved in the denominator. It shows that Raphael (£34.34) appears the most cost-efficient way to create

229 Smallholder engagement rate: £17 WAFM, £26 GEMS4 Nigeria, £52 MOST Malawi, £55 FTESA, £61 Samarth Nepal and £211 MADE Ghana. 230 Portfolio-wide leverage: FTESA 2.73, MADE Ghana 2.34 and WAFM 1.44.

2016 2018

Economy

Fund management cost ratio 29% 21.5%

Administrative cost ratio 32% 23.6%

Efficiency

Smallholder engagement rate £47.74 £55.34

Portfolio-wide leveraging ratio 1.52 2.73

Effectiveness

Volume of staple food sales contributed to per farmer reached 394kg 459kg

Equity

Cost of female outreach n/a £135

Female outreach (%) 43% 41%

FINAL EVALUATION

Page | 57

more capacity for grain storage, followed by Kaderes (£49.30), with Yak (£80.89) the most expensive. Since other FTESA grants or similar programmes do not use this indicator, we cannot do a comparison. Moreover, since it only covers three grants, we cannot use this to assess FTESA’s overall VfM.

Effectiveness

Effectiveness assesses performance in terms of converting outputs into outcomes. The volume of sales contributed per beneficiary farmer increased between 2016 and 2018 (from 394 kilos to 459 kilos). However, there is no comparable data from similar programmes. Musoma is most effective in contributing to sales per farmer – it is a relatively low-cost grant with strong performance. Farm Africa and Kilimo both recorded high volumes per farmer reached (1,494 and 785 kilos, respectively), but with relatively high costs. Raphael contributed to high volumes of food sales per farmer reached (1,251 kilos), and at a much lower cost than Farm Africa and Kilimo. Grants to Sosoma, Seba, Yak, Afritec, Joseph, EAGC and Mount Meru have not translated into large volumes of staple food sales per farmer. In the case of EAGC, Joseph, Mount Meru and Seba, the costs are considerably higher than others, yet sales are very low. Looking at cost per kilo sold, Musoma, Farm Africa, Shalem, WFP, Raphael, Kaderes and Kilimo all perform well (between £0.02 and £0.24 per kilo sold). The lead firm model performs better than the off-taker model, Farm Africa and WFP, and the DF grants are more than three times as effective as the CF grants in terms of volumes sold per farmer.

Equity

Equity assesses the degree to which results are equitably distributed. The PMU used the percentage of women among smallholders engaged as the equity metric. We also calculated the cost of female outreach. Female outreach (%) shows a slight decrease between 2016 and 2018 (from 43% to 41%) and a significantly higher cost (£135) than the overall outreach cost (£55). This suggests that achieving equity is a cost driver, but it is within the range of other comparable programmes.

• Female outreach (cost): Esoko (£22), Shalem (£25), Sosoma (£26), Yak (£37), WFP (£63), Classic (£64) and Afritec (£117) all performed better than the average (£135).

• Female outreach (%): Shalem (72%), Sosoma (57%), Afritec (54%), Raphael (49%), Kilimo (48%), EAGC (47%), WFP (46%), Yak (46%), Seba (45%) and Classic (45%) performed better than the average (41%).

Grantees such as Shalem, Sosoma and Afritec performed strongly in engaging women. Shalem is the top performer in terms of percentage outreach, with 71% female outreach at a relatively low cost of outreach (£25). ENAS, Mount Meru and Musoma demonstrated a low level of performance in terms of both female outreach (low) and costs (high).

In terms of cost of female outreach, Farm Africa and WFP perform better than the off-taker and lead firm models, while the lead firm model performs better than the off-taker model. The average cost of female outreach is very similar for the CF and DF groupings. We compared percentage and cost of outreach with other programmes and FTESA performed reasonably well compared to similar programmes, and better than its sister programme WAFM.231

Overall assessment

Economy improved between 2016 and 2018 and FTESA’s VfM metrics indicate better economy compared to WAFM. While the smallholder engagement cost (efficiency) increased between 2016 and 2018, it lies within the range of other comparable programmes. Portfolio leverage rates (efficiency) improved and are higher than similar programmes. The volume of sales contributed per beneficiary farmer (effectiveness) increased between 2016 and 2018, although there is no comparable data from similar programmes. The percentage of

231 Cost of female outreach: £80 GEMS4 Nigeria, £115 Samarth Nepal, £135 FTESA, £140 WAFM and £435 GEMS1 Nigeria. Percentage of female outreach: 52% Samarth Nepal, 41% FTESA, 35% GEMS4 Nigeria and 20% WAFM.

FINAL EVALUATION

Page | 58

females reached (equity) fell, and the cost of female outreach (equity) was significantly higher than overall reach. This again suggests that achieving equity is a cost driver, but it is within the range of other comparable programmes.

In terms of smallholder engagement rates, Farm Africa and WFP are more efficient than the off-taker and lead firm models, while the lead firm model performs better than the off-taker model. The lead firm model performs better than all other models with respect to volumes sold per farmer. Farm Africa and WFP have lower costs of female outreach than the off-taker and lead firm models, and the lead firm model performs better than the off-taker model. Based on the metrics analysed and benchmarking conducted, there are areas (economy and leverage) where FTESA performed well in comparison to other similar programmes.

FINAL EVALUATION

Page | 59

Main findings and conclusions

To what extent is FTESA a collection of individual interventions or a coherent portfolio (EQ1)?

The FTESA portfolio of grants offered opportunities for projects to benefit from complementarities and synergies between them. There is some evidence of interlinkages and complementarities materialising between FTESA grantees that have generated results beyond those that would have been achievable through an individual grant, with some grantees having been able to tap into support provided by others to increase the impact of their interventions (including combining support at different points along the value chain). Partnerships have enabled grantees to capitalise on each other’s services. However, there are concerns regarding the sustainability of some of these interactions, particularly those reliant on DF interventions rather than commercial entities.

The most frequently cited connection is between grantees and EAGC. The PMU focused much of its effort on building interlinkages and complementarities across the portfolio around the EAGC grant, especially warehouse certification and the G-Soko platform, with the intention that several different FTESA-funded grantees would use the G-Soko platform and create a group of early adopters. However, the failure of G-Soko to take off successfully and demonstrate consistent results through early adopters significantly reduced the opportunity for synergies and seriously hampered FTESA’s ability to create a portfolio that delivered more than the sum of its parts through grantees using the platform and accessing larger markets, with others copying, which limited sustainability and curtailed the impact of the FTESA programme in the wider market.

Linkages between FTESA grantees were in part facilitated by the PMU but were also due to grantees’ knowledge of other grantees, existing relationships and/or searching out synergies themselves. The PMU’s model restricted the programme’s ability to design strong interlinkages and build a coherent, comprehensive portfolio. In several cases, the PMU attempted to build linkages into the design of projects after awarding the grants. With relatively few rounds and lack of real-time monitoring and learning built into the programme design and decision-making, there were limited opportunities to quickly learn from previous rounds and tailor future rounds to attempt to generate interlinkages, building on existing projects to improve the performance of the overall portfolio. Moreover, the PMU itself was not set up with the level of technical assistance required to provide substantive design inputs.


Overall, evidence of FTESA-led improvements in trade support systems shows mixed success, partly due to delays in implementation including where some interventions are reliant on the completion of other activities (e.g. a WRS requires the warehouse to be in place to the required standards), as well as the short timeframes for results to transpire. There was good progress and achievements on activities that set the foundations for improvements in trade support systems. Despite differences in context and implementation models, several of the grants helped farmers reduce post-harvest losses, increase volumes and quality of produce stored and aggregated, as well as improving farmers’ position in the market. Nonetheless, the results fell short of expectations. Training on post-harvest handling, in combination with the incentive of higher prices for aggregated better-quality produce, helped farmers understand the value of improving post-harvest handling and aggregating produce, as well as motivated farmers to improve post-harvest handling and store and aggregate their produce with others, reaching required standards, and marketing collectively to access better markets, given the right incentives and demonstrated benefits – namely, better prices and market access.

However, in many cases the aggregation volumes fell short of the volumes expected. Barriers to storage and aggregation have curtailed the achievement of expected results, including construction delays, smallholder farmers’ preference for immediate payment, issues of trust, transport distances and costs, and lack of better markets. The case study evidence shows that some of the grants were unable to deliver key output milestones within the relatively short timeframes. Also, access to finance remains a major challenge, with considerable institutional barriers existing in the financial markets, where in many cases banks remain risk averse in regard to lending to smallholder farmers, with the existence of collateral alone not sufficient to engender confidence.

FINAL EVALUATION

Page | 60

To what extent have improved availability and use of inputs and application of good agricultural practices increased production and trade (EQ4)?

Several cases present good evidence of improvements in the availability and use of inputs and farmers simultaneously applying good agricultural practices due to FTESA-funded activities, although the numbers reached are low, with supplies of inputs and numbers trained lower than expected. Most progress is in training to support the application of good agricultural practices. Where farmers have applied good agricultural practices and used improved inputs, productivity and quality have improved. There is strong evidence across the cases that farmers are willing to adopt new/improved inputs and practices where the benefits are clear (i.e. there are demonstration effects and proof of concept) and are in line with a farmer’s own risk appetite.

The use of known and respected institutions (grantees, research institutions, input suppliers and local SACCOs) to mobilise farmers enhanced the credibility of the intervention and generated trust, especially in contexts where farmers were risk averse and had lower levels of education, leading to greater participation rates and adoption of new and/or improved inputs and practices. The use of the lead farmer approach worked well when farmers considered the lead farmer as experienced and successful, increasing their credibility, with farmers more likely to apply lessons from training provided by such farmers.

In some cases, gaps in service provision (e.g. lack of finance) reduced the uptake of new and/or improved inputs. Also, despite efforts to improve access to quality inputs, delays in accessing inputs at the required quantity and quality limited the benefits. Factors outside of the direct control of the programme, such as fake seeds, government policy leading to delays in accessing improved seed and weather-related issues, reduced the benefits of applying better practices. There is limited evidence across the grants that these interventions have led to higher-level results (prices, sales, incomes), partly due to the stage of implementation and lack of success in finding markets (see EQ5).


Increased smallholder farmer participation in structured regional markets was a central focus for FTESA and grantees. Most of the grantees worked with smallholder farmers with existing or potential tradeable surpluses, with women accounting for approximately 40% of participants. Several of the grants helped smallholder farmers improve yields, production and quality, as well as store and aggregate greater volumes, making them ‘market-ready’ and able to sell to a wider range of buyers. Unfortunately, lack of data and information on specific groups limited analysis on those different groups.

The evidence strongly suggests that lack of storage, aggregation and collective marketing (e.g. selling as a group) represent a fundamental barrier to improving farmers’ position in the market and their ability to command higher prices. Several cases show that the interventions helped farmers understand that storing produce and deferring sales can lead to higher prices and aggregating good-quality produce and marketing collectively can lead to access to better markets as buyers prefer the convenience of purchasing aggregated produce that reaches required standards. The cases provide evidence of changes in the mindset of participating farmers toward aggregation and selling as a group, and examples of farmers’ bargaining power and ability to negotiate better prices increasing when farmers increased and improved aggregation and sold as a group, creating competition among buyers for their produce. In some cases, the programme facilitated connections between farmer groups and buyers, which led to deals, increased sales and better prices.


FINAL EVALUATION

Page | 61

To what extent is FTESA likely to improve the functioning of national and regional staple food markets and generate systemic change (EQ2)?

The grants show limited evidence of systemic change and only early signs of the potential for spreading new behaviours to others, largely due to the stage of implementation: several activities are yet to demonstrate consistent and enduring benefits to participants and therefore such results have not yet been observed and had chance to spread to non-participants. While there is evidence that farmers have adopted new ways of doing business and accessed new markets, in many cases there was over-reliance on support from the grantee for inputs, services and market access, limiting sustainability and impact in the future. Some grantees did not provide support for interventions for long enough to deliver the ‘critical mass’ of consistent success required to build buy-in for the intervention and encourage others to crowd in. However, there is anecdotal evidence that FTESA has generated systemic change in some cases, as indicated by examples of behaviour change in smallholder farmers (e.g. adoption of new methods and some copying by others) and buyers offering better prices. Again, enablers of behaviour change include transparency and trust between market actors. The most frequently cited barriers were the absence of supporting rules and limited capital.


Given the limited scale of most of the interventions, including in terms of their geographical reach (with several projects having a limited footprint across the region), the programme has not generated the substantial volumes required to pass through the market to lead to price smoothing at a regional level, partly due to the underperformance of G-Soko. While there is no systematic reporting on the benefits to the end consumer, there is, however, anecdotal evidence that grantees are producing improved quality and value-added products.


Except for one case, the intended results of policy-influencing efforts are yet to materialise partly due to long and drawn-out political and bureaucratic processes outside the control of the programme. However, even where they did materialise, there are significant risks of policy reversal.

FTESA’s support to policy-influencing efforts has helped remove export bans and set processes in motion. There is strong evidence that taking an evidence-based approach and working through partners were critical factors for successful influencing of policies and regulations in both Kenyan and Zambian policy case studies. Across the cases, no one activity type was better than another. Rather, each policy issue required activities that combine and interact, building on each other. Examples include producing technical analyses that fed into position papers by key partners, shared with policy-makers, and presented and discussed in public–private dialogues.

In Kenya, through its persistent engagement with constituents who are seeking change and those who the programme sought to influence, FTESA set in motion processes that may lead to changes with positive repercussions for the market, particularly where the EAGC continues its policy-influencing efforts. However, in Zambia the inactivity of SOPAG since the programme closure, as well as the exclusion of FTESA partners in policy processes, threatens the sustainability of efforts.


Based on the VfM metrics analysed and benchmarking conducted, there are several areas where the programme performed well in comparison to other similar programmes, as well as improvements in some of the metrics since 2016. VfM metrics on economy improved since the mid-term evaluation in 2016, and FTESA performs better than comparable programmes such as WAFM. The smallholder engagement cost (efficiency) increased between 2016 and 2018, although it remains within the range of other comparable programmes. Portfolio leverage rates (efficiency) improved and are higher than similar programmes. The volume of sales contributed per beneficiary farmer (effectiveness) increased between 2016 and 2018, most likely due to progress in implementation, with results transpiring further along the value chain, as well as the addition of

FINAL EVALUATION

Page | 62

new grants in 2016 that generated substantial sales volumes (e.g. Farm Africa and WFP). The cost of female outreach (equity) was significantly higher than overall reach, suggesting that achieving equity is a cost driver, but it is nonetheless still within the range of other comparable programmes.

The PMU was slow to develop VfM indicators, only starting to collect data in 2016 and not setting any targets in this regard. There was no evidence to suggest that the PMU was using VfM data to feed into ongoing analysis and learning by programme teams and at programme close the PMU provided limited assessment of the VfM data.

FINAL EVALUATION

Page | 63

Lessons and recommendations

The following are lessons and subsequent recommendations drawn from the findings and conclusions of the evaluation, including those on how market development programmes can promote structured trade and systemic change. They are ordered according to the top three priority areas DFID has identified, followed by additional recommendations from the evaluators for similar, future programmes. Some priority areas contain multiple lessons and recommendations.

1. Bringing smallholder farmers into structured regional markets

a) Improved post-harvest markets: access to finance

Lesson: While collateral can help farmers to access credit (one of the original programme assumptions), the experience across the grants shows a wider range of conditions necessary to improve the bankability of smallholder farmers and their access to credit, including: (i) lending to a registered farmer-based organisation (not individuals) that can provide a group guarantee; (ii) trusted suppliers providing inputs and evidence of good quality training conducted; (iii) farmer-based organisations’ exposure to formal sales demonstrated through contracts with buyers; (iv) evidence of high loan repayment rates.

Recommendation: Similar programmes should take a more comprehensive, multi-faceted approach to improving access to finance, focusing on improving the credit worthiness of farmers.

b) Systemic Change: Transparency and trust

Lesson: Farmers altered and improved their practices when there was transparency and trust between the farmers and FTESA grantees, and confidence in the viability of the intervention. Conversely, farmers’ negative experiences with buyers and agro-dealers in the past led to entrenched negative perceptions of some actors in the value chain.

Recommendation: Similar programmes should give adequate attention to the demand-side and to facilitating relationships and contractual arrangements that are open, transparent and based on regular communication which incentivises both parties – the buyer and seller – to fully honour their commitments in a timely manner. They should also allow enough time for interventions to get up-and-running and broker relationships (through trusted partners) across the value chain.

c) Systemic Change: Positive demonstration effects

Lesson: Demonstration effects were important catalysts to generating trust in the intervention and increasing uptake of unproven methods and crops. Changes in practices endure where farmers experienced ‘proof of concept’ (e.g. where new practices led to increased yields) including securing better markets and associated benefits consistently.

Farmers typically require follow-up demonstrations and repeated interactions with trainers to build confidence in adopting new crops, inputs and methods, as well as to help adapt practices as challenges arise over time.

Recommendation: Similar programmes must allow enough time for early adopters to demonstrate benefits to others for wide-spread adoption to occur.

2. Reform to food trade policies and regulations

Lesson: The FTESA experience shows that working through local actors is most effective, particularly those embedded in the local context who can build coalitions and search out like-minded actors and who know who to target and how, and who have strong incentives to reduce barriers to improving the market system. This also helps to ensure accountability, effectiveness and sustainability. This is applicable to other areas of the programme, where developing strategic partnerships with those strongly motivated to change the market system (e.g. partners already established and embedded in the local political context affected by the detrimental effects of market inefficiencies) may lead to long-term sustainable impact.

FINAL EVALUATION

Page | 64

Recommendation: Similar programmes should ensure that they work with local partners who have the incentive and therefore motivation to continue policy influencing engagements and activities, and those who already have influence, particularly those representing constituents most affected by the policy and regulatory constraints.

3. Portfolio inter-linkages and complementarities

Lesson: Award-based mechanisms, by design, can limit the ability to directly build in strong inter-linkages and complementarities across the portfolio.

Recommendation: DFID should ensure that the design of future portfolio-approach programmes, which rely on inter-linkages and complementarities to generate expected results, includes more active hands-on support from PMUs (or similar) in designing projects when needed. This requires a greater investment in technical assistance.

It also requires mechanisms to generate real-time learning and foster coordination and collaboration between implementing partners. This may require alternative models to award-based mechanisms, with PMUs taking a more active role in shaping and designing the portfolio of projects to improve coherence and complementarities through active and ongoing learning.

4. Additional recommendations

d) FTESA’s Value for Money (VfM)

Lesson: The existing VfM metrics are suitable for the assessment but we were unable to find comparable indicators on effectiveness across DFID programmes, which suggests that in future there needs to be more careful selection of appropriate indicators that allow for comparison (e.g. jobs created, etc.). Moreover, future programmes need to assign targets to indicators to track progress and provide incentives to achieve these, as we recommended in the mid-term evaluation.

Recommendation: At the outset, programmes need complete VfM frameworks with metrics that: are comparable with similar programmes; provide adequate coverage of equity; align to the logframe and targets; and outline clear definitions and plans on how and who will collect and analyse data. Such plans need consistent implementation and should feed into ongoing learning and decision-making by programme teams.

e) Systemic Change: Crowding in and sustainability

Lesson: Sustainability hinges on a continuous supply of improved inputs from permanent market actors (e.g. agro-dealers) and a continued demand for them from farmers. It also relies on permanent market actors (e.g. agro-input suppliers, buyers) repeating training and consistently demonstrating benefits, with farmers seeing the benefit. Ultimately farmers should be willing and able to pay for inputs and services if they experience consistent benefits, and market actors therefore have a commercial incentive to continue these functions. As a result, farmers adopt new behaviours and these spread to others as other farmers observe the benefits, spreading the new behaviours further afield through demonstration effects.

Recommendation: The assumption that the benefits generated through interventions catalyse longer term, widespread changes that are sustainable is only likely where programmes give adequate attention to the following:

• Facilitating mechanisms for continual updating of knowledge and learning (for farmers and businesses) to ensure better practices continue and can adapt to the external environment (e.g. new technologies, threats, etc.), alongside consistent positive demonstration effects that lead to wider adoption rates.

• Crowding in other commercial actors to the system who provide incentives for farmers to maintain and continually improve changed practices.

• Ensuring grantees (or similar) develop effective exit strategies, and permanent market actors have the incentive to provide, scale and adapt services.

FINAL EVALUATION

Page | 65

f) Systemic Change: Scaling up, timeframes and targets

Lesson: Rolling out and attempting to scale up interventions quickly often leads to insufficient time to engage users, experiment, pilot, learn and adapt interventions, as was the case with the FTESA-funded G-Soko electronic trading platform.

Recommendation: Similar programmes should take a more cautious approach to scaling up, by extending timeframes and reducing targets, giving time for experimenting, piloting, learning, adaptation etc. and demonstrating results before scaling up. Projects should not (and cannot successfully) attempt to pilot and scale-up simultaneously.

Programmes should also give more attention to the time required to roll out activities and deliver results (e.g. some interventions covered only one or two harvests) and the sequencing of different elements of an intervention (e.g. constructing a warehouse and then establishing a warehouse receipt system).

Moreover, similar programmes attempting to generate change in the wider market system should identify strategic partnerships and leverage a wider network of players already working in the same fields.

FINAL EVALUATION

Page | 66

Annex 1: Common and output theories of change

FINAL EVALUATION

Page | 67

FINAL EVALUATION

Page | 68

Programme theories

Storage and aggregation

By addressing market constraints/failures through improving warehouse, storage and aggregation facilities, FTESA interventions will increase access to warehouse, storage and aggregation facilities for traders and farmers, leading to better (higher and/or more stable) prices, greater sales and increased incomes: by enabling farmers and traders to store surpluses at harvest time when prices are low and sell when prices are higher; by reducing post-harvest losses, hence increasing sales volumes for farmers and traders; by encouraging farmers to grow more produce, leading to more produce stored and sold; by giving farmers greater bargaining power with traders as they do not need to sell all their produce (to avoid spoilage); by enabling traders and farmers to aggregate their produce with others, and negotiate better prices for bulk purchases with buyers

Market information

By addressing market constraints/failures through improving improved market information, FTESA interventions will increase access to market information for traders and farmers, enabling them to use information to base their decisions (production, storage, sales) on more accurate and timely information, increasing trade between surplus and deficit areas and providing better information on when to store and release produce, leading to better (higher and/or more stable) prices, greater sales and increased incomes.

Credit

Original programme theory: By addressing market constraints/failures through improving credit facilities, FTESA interventions will increase access to credit, enabling farmers to use credit to increase investments in inputs and other services, leading to improved yields, higher volumes, greater sales and increased incomes.

Updated programme theory: By addressing market constraints/failures through improving credit facilities, working with financial partners who are willing to work with smallholder farmers, FTESA interventions will increase access to credit, enabling farmers to use credit to increase investments in inputs and other services, leading to improved yields, higher volumes, greater sales and increased incomes.

Grades and standards

Original programme theory: By addressing market constraints/failures through improving the application of standards and grades for staple foods by traders and farmers, FTESA will improve the quality of produce and access to good quality storage facilities, leading to better (higher and/or more stable) prices, greater sales and increased incomes.

Updated programme theory: By addressing market constraints/failures through improving the application of standards and grades for staple foods by traders and farmers, and farmers are incentivised and able to reach standards, FTESA will improve the quality of produce and access to good quality storage facilities, leading to better (higher and/or more stable) prices, greater sales and increased incomes.

Inputs and good agricultural practices

By addressing market constraints/failures in the input market (especially seeds) through the provision of good quality inputs, and inadequate agricultural practices through GAP training, FTESA interventions will improve agricultural practices and use of better quality inputs (including use of market-demanded seed varieties), leading to increased productivity, production and quality, resulting in higher prices, greater sales and increased incomes.

Smallholder farmers’ integration in structured markets

Original programme theory: By increasing access to more and better-quality services and inputs for farmers (including disadvantaged/poor groups – e.g. women), FTESA interventions will enhance the availability of better quality services and inputs for farmers, leading to increased use, improved yields and production, and better-quality produce, leading to better (higher and/or more stable) prices, greater sales and increased incomes for farmers.

Updated programme theory: By increasing access to more and better-quality services and inputs for farmers (including disadvantaged/poor groups – e.g. women), and farmers trust the intervention, find the activities credible and markets are available, FTESA interventions will lead to increased use, improved yields and production, and better-quality produce, leading to better (higher and/or more stable) prices, greater sales and increased incomes for farmers.

Functioning of markets and systemic change

By funding interventions where there is potential to generate wider change in the market (beyond the intervention itself), FTESA interventions deliver changes in market functioning that trigger widespread changes in behaviour (interest, motivations, practices), maintained after external support has ended, leading to higher volumes sold, better prices received, greater integration into value chains and higher profits.

Consumers

By improving availability of food across space, FTESA interventions will stabilise prices for consumers between surplus and deficit regions; by improving availability of food across time, FTESA interventions will stabilise prices for consumers between harvest and hunger seasons.

Linkages and complementarities between grants

By funding interventions where there are identifiable complementarities and synergies, and creating links between interventions (some or all intervention components of a grant, e.g. linking grantees with G-Soko), FTESA generates results that are greater than if interventions were implemented in isolation, and these results continue beyond programme close.

FINAL EVALUATION

Page | 69

Annex 2: Evaluation matrix

Evaluation question and sub-questions Hypotheses/ programme theories Realist enquiry Modules, sources, analysis

Portfolio-level (complementarity/synergies)

1) To what extent is FTESA a collection of individual interventions or a coherent portfolio?

a) What: To what extent has the combination of interventions generated results in excess of the programme's component parts (i.e. generated complementarities/synergies)?

b) How, why and for whom and in what circumstances: How and why have these complementarities/synergies materialised? What were the mechanisms at play? Who has benefitted from the complementarities/synergies? What circumstances (conditions, enabling/ constraining factors) are conducive (or not) to generating benefits in excess of the programme's component parts?

c) What indications are there of sustainability? What is the likelihood these will be sustained after direct support has ended?

By funding interventions where there are identifiable complementarities and synergies, and creating links between interventions (some or all intervention components of a grant – e.g. linking grantees with G-Soko), FTESA generates results that are greater than if interventions were implemented in isolation, and these results continue beyond programme close.

- Are grantees collaborating? To what extent? Who? How and why are grantees collaborating? In what circumstances? What are the results? Any unanticipated outcomes?

- What do you think caused these changes? How has FTESA contributed? Are these changes likely to endure?

Case studies and synthesis; thematic studies; portfolio review

- Key informant interviews (KIIs) and semi-structured interviews (SSIs)

- PMU and grantee documents and data

- Thematic analysis and synthesis

Market-level (systemic change/ sustainability)

2) To what extent is FTESA likely to improve the functioning of national and regional staple food markets and generate systemic change? a) What: To what extent is FTESA likely to improve the

functioning of national and regional staple food markets and generate systemic change? Is there any evidence it has done so, so far?

b) How, why and for whom and in what circumstances: How and why have changes materialised, or are likely to materialise? What are the likely mechanisms for the spread of behaviour changes across networks of actors? Which actors are pivotal to the spread of new behaviours? Who is likely to benefit? What circumstances (conditions, enabling/constraining factors) are conducive (or not) to generating systemic change?

c) What indications are there of sustainability?232 What is the likelihood these will be sustained after direct support has ended?

By funding interventions where there is potential to generate wider change in the market (beyond the intervention itself), FTESA interventions deliver changes in market functioning that trigger widespread changes in behaviour (interest, motivations, practices) that are maintained after external support has ended, leading to higher volumes sold, better prices received, greater integration into value chains, and higher profits.

- Have motivations, interests, behaviours, practices, relationships, etc. of actors in the supply chain changed? To what extent? For whom? How and why? In what circumstances? What are the results? Any unanticipated outcomes?



- KIIs and SSIs



- Adopt-Adapt-Expand-Respond (AAER) framework

232 The EMU will look for early indicators of sustainability—from the trade and market systems literature.

FINAL EVALUATION

Page | 70


Individual-level (producer/farmer/trader/firm/consumer)

3) To what extent have improved trade support systems (output 1: storage, aggregation, information, value chain coordination, grades and standards, credit) increased production and trade?

a) What: To what extent has FTESA improved trade support systems? To what extent has production and trade increased as a result? Where there has been an increase in trade, to what extent has this trade been cross-border or within national boundaries?

b) How, why, for whom and in what circumstances: How and why have these changes materialised? What were the mechanisms at play? Who has benefitted? What circumstances (conditions, enabling/constraining factors) were conducive (or not) to generating benefits for producers, farmers, traders and firms?


i. Storage and aggregation: By addressing market constraints/failures through improving warehouse, storage and aggregation facilities, FTESA interventions will increase access to warehouse, storage and aggregation facilities for traders and farmers, leading to better (higher and/or more stable) prices, greater sales and increased incomes:

- by enabling farmers and traders to store surpluses at harvest time when prices are low and sell when prices are higher;

- by reducing post-harvest losses, hence increasing sales volumes for farmers and traders;

- by encouraging farmers to grow more produce, leading to more produce stored and sold;

- by giving farmers greater bargaining power with traders as they do not need to sell all their produce (to avoid spoilage);

- by enabling traders and farmers to aggregate their produce with others, and negotiate better prices for bulk purchases with buyers

ii. Market information: By addressing market constraints/failures through improving improved market information, FTESA interventions will increase access to market information for traders and farmers, enabling them to use information to base their decisions (production, storage, sales) on more accurate and timely information, increasing trade between surplus and deficit areas and providing better information on when to store and release produce, leading to better (higher and/or more stable) prices, greater sales and increased incomes.

iii. Credit: By addressing market constraints/failures through improving credit facilities, FTESA interventions will increase access to credit, enabling farmers to use credit to increase investments in inputs and other services, leading to improved yields, higher volumes, greater sales and increased incomes.

iv. Grades and standards: By addressing market constraints/failures through improving the

- What [insert storage, credit, etc.] are being accessed or used or adopted? To what extent? How and why? By whom? In what circumstances? What are the results (prices, yields, etc.)? Any unanticipated outcomes?


Case studies, quantitative surveys and synthesis; thematic studies; portfolio review

- KIIs and SSIs



- Contribution analysis (case studies and thematic studies)

FINAL EVALUATION

Page | 71


application of standards and grades for staple foods by traders and farmers, FTESA will improve the quality of produce and access to good quality storage facilities, leading to better (higher and/or more stable) prices, greater sales and increased incomes.

4) To what extent have improved availability and use of inputs (output 2: inputs) increased production and trade?

a) What: To what extent has FTESA improved availability and use of inputs? To what extent has production and trade increased as a result?

b) How, why, for whom and in what circumstances: How and why have these changes materialised? What were the mechanisms at play? Who has benefitted? What circumstances (conditions, enabling/constraining factors) were conducive (or not) to generating benefits for producers, farmers, traders and firms?


i. Inputs: By addressing market constraints/failures by facilitating greater private sector participation in seed and fertilizer markets, FTESA will enhance the availability of better quality inputs, leading to increased use, improved yields and production, and better quality produce, leading to better (higher and/or more stable) prices, greater sales and increased incomes.

ii. Seeds: By addressing market constraints/failures by improving and harmonising seed policy and regulations, FTESA will accelerate the adoption and use of improved seed varieties, leading to improved yields and production, and better quality produce, leading to better (higher and/or more stable) prices, greater sales and increased incomes.

iii. Fertiliser: By addressing market constraints/failures through improving the fertiliser and regulatory environment, FTESA will improve the functioning of fertiliser markets, leading to improved use of fertiliser and improved yields and production, and better quality produce, leading to better (higher and/or more stable) prices, greater sales and increased incomes.

- What inputs (seeds, fertiliser) are being traded by programme partners? To what extent? By whom? How and why? In what circumstances? What are the results (volumes, etc.)? Any unanticipated outcomes?


- What inputs (seeds, fertilisers) are being accessed or used? To what extent? By whom? How and why? In what circumstances? What are the results (prices, yields, etc.)? Any unanticipated outcomes?



- KIIs and SSIs




5) To what extent and how has FTESA brought in (or facilitated) SHFs in structured regional markets?

a) What: To what extent has FTESA brought SHFs into structured regional markets?

b) How, why and for whom and in what circumstances: How and why have these changes materialised? What were the mechanisms at play creating these changes? How have SHFs participated in these markets? Who has benefitted (poor people, women)? What circumstances (conditions, enabling/constraining factors) are conducive (or not) to bringing in SHFs?


By increasing access to more and better quality services and inputs for SHFs (including disadvantaged/poor groups – e.g. women), FTESA interventions will enhance the availability of better quality services and inputs for SHF, leading to increased use, improved yields and production, and better quality produce, leading to better (higher and/or more stable) prices, greater sales and increased incomes for SHF.

- Are SHF participating? To what extent? Who? How and why? In what circumstances? What are the results (prices, etc.)? Any unanticipated outcomes?



- KIIs and SSIs




FINAL EVALUATION

Page | 72


6) To what extent has FTESA benefitted consumers?

a) What: To what extent has FTESA delivered benefits for consumers?

b) How, why, for whom and in what circumstances: How and why have these changes materialised? What were the mechanisms at play creating these changes? Who has benefitted? What circumstances (conditions, enabling/constraining factors) are conducive (or not) to generating benefits for consumers?


i. By improving availability of food across space, FTESA interventions will stabilise prices for consumers between surplus and deficit regions.

ii. By improving availability of food across time, FTESA interventions will stabilise prices for consumers between harvest and hunger seasons.

- Have prices changed? To what extent? For whom? How and why? In what circumstances? What are the results? Any unanticipated outcomes?



- KIIs and SSIs




Regulatory/policy level

7) To what extent have FTESA approaches to supporting reform to relevant policies, regulations, etc. contributed to change?

a) What: To what extent has FTESA delivered policy change?

b) How, why, for whom and in what circumstances: How and why have these changes materialised? What were the mechanisms at play creating these changes? Who has benefitted? What circumstances (conditions, enabling/constraining factors) are conducive (or not) to generating policy change?


By identifying and targeting specific policy and regulatory reform that impedes functioning of the FTESA programme, FTESA will work through partners to facilitate policy and regulatory changes that improves the results generated by the programme and improve market functioning more broadly.

- Have policies changed? To what extent? How and why? For whom? In what circumstances? Any unanticipated outcomes?


Policy dialogue and influencing studies

- KIIs and SSIs



Value for Money

8) Does FTESA offer Value for Money in the results it achieves, compared with possible alternatives?

a) What: To what extent has FTESA delivered VfM overall? Which approaches provide more/less VfM (within the programme and compared to other programmes)?

b) How and why: How and why has FTESA delivered VfM (or not)?

i. FTESA is economical in terms of the cost of the resources used.

ii. FTESA maximises both technical and allocative efficiency (i.e. outputs achieved for a given input).

iii. FTESA is the most cost-effective way of addressing the constraints and achieving expected results.

Not applicable VfM assessment

- KIIs with PMU

- PMU documents and data

- 4E VfM assessment

FINAL EVALUATION

Page | 73

Annex 3: Methodology

The individual evaluation modules provide more details on the methods applied. We provide a summary of key methodological issues here.

1. Sampling strategy and coverage

Case studies: We selected a purposive sample of grant-funded interventions based on a long list of criteria (as outlined in the EMU’s inception report), consulting with the PMU. The following criteria proved most critical:

• Interventions where we could identify and locate farmers (direct beneficiaries) for interview.

• Identifying similar interventions to enable the exploration and testing of programme theories across interventions to enable cross-case comparison.

• Interventions that represent a significant proportion of investment and reflect the geographical spread of the overall portfolio, as well as different business models.

• Interventions that enable the exploration of specific lines of enquiry.

• Balance between different funding modalities (development and challenge fund).

Thematic studies: Originally, the focus of the thematic study was on EAGC/G-Soko and the grants linked to the G-Soko platform. Given several reviews of G-Soko in 2017 and DFID’s interest in exploring other farmer aggregation mechanisms to help inform the new FTESA programme, DFID requested that the EMU also explore Farm Africa (not covered at baseline and only briefly at mid-term). We mapped the physical sites for the two grantees to identify those where we could speak with as many actors as practically possible (given logistical considerations) and cover the most grants with potential/actual links to EAGC/G-Soko and Farm Africa. The other grants included Classic Foods, Kilimo, Raphael, Shalem and Virtual City.

Grant coverage of modules

Grants

Qu

alit

ativ

e en

dlin

e ca

se

stu

die

s Q

uan

tita

tive

en

dlin

e su

rvey

s

Them

atic

stu

dies

Po

licy

dia

logu

e

and

infl

uen

cin

g st

ud

ies

Po

rtfo

lio r

evie

w

VfM

ass

essm

ent

ACTESA

All

20

gran

ts

All

20

gran

ts

Classic x

EAGC/G-Soko x x x

Farm Africa x

Joseph Initiative x x

Kaderes x x

Kilimo x x

Mount Meru x

Raphael x

Shalem x

Virtual City x x

Other grants

The case and thematic studies covered seven out of 20 grants in detail, accounting for 51% of the total value of the portfolio, and 56% of the portfolio when we include the additional grants linked to EAGC/G-Soko and Farm Africa interventions.

FINAL EVALUATION

Page | 74

Policy Dialogue and Influencing Study: We selected policy issues for analysis based on the following sampling criteria:

• Where the intended result happened.

• Where activities are ongoing.

• Policy issues central to the success of FTESA’s overall aims and objectives.

• Pillar coverage: structured regional market systems, seeds and soya.

• Policy issues where FTESA is applying a range of approaches and activities.

2. Evaluation design

2a. Qualitative case studies

Combining contribution analysis and realist enquiry

Combining contribution analysis233 and realist enquiry helps us to understand for each grant:

1. STEP 1 – Contribution analysis: The extent to which the intervention contributed to change and the role played by the intervention versus other factors by assessing: what change has happened, has the intervention contributed to the observed results, how much of a difference/contribution has the intervention made, and what other factors led to the change?

a) Change: What is the pathway to change (drawing on GToC, programme theories and ICMOs)? Have these theorised changes happened? To what extent have these changes occurred, and for which

233 Mayne, J. (2008) ‘Contribution Analysis: An Approach to Exploring Cause and Effect’, ILAC Brief 16.

http://hdl.handle.net/10568/70124

FINAL EVALUATION

Page | 75

different groups, etc.? Are there any unintended results, positive or negative? Are there are any indications of sustainability, and whether these will endure after the programme closes?

b) FTESA contribution: How significant was the FTESA grant’s contribution to the observed changes? What other factors (not related to the FTESA grant) led to the change?

i. What is the evidence that the FTESA grant contributed to causing the observed changes? What is the evidence that non-FTESA factors contributed?

ii. What is the relative importance of the FTESA grant and non-FTESA factors in explaining the observed changes? How much of a difference (contribution) has the FTESA grant made?

2. STEP 2 – Realist enquiry: how and why the intervention worked (or not) and for whom (e.g. youth, poor, women), and in what circumstances/contexts, exploring the influence of the features of the intervention and contextual factors (enabling and constraining factors) on the underlying mechanisms that helped to generate change (or not).

a) How and why did FTESA contribute (or fail to contribute) to the observed changes? For whom (e.g. youth, poor, women) and in what circumstances/contexts?

b) What features of the interventions/grant and contextual factors (individual, interpersonal, organisational and institutional) triggered the mechanisms that contributed to the observed changes (or blocked the mechanisms from occurring)?

c) Assess likely sustainability: Whether there are any indications of sustainability, and whether the changes will endure after programme closure (exploring how and why including contextual factors).

Data quality and strength of evidence

• Quality of data in the underlying case studies: to what extent can we be sure that reported outcomes and the change pathways described in the case study reports happened? Each qualitative case study describes the strength of evidence for reported outcomes based on the following indications of strength of evidence for qualitative research:234

▪ A good degree of triangulation: (a) within interviews, (b) across stakeholders and types of stakeholders, and/or (c) across data sources.

▪ The position, knowledge, analytical capacity, reflexivity235 and potential biases of primary informants.

▪ What we know about the broader context.

We carry this assessment through into the synthesis with further aggregation of the evidence contributing to triangulation.

• Second, in the synthesis, how confident are we that a specific programme theory explains the outcomes and change processes? Is there a range of evidence from across several projects? Or strong evidence from individual projects or activities? The extent to which we are confident that our synthesis of programme theory explains the outcomes and change processes is based on a combination of the strength of evidence for outcomes and how emerging theory compares to the ToC and baseline and mid-term evaluation findings, as well as the degree and extent to which we have evidence from the projects against the programme theories and/or ICMOs.

To assess strength of evidence for (and project contribution to) outcomes and ICMOs (realist enquiry), we apply the criteria developed by the BCURE evaluation team for the realist enquiry in their final evaluation, to

234 We have drawn on the approach developed under the evaluation of the Building Capacity to Use Research Evidence Programme (BCURE). This was a £15.7 million initiative funded by the DFID from 2013–17. It aimed to increase the capacity of policymakers to use research more effectively, through building the skills, incentives and systems required to access, appraise and apply evidence in decision making. The annexes of the realist Final Evaluation (Vogel and Punton, 2018) can be found here: http://itad.com/reports/annexes-final-evaluation-building-capacity-use-research-evidence-bcure-programme/ (accessed 14 June 2018). 235 In this context, reflexivity refers to self-awareness and the degree to which someone has/is able to reflect on their own biases (preconceptions, position, assumptions, values and beliefs) and the ways in which these may affect an investigation and the evidence presented.

http://itad.com/reports/annexes-final-evaluation-building-capacity-use-research-evidence-bcure-programme/

http://itad.com/reports/annexes-final-evaluation-building-capacity-use-research-evidence-bcure-programme/

FINAL EVALUATION

Page | 76

make systematic judgments about the strength of evidence and contribution that are comparable across the case studies.

Outcome and ICMO strength of evidence and contribution

Strength of evidence

Outcomes Realist enquiry Contribution

Strong evidence

High level of confidence that the outcome occurred…

High level of confidence that the outcome occurred/did not occur because of x mechanism, operating in y context and because of z features of the intervention…

High level of confidence that FTESA contributed to the outcome…

• Based on a good degree of triangulation (a) within interviews, (b) across stakeholders and types of stakeholders, and/or (c) across data sources;

• …considering the position, knowledge, analytical capacity, reflexivity and potential biases of primary informants; and

• …also considering what we know about the broader context. Some evidence

More confident than not that the outcome occurred…

More confident than not that the outcome occurred/did not occur because of x mechanism, operating in y context and because of z features of the intervention…

More confident than not that FTESA contributed to the outcome…

But reduced confidence due to: • …shortcomings regarding triangulation;

• …concerns that the position, knowledge, analytical capacity, reflexivity and potential biases of primary informants lowers the reliability of evidence; and/or

• …what we know about what is happening within the broader context.

Limited evidence

Low level of confidence that the outcome occurred, given that…

Low level of confidence that the outcome occurred/did not occur because of x mechanism, operating in y context and because of z features of the intervention, given that…

Low level of confidence that FTESA contributed to the outcome, given that…

• …evidence comes from a small number of sources with limited triangulation;

• …there are major concerns that the position, knowledge, analytical capacity, reflexivity and potential biases of primary informants lowers the reliability of evidence; and/or

• …there are contradictory insights into what is happening within the broader context.

Coding system

We synthesised across the qualitative and quantitative case studies to answer the evaluation questions (1-6). We coded for themes that included outcomes, ICMOs (kept together where possible) and the evaluation questions and programme theories. We coded manually into the case study reports. Coding into the case study documents used a combination of colour coding and using comments to add codes/ keywords to chunks of text. We used multiple codes for sections of text, where relevant.

2b. Quantitative surveys

We conducted quantitative endline household surveys for the two grants subject to quantitative baseline surveys. We carried out before-and-after analysis236 for Kaderes and difference-in-difference237 for Joseph Initiative to assess impact on beneficiaries.

Kaderes quantitative survey238

The team undertook surveys at baseline (Q4 2015) and again in Q4 2017 to measure changes in output and outcome indicators for the Kaderes project. The team interviewed the same respondents at baseline and endline. The baseline panel consisted of an intervention group made up of farmers enrolled with Kaderes, and

236 Without control group. 237 With control group. 238 Itad, Kaderes 2017 Monitoring Study Report (April 2018).

FINAL EVALUATION

Page | 77

a suitable control group (counterfactual) not enrolled with Kaderes.239 The intention was that the baseline design would enable comparison of the level of change in key indicators in the intervention group against the change in the control group to assess the effects of the Kaderes project on its beneficiaries and to capture the effect of the main intervention funded by FTESA, the warehouse. However, given delays in construction, the warehouse was not yet operational in December 2017. Therefore, the EMU and DFID agreed to downscale the endline survey to a monitoring survey, interviewing only Kaderes beneficiary farmers in the intervention group and not the baseline control group. We employed a before-and-after analysis to assess whether extension services (e.g. farmer training) took place between 2015 and 2017, to what extent beneficiary farmers benefited, and whether there were measurable changes in farm indicators such as gross margins and harvest volumes.

Joseph Initiative survey240

The survey followed a quasi-experimental impact assessment design, with difference-in-difference analysis of key indicators between the intervention and control groups at baseline and endline. We conducted surveys at baseline (Q1 2016) and endline (Q1 2018) interviewing the same respondents in a longitudinal panel design, including the Joseph Initiative intervention group and a control group (counterfactual).241 This enabled comparison of the level of change in key indicators in the intervention group against the change in the control group, enabling an assessment of the project’s effect on its beneficiaries. The survey explored whether farmers registered in the project experienced significant increases in their maize crop margins compared to farmers in the control group and whether any such changes were linked to an increase in agricultural efficiency and productivity brought about by the project.

2c. Thematic studies: assessing systemic change

We use system-level frameworks for exploring the presence of systemic change. In answering the evaluation question on systemic change and the sustainability aspects of other questions, we assessed ‘systemic change’ in line with the Adopt-Adapt-Expand-Respond (AAER) framework.242 This examines the behaviours of actors in a market system to identify the degree to which they have adopted behaviours, adapted behaviours to suit their own purposes, or crowded in to a new market segment by exhibiting behaviours they observed. ‘Respond’ refers to other types of changes in the environment (e.g. regulatory) that may occur due to other changes in agent behaviour.

• Adopt. The market player has successfully adopted a behaviour/practice change to the ultimate benefit of the poor producer/worker/consumer, recognises the value of continuing with these changes irrespective of programme inputs, and has accordingly made plans to invest in upholding these changes and cover any associated recurrent costs.

• Adapt. The market player that adopted the behaviour/practice changes pioneered during the pilot has made qualitative and/or quantitative investments that allow them to continue with or augment changed practices, without programme support. These actions, independent of the programme, constitute an ‘acid test’ for whether pro-poor outcomes will be sustained at any level.

239 The intervention group sample at baseline was drawn at random from farmer lists provided by Kaderes. The monitoring survey reached 210 of the 219 baseline farmers. Of the 210 surveyed farmers, 18 had either stopped agricultural activities or dropped out of the programme. This quantitative case study analyses the information from interviews with the remaining 192 farmers, 142 of which grew maize and beans on coffee intercrop systems, comparing 2017 and 2015 information from the same households, employing a panel design. 240 Itad, Joseph Initiative Endline Qualitative Case Study (August 2018). 241 We selected the intervention sample for the baseline survey from lists of farmers living in Masindi district who had signed up with the Joseph Initiative in the 12 months before enumeration. The control group is located in the northern sub-counties Butemba, Gayaza and Nsambya of Kyankwanzi district. This area lies around 50–80km away from Masindi town to the southwest of Masindi district and has similar livelihood zones as Masindi, with a focus on maize cultivation. Sample sizes were calculated with the goal to detect an increase of 50% in the average gross margin within the intervention group between baseline and endline, at 5% significance levels with 80% power. These assumptions resulted in planned sample sizes of 231 for the intervention and 360 for the control group. 242 The Springfield Centre (2014), ‘Adopt-Adapt-Expand-Respond: a framework for managing and measuring systemic change processes’. Itad Farm Africa Thematic Study

FINAL EVALUATION

Page | 78

• Expand. Several market players, like those that pioneered the pro-poor behaviour/practice changes, have adopted comparable changes (either pure copies or variants on the original innovation) that continue without programme support.

• Respond. The emergence and continued presence of the pro-poor changes have incited market players in supporting systems to react to the new market reality by reorganising, assuming new/improved roles, developing their own offers, or moving to take advantage of any opportunities created. The response enables pro-poor behaviour/practice changes to develop further, or evolve, and indicates a new capability within the system, suggesting that it can and wants to support pro-poor solutions to emerge and grow.

2d. Policy Dialogue and Influencing Studies: assessing policy influence

In answering the question on policy influence, we examined specific policy influencing initiatives. The environment for influencing interventions is typically unpredictable: causes and effects are not easy to define clearly and may change depending on the context and circumstances. To interpret FTESA’s progress toward influencing relevant policy and regulatory changes, the study therefore explores programme implementation and the influence of context on the success and failure of interventions.

The study team developed the questions for study based on the teams’ understanding of FTESA’s influencing strategy and what we considered useful lines of enquiry, in consultation with the PMU and DFID, employing a bottom-up (from activities to outcomes) and top-down (from outcomes to activities) approach. In practice, the bottom-up approach was more relevant since many of the policy influencing initiatives were yet to deliver intended outcomes.

• Bottom up: approach to influencing: focus on understanding the approach taken (strategy, engagement plan and activities) for a policy issue, exploring the potential contribution to expected change as well as identifying any unexpected potential changes. Questions include:

▪ What does the programme do to affect change?

▪ Who does the programme aim to influence?

▪ What are the effects of the chosen approach?

▪ To what extent are activities linked and co-occur to deliver results?

▪ How does context influence the achievement of results?

▪ How does the programme learn?

• Top down: Change pathways: focus on understanding an observed change and retrospectively examining multiple potential causes. Questions include:

▪ What happened? What changes occurred?

▪ What caused the observed outcome and why did it happen?

▪ What and who influenced or contributed to the observed change?

▪ What were the most significant contributing factors?

▪ What changed over time in the context?

▪ To what extent did the programme contribute to these changes?

FINAL EVALUATION

Page | 79

3. Data collection methods against evaluation modules

Module Documents and data review

Stakeholder survey

Interviews

PM

U

FTES

A p

artn

ers

Gra

nte

es

Ben

efic

iari

es (

e.g.

farm

ers,

oth

er s

up

ply

ch

ain

acto

rs)

No

n-F

TESA

ben

efic

iari

es

(e.g

. far

mer

s, o

ther

su

pp

ly c

hai

n ac

tors

)

1a. Qualitative case studies

1b. Quantitative surveys

2. Thematic studies

3. Policy dialogue and influencing studies

4.Portfolio review (including online stakeholder survey)

5. VfM assessment

FINAL EVALUATION

Page | 80

Annex 4: Evaluation questions and main modules

Evaluation questions Findings, conclusions, recommendations and lessons learned Synthesis

Cas

e st

ud

ies

and

su

rvey

s

Them

atic

stu

die

s

Po

licy

dia

logu

e

and

infl

uen

cin

g st

ud

ies

Po

rtfo

lio r

evie

w

VfM

ass

essm

ent

Ove

rall

inte

rpre

tati

on

EQ1: To what extent is FTESA a collection of individual interventions or a coherent portfolio?

EQ2: To what extent is FTESA likely to improve the functioning of national and regional staple food markets and generate systemic change?

EQ3: To what extent have improved trade support systems increased production and trade?

EQ4: To what extent have improved availability and use of inputs and application of good agricultural practices increased production and trade?

EQ5: To what extent and how has FTESA brought (or facilitated) smallholder farmers into structured regional markets?

EQ6: To what extent has FTESA benefited consumers?

EQ7: To what extent have FTESA approaches to supporting reform to relevant policies, regulations, etc. contributed to change?

EQ8: Does FTESA offer Value for Money in the results it achieves, compared with possible alternatives?

Shading indicate depth of data collection and analysis against each module

FINAL EVALUATION

Page | 81

Annex 5: Generic data collection templates

a. Interviews

Evaluation questions/ theories

Interview questions

Findings (coded by grant where relevant) Synthesis

INT1

INT2

INT3

INT4

INT5

INT6

INT7

INT8

INT9

INT1

0

INT1

1

INT1

2

INT1

3

INT1

4

INT1

5

INT1

6

INT1

7

INT1

8

INT1

9

INT2

0 Overall

interpretation

Evaluation question and theories

Interview questions, probing questions, etc.

Sub-questions and theories

Interview questions, probing questions, etc.

b. Documents

Evaluation questions/ theories

Findings (coded by grant where relevant) Synthesis

DO

C1

DO

C2

DO

C3

DO

C4

DO

C5

DO

C6

DO

C7

DO

C8

DO

C9

DO

C10

DO

C11

DO

C12

DO

C13

DO

C14

DO

C15

DO

C16

DO

C17

DO

C18

DO

C19

DO

C20

Overall interpretation



FINAL EVALUATION

Page | 82

Annex 6: Module evidence assessment frameworks

a. Case Synthesis (evidence from review of interviews, documents and data)

Evaluation questions/theories Findings Synthesis

CS1

CS2

CS3

CS4

CS5

CS6

QS1

QS2




b. Thematic Studies (evidence from review of interviews, documents and data)


GR

1

GR

2

GR

3

GR

4

GR

5

GR

6

GR

7 Overall interpretation



c. Policy Dialogue and Influencing Studies (evidence from review of interviews, documents and data)


GR

1

GR

2

GR

3

GEN

GEN

GEN

GEN

GEN

GEN

GEN

GEN

GEN

GEN

GEN

GEN

GEN

GEN

GEN

GEN




d. Portfolio Review (evidence from review of documents, data and online survey)


GR

1

GR

2

GR

3

GR

4

GR

5

GR

6

GR

7

GR

8

GR

9

GR

10

GR

11

GR

12

GR

13

GR

14

GR

15

GR

16

GR

17

GR

18

GR

19

GR

20 Overall interpretation



FINAL EVALUATION

Page | 83

Annex 7: Strength of evidence and triangulation

5 very strong; 4 strong; 3 medium; 2 low; 1 very low

Evidence available for each grant

Evidence sources

Jose

ph

Kad

eres

EAG

C

Farm

Afr

ica

Kili

mo

Mo

unt

Mer

u

Vir

tual

Cit

y

Cla

ssic

Rap

hae

l

Shal

em

Afr

itec

ENA

S

Mus

om

a

Seb

a

Soso

ma

WFP

Yak

AC

TESA

Eso

ko

Pee

Pee

Grantee monitoring data and reports Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

PMU verification missions and reports Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

Qualitative case studies Y Y Y Y Y Y

Quantitative surveys Y Y

Thematic studies Y Y Y Y Y Y Y Policy dialogue and influencing studies Y

Portfolio review Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

Overall strength of evidence and triangulation 5 5 4 4 4 4 4 3 3 3 2 2 2 2 2 2 2 1 1 1

Evidence available against each evaluation question

Evidence sources EQ

1

EQ2

EQ3

EQ4

EQ5

EQ6

EQ7

EQ8

Grantee monitoring data and progress reports Y Y Y Y Y Y

PMU monitoring data and progress reports Y Y Y Y Y Y Y Y Key informant interviews Y Y Y Y Y Y Y

Overall strength of evidence and triangulation 3 3 4 4 4 2 3 3

Evaluation modules against each evaluation question

Modules

EQ1

EQ2

EQ3

EQ4

EQ5

EQ6

EQ7

EQ8

Qualitative case studies Y Y Y Y Y Y

Quantitative surveys Y Y Y Y Y Y

Thematic studies Y Y Y Y Y Y

Policy dialogue and influencing studies Y Portfolio review Y Y Y Y Y Y

VfM Assessment Y

Overall strength of evidence and triangulation 3 3 4 4 4 2 3 3

FINAL EVALUATION

Page | 84

Annex 8: Programme context

Snapshot of the country context for the main countries currently receiving support under FTESA.

Current situation in the region (October 2018)

In East Africa, staple food price declined across most markets in Uganda, and Tanzania because of ongoing May-to-August harvests. Prices increased seasonably in most markets in Kenya as supplies tightened with the start of the lean season.243 In Southern Africa, domestic maize availability remained adequate with above average supply levels for Zambia and Tanzania. Maize prices were generally stable. Maize grain and maize meal were able to circulate between surplus and deficit areas without major trade restrictions within the region.244

According to the WFP, El Niño is likely to start towards late 2018, petering out by mid-2019, with a peak sometime in January-February 2019. It is expected to be weaker and shorter than 2015-16’s El Niño. In Southern Africa, rainfall deficits may cause staple food prices to rise from late-2019 enough to affect food security for poor and vulnerable populations. In East Africa, increased rainfall is expected to result in increased production, but will also increase flood risks in Kenya’s river systems.245

Kenya

Although agriculture contributes about 30% of GDP, the country commonly faces food security concerns and therefore depends on formal and informal imports from its neighbours (especially Uganda and Tanzania, and Ethiopia246) and key global trading partners to meet national demand.247 This makes the country vulnerable to volatility of world food prices and trade barriers by other countries.

In Kenya, smallholders account for 75% of agricultural output and 70% of market supplies.248 Women provide 80% of farm labour and manage 40% of smallholder farms, but own only 1% of agricultural land and obtain only 10% of agricultural credit.249 Yields of several staple crops have declined because of land degradation, high costs of inputs and services, overdependence on rainfed production and post-harvest losses of 20–30% for cereals.250

However, Kenya experienced historically above-average March to May long rains in 2018 and this has continued to drive food security improvements.251 In August 2018 prices increased seasonably in most markets in Kenya as supplies tightened with the of the lean season.252

Maize is Kenya’s most important staple food crop, and will remain so in the foreseeable future.253 Over 95% of smallholder farmers (3.5 million) grow maize. It is crucial for the country’s food security.254 There is a forecast marginal increase in maize harvested in 2018-19 due to the Government of Kenya’s new incentives for farmers under its ‘Big Four’ programme, and due to the country’s emergence from a period of erratic

243 FEWSNET (Aug 2018) August 2018 Global Price Watch 244 Ibid. 245 WFP (Sept 2018) El Nino: Outlook 2018 246 WFP (2017) Greater Horn of Africa Climate Risk and Food Security Atlas: Technical Summary 247 Comprehensive Food Security and Vulnerability Analysis (CFSVA) Kenya 2016, World Food Programme 248 Government of Kenya (2009) Agricultural Sector Development Strategy 2009–2020, in: WFP (June 2018) Kenya country strategic plan (2018–2023) 249 Ibid. 250 Mutungi and Affognon (2013) Addressing Food Losses: Status and Way Forward for Postharvest Research and Innovations in Kenya. International Centre of Insect Physiology and Ecology (ICIPE) Policy Brief No. 5/13, in: WFP (June 2018) Kenya country strategic plan (2018–2023) 251 FEWSNET (September 2018) Food security improvements driven by above-average long rains and low staple food prices 252 FEWSNET (Aug 2018) August 2018 Global Price Watch. 253 USDA Foreign Agricultural Service (Feb 2018) Kenya Grain and Feed Annual 2018 254 Comprehensive Food Security and Vulnerability Analysis (CFSVA) Kenya 2016, World Food Programme

http://fews.net/global/price-watch/august-2018-0

https://docs.wfp.org/api/documents/WFP-0000074434/download/?_ga=2.232670687.1302412942.1538494049-1367803070.1536071154

https://docs.wfp.org/api/documents/WFP-0000098939/download/?_ga=2.11840946.72077327.1538651619-1367803070.1536071154

https://docs.wfp.org/api/documents/8681e711600d423caf6c595139c10da4/download/?_ga=2.217418128.72077327.1538651619-1367803070.1536071154

https://docs.wfp.org/api/documents/8681e711600d423caf6c595139c10da4/download/?_ga=2.217418128.72077327.1538651619-1367803070.1536071154

http://fews.net/east-africa/Kenya

https://gain.fas.usda.gov/Recent%20GAIN%20Publications/Grain%20and%20Feed%20Annual_Nairobi_Kenya_3-28-2018.pdf

FINAL EVALUATION

Page | 85

weather and Fall Army Worm infestations. Kenya remains a maize production deficit country, and importations—primarily from COMESA and the EAC—are expected to pick up in 2018-19.255

Rice production, like maize, is part of the Government of Kenya’s food security ‘Big Four’ agenda. Rice consumption continues to increase due to consumer preferences. Demand exceeds supply, and the deficit is currently offset by imports from Asian countries. Rice retail prices were very volatile during 2016-17, but are expected to stabilise.256

Rwanda

FEWSNET expects Rwanda will experience an above average harvest in 2018-19, which is expected to improve household food security and income. Recent successive good seasons have meant better food security for poor households and lower staple food prices as compared to last year.257 A good harvest in mid-2018 increased availability of all staple food crops except beans, for which there were production shortfalls.258

In Rwanda, 88% of agricultural households grow beans, 49% grow maize and 45% grow potatoes.259 Rwanda has the highest population density in Africa, which limits the opportunity to expand area under production. With a growing population, food security is an issue. The government is addressing this through the Plan for the Strategic Transformation of Agriculture II – the main objective is the intensification and development of sustainable production systems.

Tanzania

Maize is Tanzania’s main staple crop,260 but maize productivity is very low in spite of its importance, due to lack of improved seeds, inefficient fertilizer delivery system, post-harvest loss, pests and diseases.261 Tanzania exports maize to Rwanda, Zambia and Kenya, among others.262 Maize prices declined in 2017-18, in part due to the export ban introduced in June 2016.263 In August 2018 staple food price declined across most markets in Tanzania because of ongoing May-to-August harvests.264

Almost 20% of farmers are involved in rice production.265 Rice is also a staple food, which Tanzania imports primarily from Asian countries, but also exports to Rwanda, Uganda and Kenya—in 2017-18 Tanzania exported 5,000MT of rice to Kenya.266 In mid-2017, Tanzania was the second largest exporter of beans in East Africa.267

Uganda

Cooking banana (matoke), dry cassava chips, sorghum, millet, beans, and white maize are important food commodities for Ugandans.268 In August 2018 staple food price declined across most markets in Uganda because of ongoing May-to-August harvests.269

Unlike Kenya and Tanzania, maize is not Uganda’s primary staple and production of maize is primarily an income earner. In mid-2017, Uganda was the biggest exporter of maize and dry beans in East Africa, exporting

255 USDA Foreign Agricultural Service (Feb 2018) Kenya Grain and Feed Annual 2018. 256 Ibid. 257 FEWSNET (Sept 2018) Early start of Season A rains and favorable forecasts bode well for sustained food security 258 FEWSNET (July 2018) Recent harvests improve food availability of all major staples except beans 259 WFP (2015) Rwanda 2015 Comprehensive food security and vulnerability analysis. 260 FEWSNET (Aug 2018) Tanzania Price Bulletin 261 USDA Foreign Agricultural Service (April 2018) Tanzania Grain and Feed Annual 2018 262 Ibid. 263 Ibid. 264 FEWSNET (Aug 2018) August 2018 Global Price Watch 265 USDA Foreign Agricultural Service (April 2018) Tanzania Grain and Feed Annual 2018. 266 Ibid. 267 FAO (July 2017) East Africa Cross-border Trade Bulletin 268 FEWSNET (Sept 2018) Uganda Price Bulletin 269 FEWSNET (Aug 2018) August 2018 Global Price Watch

http://fews.net/east-africa/rwanda/remote-monitoring-report/september-2018

http://fews.net/east-africa/rwanda/key-message-update/july-2018

http://fews.net/east-africa/tanzania/price-bulletin/august-2018-0

https://gain.fas.usda.gov/Recent%20GAIN%20Publications/Grain%20and%20Feed%20Annual_Dar%20es%20Salaam_Tanzania%20-%20United%20Republic%20of_4-6-2018.pdf


http://fews.net/sites/default/files/documents/reports/Uganda_2018_09_PB.pdf


FINAL EVALUATION

Page | 86

primarily to Kenya, followed by South Sudan.270 Uganda imported more rice in this same period, due to two consecutive seasons of below average harvests.271

Zambia

Maize is Zambia’s main staple commodity. FEWSNET reported that Zambia’s large 2016/17 harvest resulted in lower than normal prices for maize (a record harvest of 3.6 million MT) and some cash crops—including soy, and increased access to these foods for farmers and market-dependent households.272 The supply of soy increased in response to high prices offered for it in the previous season, though these prices fell drastically in response to the large supply.273

Zambia is an important exporter of maize in Southern Africa.274 In mid-2017 Zambian maize grain was the cheapest in East Africa and was likely to remain so—in this period Kenya imported 100,000MT of maize from Zambia.275

270 FAO (July 2017) East Africa Cross-border Trade Bulletin. 271 Ibid. 272 FEWSNET (Sept 2017) Food Security Outlook Update: ZAMBIA Food Security Report 273 Ibid. 274 Regional Maize Market Fundamentals, August 2016, FEWSNET 275 FAO (July 2017) East Africa Cross-border Trade Bulletin.

http://fews.net/southern-africa/zambia/food-security-outlook-update/september-2017

FINAL EVALUATION

Page | 87

Annex 9: Results against DFID logframe276

276 FTESA (2018) PMU Final Report.

FINAL EVALUATION

Page | 88

FINAL EVALUATION

Page | 89

Annex 10: Grant portfolio

This annex provides a snapshot of the overall portfolio of grants based on the Portfolio Review Report.277

Grants, type, round and budgets

The total portfolio includes six development fund and 16 challenge fund grants, with two grants cancelled in 2015/16. There were four funding rounds for the challenge fund:

• 2013 round 1: Early bird window invested in new technologies

• 2014 round 2: Inputs and related services

• 2015 round 3: Farmer aggregation mechanisms

• 2016 round 4: Soybean value chain

Between 2014 and 2016, the following received awards under the development fund:

• 2014: EAGC and ACTESA

• 2015: Kilimo and World Food Programme (WFP)

• 2016: Farm Africa non-governmental organisation (NGO) consortium

Grant size

The development fund grants comprise 59% of the total value of grants while the challenge fund accounts for 41%. The WFP grant is the largest and Sosoma the smallest. The average value for challenge fund grants is £0.54 million and £1.8 million for development fund grants.

FTESA grants by disbursement value

277 Itad (2018) Portfolio Review.

£-

£500,000

£1,000,000

£1,500,000

£2,000,000

£2,500,000

£3,000,000

£3,500,000

£4,000,000

FINAL EVALUATION

Page | 90

Overview of grants278

Grant CF/DF Round Year Output Funds disbursed £ Model Specific crops, if applicable

1 Esoko CF Early bird 2013 1 387,048 Market information system

2 Mount Meru CF Early bird 2013 1, 2 933,302 Off-taker Soy

3 Virtual City CF Early bird 2013 1, 2 566,978 Trade platform

4 Joseph Initiative CF Round 2 2014 1, 2 981,311 Off-taker Maize

5 Kaderes CF Round 2 2014 1, 2 450,000 Lead firm consortium Beans

6 Ets Nkubili Alfred & Sons (ENAS) CF Round 2 2014 2 955,633 Inputs (fertiliser)

7 Pee Pee CF Round 2 2014 1 500,003 Services Maize, beans

8 Afritec CF Round 2 2014 2 449,220 Inputs (seed) Rice

9 Musoma Food Co. Ltd CF Round 3 2015 1, 2 329,451 Lead firm consortium Maize

10 Yak Fair Trade CF Round 3 2015 1, 2 542,153 Off-taker Maize, beans

11 Sosoma Industries Ltd CF Round 3 2015 1, 2 103,463 Off-taker Maize, soy

12 Shalem Investment CF Round 3 2015 1, 2 250,034 Lead firm consortium Maize, beans, soy

13 Raphael Group Ltd CF Round 3 2015 1, 2 444,351 Lead firm consortium Beans

14 Classic Foods Ltd. CF Round 4 2016 1, 2 430,355 Off-taker Soy, maize

15 Seba Foods Ltd. CF Round 4 2016 1, 2 754,021 Off-taker Soy

16 ACTESA DF 2014 2014 3 1,057,922 Policy

17 EAGC/G-Soko DF 2014 2014 1, 3 3,647,720 Services, policy, trade platform

18 Kilimo Trust DF 2015 2015 1, 2 1,300,243 Lead firm consortium Beans

19 WFP Farm to Market Alliance DF 2016 2016 1, 2 3,772,760 Forward delivery contract Maize

20 Farm Africa DF 2016 2016 1 2,401,631 NGO consortium Maize, rice, beans

Total 20,257,599

278 We exclude the cancelled grants, Technoserve and Victoria Seeds

FINAL EVALUATION

Page | 91

Grants by output, sub-output and model

Grants typically cover more than one of FTESA’s three output areas:

• Output 1: improved post-harvest markets (storage and aggregation, market information, value chain coordination, credit, standards and grades)

• Output 2: improved input markets (seeds and fertiliser)

• Output 3: improved policy and regulatory framework

Three grants fall under output 1 only; two grants under output 2 only; one grant under output 3 only; 13 grants under both output 1 and output 2; and one grant falls under both output 1 and 3.

Number of grants according to sub-output279

# Output and sub-output No. of grants

1.1 Number of male/female farmers accessing new/improved storage/aggregation services/facilities as a result of FTESA

14

1.2 Number of male/female farmers accessing improved market information system as a result of FTESA

3

1.3 Number of male/female farmers accessing improved value chain coordination (e.g. application of grades and standards to their products, improved logistics and virtual market place) as a result of FTESA

17

1.4 Number of male/female farmers accessing warehouse receipts and supplier credit as a result of FTESA

4

1.5 Number of private sector entities that adopt common grades and standards as a result of FTESA

2

2.1a Volume (MT) of new or improved inputs traded by programme partners as a result of FTESA (seeds)

13

2.1b Volume (MT) of new or improved inputs traded by programme partners as a result of FTESA (fertiliser)

11

2.2 Number of male/female farmers using improved inputs as a result of the activities of programme beneficiary input suppliers

14

3.1 Number of achievable regulatory and policy changes identified for which a dedicated influencing strategy is developed

2

3.2 Number of achievable regulatory and policy changes for which a dedicated influencing strategy is being implemented

2

3.3 Number of identified regulatory or policy changes for which public–private dialogue platform functioning as outlined in each influencing strategy

1

Most grantees are active in improving value chain coordination, supporting farmers to use improved inputs and improving farmer access to storage and aggregation. Only a few grantees are working on improving farmers’ access to market information systems and credit, and with private sector entities on adopting common grades and standards. Only two grantees (EAGC and ACTESA) are working on policy issues.

Several grants implemented farmer aggregation mechanisms under output 1:

• WFP forward delivery contracts: Under the Farm to Market Alliance (FtMA) project, WFP facilitated the signing of forward delivery contracts between aggregators and buyers/off-takers. These contracts included a pre-set ‘floor price’ (a minimum price), reviewed and revised at the time of sale to reflect current market

279 PMU end-of-project grantee data (April 2018)

FINAL EVALUATION

Page | 92

prices. WFP also linked aggregators to agro-dealers to improve smallholder farmers access to services and inputs.

• Farm Africa NGO consortium: Farm Africa’s intervention facilitates farmer-based organisations entering into supply contracts with local buyers. Farm Africa’s project partner RUDI facilitates consortia between farmer-based organisations, input suppliers, buyers and banks. Farm Africa also linked farmer-based organisations to the EAGC’s G-Soko trading platform to enable them to aggregate and sell grain using the platform.

• Lead firm consortium model: Kilimo Trust formed 12 trading consortia, formalised with Memoranda of Understanding (MoUs), that linked smallholder farmers to buyers (a lead firm), agro-input suppliers and financial institutions. Within each consortium, farmer-based organisations signed supply contracts with buyers. Lead firms include Kaderes, Shalem Investments, Musoma Foods and Raphael Group.

• Off-taker model: Off-takers made agreements with farmer-based organisations or individual farmers to buy aggregated quantities of grain. In most cases, off-takers also provided farmers with support on post-harvest handling and access to inputs. Off-takers include Classic Foods, Joseph Initiative, Mount Meru Millers, Seba Foods, Sosoma Foods and Yak Fair Trade.

Other grantees include:

• Esoko, providing access to market information

• Virtual City, providing an agro-voucher application

• Pee Pee, selling Purdue Improved Crop Storage (PICS) bags

• EAGC, promoting grades and standards, certifying warehouses, supporting village aggregation centres, providing an electronic trading platform and influencing staple food policy

• ENAS and Afritec, both retailers/service providers in the inputs market

• ACTESA, working to change staple food policy

Geographical areas

FTESA’s intended programme scope covered nine countries across East and Southern Africa.280 The programme’s operations and grant coverage focused largely on four countries in East Africa (Kenya, Rwanda, Tanzania and Uganda) and one in Southern Africa (Zambia). Tanzania is the country with the greatest grant coverage/activity, followed by Kenya and Uganda.

280 Burundi, Kenya, Malawi, Mozambique, Rwanda, Tanzania, Uganda, Zambia and Zimbabwe.

FINAL EVALUATION

Page | 93

Annex 11: Grant summaries281

FTESA grantees

Alliance for Commodity Trade in East and Southern Africa (ACTESA)

The Alliance for Commodity Trade in East and Southern Africa (ACTESA) is a regional agricultural specialised agency of the Common Market for Eastern and Southern Africa (COMESA). ACTESA's mission is to integrate smallholder farmers into domestic, regional and international markets through an improved policy environment and expanded market facilities and services. FoodTrade ESA’s investment has assisted ACTESA to facilitate the domestication of harmonised seed trade regulations in East and Southern Africa. The improvements in the policy and regulatory framework for seed production and trade within the region is expected to result in: 1) increased trade volume for improved seed, 2) increased number of and incomes for smallholder seed producers, 3) increased volume of improved seed produced, 4) increased number of farmers accessing improved seed. Farm Africa - NGO Consortium

Afritec Seed Company Ltd.

Afritec Seeds Ltd. is a limited for-profit company registered in Kenya with its core business in hybrid rice seed. The company is a beneficiary of product development contracts with BBSL and Hybrids East Africa Limited (HEAL). With the grant from FoodTrade ESA, the project is taking hybrid rice technology from the research farms of HEAL, and into the hands of small farmers around Eastern Africa. A key to Afritec’s strategy is ensuring that farmers get provided pure, high quality clean seed and to help them transition from traditional varietals to the hybrid types. Afritec is marketing the very first hybrids developed in sub-Saharan Africa, seeds bred specifically for the needs of African farmers.

Classic Foods Ltd.

Classic Foods Ltd. is a food processing company established in 2007 to buy milk in-bulk quantities from farmers and sell it to milk processors, transforming into a milk processor in 2009. The business has since expanded into maize flour, porridge composite flour and animal feeds. To overcome the adverse climate change related challenges and the resultant erratic supply of raw materials, Classic Foods is providing capacity building and farmers’ support initiatives, with the aim of increasing farmers’ soybean produce and growing their markets. The Enhanced Crop Production, Value Chain Additions and Post-Harvest Crop Management project is further providing farmers with a ready market for their produce by procuring all sellable quantities of soybean to process maize-soya porridge flour and use the soy cake to process animal feeds. The farmers are benefitting from capacity building and extension service programs implemented by the company in Kenya and Uganda. Farmers are organised into groups and cooperatives for ease of administration.

The Eastern African Grain Council (EAGC)

The Eastern African Grain Council (EAGC) is a private sector membership-based organisation registered in Kenya. With the grant awarded by UK Aid through the FoodTrade ESA programme, EAGC in partnership with various industry stakeholders developed a private sector-driven, market platform that is linking buyers and sellers in staple foods trade across East Africa. The G-Soko platform is promoting structure and consistency in grain trade. It is facilitating title transfer, market transparency, and price discovery. The platform reduces transaction costs because coordination is managed through a centralised exchange. The G-Soko market platform is also accelerating utilisation of grades and standards for maize and beans, promoting certification of regional warehouses and extending the use of a regional warehouse receipt mechanism. Certified warehouses are automated and linked to G-Soko. In summary, G-Soko is ensuring that farmers growing maize and beans in East Africa and can sell their produce across the region through regionally certified warehouses.

Ets Nkubilli Alfred & Sons Ltd. (ENAS)

Established in 1997, Ets Nkubilli Alfred & Sons (ENAS) is a limited, for-profit company registered in Rwanda with the core business of fertiliser distribution. In addition to fertiliser importation and distribution, ENAS

281 FTESA (2017) Summary Grantee Profiles.

FINAL EVALUATION

Page | 94

is involved in livestock breeding, coffee washing stations, agriculture extension services, agricultural inputs and the buying and selling of staple crops. The goal of the grant is to establish the first Nitrogen-Phosphorous-Potassium (NPK) blending plant in Rwanda to blend fertilisers for the Rwandan market and secondarily, Burundi, Uganda and the DRC.

Esoko Networks Ltd.

Esoko Networks Ltd. is a for profit company aimed at making agriculture more profitable for smallholder farmers. Esoko has grown to become a leading initiative in delivering market information to farmers across Africa. The company developed a real time, SMS-driven, market information service for grain farmers, buyers, resellers and other agri-businesses. The East African Regional Market Information System (EARM) enables farmers to learn market prices, understand price trends, gain knowledge on crop management and climate issues, and better manage input supplies.

Farm Africa - NGO Consortium

Farm Africa works to reduce poverty permanently by unleashing smallholder farmers’ ability to grow their incomes and manage their natural resources sustainably. The project is supporting smallholders in rural East Africa to increase their household income through participating in regional trade of staple crops. This was achieved through improved post-harvest handling and storage of staple crops, and linking farmers with surpluses in the key crops (maize, rice and beans) to new markets. Project activities are resulting in increased volume and value of staple crop sales by smallholder beneficiaries

Joseph Initiative Ltd.

The Joseph Initiative Ltd. is an agricultural commodity trading and processing company registered in Uganda. The company integrates small-scale production with regional buyers of whole grain and pulses by managing a fully capitalised and coordinated supply chain for dried agricultural commodities. It manages massive micro-quantity production support and village procurement - supporting and executing thousands of sub-one metric ton purchases while simultaneously managing upstream activities such as fulfilling large-scale, premium quality contracts. The Joseph Initiative has created commercially-viable access points to agricultural and financial services that are catalysing productive growth for Ugandan maize producers. The Joseph Initiative Rural Injini Project is improving end-to-end integration of smallholder maize and legume farmers into regional markets by reducing post-harvest loss at the farm level and increasing productivity by facilitating access to finance. The Rural Injini Project has developed village-based infrastructure to engage smallholder farmers, supplying them with inputs, storage solutions and financial services, and integrating them into a coordinated supply network that sources and distributes grains from farm gate to regional markets.

Kaderes Peasants Development (KPD) PLC

Kaderes Peasants Development (KPD) PLC is a public, limited for-profit company registered in Tanzania, whose core business activities are the processing, marketing and export of both fair trade and organic coffee (primarily to the European market) and beans. KPD’s Warehouse for the Poor (W4P) project has created access to a warehouse in Kagera, in which agricultural products of smallholder farmers will be deposited. With W4P, farmers will receive warehouse receipts as proof of ownership of their products upon deposit and will also be able to obtain credit against their inventory at affordable interest rates. Loans will be provided by a partner financial institution in Tanzania. Farmers are being supported to market their crops during and after the season through cross-border trade.

Kilimo Trust

Kilimo Trust is an independent organisation working on agriculture for development across the East Africa Community (EAC). This project is focused on establishing trading systems that build the confidence of large buyers such as exporters, processors and large institutions, who hesitate to depend on local smallholders for supplies. This will break the vicious cycle typical with beans, where those serving domestic markets resort to importation, while potential large exporters to global markets are reluctant to take orders from international markets for fear of failure to deliver. The organisation is expanding structured trade of beans in response to specific market demand, enabling small holder producers of dry beans in the EAC, to operate commercial production enterprises that are profitable and become consistent, reliable and competitive

FINAL EVALUATION

Page | 95

suppliers of raw and semi-processed beans to medium and bulk buyers and processors serving national, regional and international markets.

Mount Meru Millers Ltd. (MMML)

Mount Meru Millers Ltd., a subsidiary the Mount Meru Group, manufactures soft edible oil procured from locally grown oilseeds. MMML is currently the only processor in Tanzania equipped with a solvent plant which helps in the extraction of oil from cake, with three production units in Tanzania. The company’s total crushing capacity is 50,000 Metric Tons per annum. Under the FoodTrade Project, MMML is promoting soybean growing among smallholder farmers in Tanzania and North East Zambia as a profitable rotation crop, and providing them with technical assistance, agro-inputs, pre-financing and a guaranteed off-take through an out-growers’ scheme. MMML will process soybean at its milling plants, producing refined soybean oil fortified with vitamin A + D as cooking oil and soy cake for the animal feed industry in Kenya and Tanzania.

Musoma Food Company Ltd.

Musoma Food Company Limited (MFCL) is a company established in 2008 trading in maize and rice. The main markets for the grains are Tanzania National Food Reserve (NFRA), domestic wholesalers and cross border traders mostly from Kenya, Rwanda and Burundi. The Sustainable Quality Maize (SQM) Project is solving the problem faced by MFCL where there is frequent shortage of maize and its products. The company has increased business efficiency by supporting smallholder maize farmers to enhance their productivity and quality of maize. MFCL with support from FoodTrade ESA has set up a sustainable maize aggregation system, ensuring the company procures the kind of quality maize needed. The company is aggregating maize of specific quantity and quality levels to meet market demands. Adherence to quality involves training on acceptable East African cereal and grain quality parameters. The company has contracted the Tanzania Food and Drug Authority to perform quality assurance on a periodic basis.

Pee Pee Tanzania Limited (PPTL)

Pee Pee Tanzania Limited (PPTL) is a limited, for-profit company registered in Tanzania, whose core business is the production and sale of woven packaging and sheeting for the agricultural and industrial sectors. Based in Tanga, the company is currently the regional market leader in the manufacture of packaging for agricultural produce, particularly fertiliser, seeds and other industrial products. Under this project, PPTL has enhanced its existing capacity to produce and sell PICS bags designed with a polypropylene outer and two inner high-density ethylene liners. This product has improved post-harvest storage capabilities and incomes by allowing farmers the flexibility between storing or selling their crops.

Raphael Group Ltd.

Raphael Group Limited (RGL) is one of the leading grain processors and distributors in Tanzania, with head offices in the Southern Highlands. It was established in 1995 and incorporated as a Limited Liability company in 2010. The core business activities of RGL include processing and selling of more than four grain products including rice, beans, groundnuts, sunflower cooking oil and other cereal grains. The project Raphael Group Bean Marketing Centre has setup a centralised marketing centre for the highly demanded bean varieties grown in the Southern Highlands region, supported by mini aggregation centres situated in the bean production locations. The marketing centre is owned by RGL, while the mini-aggregation centres are co-owned by RGL and the bean producer groups. This project is unique because it has set up an aggregation system which is partly owned by the farmers. By making farmers part owners of the village warehouses, the project is giving them opportunity to increase their income through revenue generated by the warehouses.

Seba Foods Zambia Ltd.

Seba Foods Zambia Limited is a Zambian registered company and a renowned producer of maize and soybean based consumer food products. Over the years, the company`s products have become house hold names in Zambia, priding themselves in very high quality products through well-known brands such as Golden Goodness. The project is increasing the capacity of soybean production and providing value addition to the oil seed for local consumption at competitive prices. The company is also playing the role of off-taker, aggregating and processing the soybean. The company is supporting smallholder farmers to commercialise their activities, increasing overall production of soybeans in Zambia and promoting cross-border trade of the finished products.

FINAL EVALUATION

Page | 96

Shalem Investments

Shalem Investments is a private for profit Social Enterprise whose core business is aggregating, transporting and marketing grains and legumes (maize, sorghum, beans, wheat, and sunflower). It was established by the lead founder, Ruth Kinoti, to respond to a desperate need of the people closest to her. “A stitch in time” project is innovatively “stitching” together target value chain actors with relevant strategies to create a unique mechanism of facilitating collective marketing for smallholder farmers. The project has built on Shalem’s unique business model, and is incentivising farmers and aggregators to take responsibility for cleaning and grading their produce, increasing their bargaining power and economies of scale. To ensure accessibility of appropriate agronomy and post-harvest management practices, the project has invested in creating relevant linkages, improving collection centres and providing capacity building.

Sosoma Industries Ltd.

Sosoma Industries Ltd. is a limited company established in 2008 in Kigali, Rwanda. The company started its business by manufacturing maize meal, roasted soybean flour, as well as red and white sorghum flours originally produced by the Sosoma production unit. The project has increased the company’s storage capacity of grains from 800MT to 2,600MT, with 88% of the grain being maize and soya. The company has provided training to smallholder farmers in order to increase the production and quality of maize and soya beans, and enhanced smallholder grain trade by promoting aggregation, building awareness of grades and standards, and storing the grain. The project contracted three rural cooperatives which have 10,000 members to supply maize and soya grain.

Virtual City Ltd.

Virtual City is a Kenya based mobile technology provider that specialises in supply chain automation, particularly in the agriculture and Fast Moving Customer Goods (FMCG) sectors. Its solutions provide supply chain visibility, reducing fraud, increasing efficiency, and creating data-driven insights. The Agro Voucher solution presents a new business model that enables transactional revenue to be shared amongst market actors, not just the top of the value chain.

UN World Food Programme (WFP)

WFP is the food aid arm of the United Nations system. Food aid is one of the many instruments that can help to promote food security, which is defined as access of all people at all times to the food needed for an active and healthy life. The project is building structured grain markets by linking smallholder farmers to formal markets locally and regionally. This is achieved by promoting demand, and strengthening farmers’ resilience. The Farm to Market Alliance (formerly known as Patient Procurement Platform) is a public-private sector led consortium of organisations seeking to transform food value chains in emerging markets. It is building a demand-led value-chain based on long-term linkages between suppliers (farmers), buyers and other key market players such as suppliers of finance, inputs and technical expertise. The Alliance is also actively engaging smallholders, increasing their productivity, profitability and resilience and their strength as reliable market players.

Yak Fair Trade Limited

The company, established in 2010, deals in agri-business (including livestock); this includes purchase and sale of agricultural commodities. Current investment in the commodities business includes an area equivalent to 100 Hectares for crop production and livestock development. “Nkunganire in Marketing and Post-Harvest Handling” is helping the farmer get access to finance, improve production and processing, decrease transaction costs and increase economies of scale as well as giving them access to wider markets. The project has helped farmers by offering them a chance to improve the quality of beans and maize produced by providing quality seeds and grading.

FINAL EVALUATION

Page | 97

Annex 12: Key documents consulted

DFID

DFID (2013) FTESA Business Case and Intervention Summary.

DFID (2013-2017) Annual Reviews.

FTESA EMU

Itad (2014) FTESA Theory of Change Workshop Report.

Itad (2014) EMU Inception Report.

Itad (2016) Baseline Case Evaluation.

Itad (2016) EAGC Baseline Qualitative Case Study.

Itad (2016) FTESA Mid-term Evaluation.

Itad (2016) Joseph Initiative Baseline Qualitative Case Study.

Itad (2016) Joseph Initiative Quantitative Impact Baseline.

Itad (2016) Kaderes Quantitative Impact Baseline.

Itad (2016) Kaderes Baseline Qualitative Case Study.

Itad (2016) Kilimo Trust Baseline Qualitative Case Study.

Itad (2016) Mount Meru Millers Ltd Baseline Qualitative Case Study.

Itad (2016) Thematic Study.

Itad (2016) Value for Money Assessment.

Itad (2016) Virtual City Baseline Qualitative Case Study.

Itad (2017) Mid-term Evaluation Report.

Itad (2018) Case Evaluation Synthesis.

Itad (2018) EAGC Endline Qualitative Case Study.

Itad (2018) Farm Africa Thematic Study.

Itad (2018) Joseph Initiative Endline Qualitative Case Study.

Itad (2018) Joseph Initiative Quantitative Impact Assessment.

Itad (2018) Kaderes 2017 Monitoring Study Report.

Itad (2018) Kaderes Endline Qualitative Case Study.

Itad (2018) Kilimo Trust Endline Qualitative Case Study.

Itad (2018) Mount Meru Millers Ltd Endline Qualitative Case Study.

Itad (2018) Policy Dialogue and Influencing Study: Kenya.

Itad (2018) Policy Dialogue and Influencing Study: Zambia.

Itad (2018) Portfolio Review.

Itad (2018) Qualitative Case Study Evaluation Design.

Itad (2018) Stakeholder Survey.

Itad (2018) Virtual City Endline Qualitative Case Study.

FINAL EVALUATION

Page | 98

FTESA PMU

FTESA (2013) PMU Inception Report.

FTESA (2013-16) grantee proposals/applications (various).

FTESA (2013-16) grantee work plans, budgets, KPIs (various).

FTESA (2013-16) grantee MRM plans (various).

FTESA (2013-16) CF and DF rounds concept notes (various).

FTESA (2013-17) grantee quarterly reports (various).

FTESA (2013-17) PMU annual reports (various).

FTESA (2014) Monitoring and Results Measurement Manual.

FTESA (2016) Enhancing the Poverty Impact of FTESA: Mainstreaming Poverty within Operations and Systems.

FTESA (2017) Summary Grantee Profiles.

FTESA (2018) grantee final reports (various).

FTESA (2018) PMU end-of-project grantee data.

FTESA (2018) PMU Final Report.

Others - methods

Barr and Christie (2014) Better Value for Money: An organising framework for management and measurement of VFM indicators. Itad.

BEAM Exchange (2015) ‘The 4E approach to VFM.’ Monitoring guidance.

Better Evaluation (2014) Content Analysis.

Mayne (2008) ‘Contribution Analysis: An Approach to Exploring Cause and Effect’, ILAC Brief 16.

Miles and Huberman (1994) Qualitative Data Analysis.

Nippard, Hitchins and Elliott (2014) ‘Adopt-Adapt-Expand-Respond: a framework for managing and measuring systemic change processes’. Springfield Centre.

Noblit and Hare (1988) Meta-Ethnography: Synthesising Qualitative Studies.

OECD (1991) DAC Principles for the Evaluation of Development Assistance.

Pawson and Tilley (1997) Realistic Evaluation.

Rogers et al. (2000) Program theory evaluation: Practice, promise, and problems. New Directions for Evaluation, 87, 5-13.

Stern et al. (2012) Broadening the Range of Designs and Methods for Impact Evaluations.

Westhorp (2014) Realist Impact Evaluation: An Introduction.

Wong et al. (2013) Realist Synthesis: Rameses Training Materials.

Yin (2003) Case Study Research: Design and Methods.

Others – context

FAO (2017) East Africa Cross-border Trade Bulletin. July 2017.

FEWSNET (2016) Regional Maize Market Fundamentals. August 2016.

FEWSNET (2017) Food Security Outlook Update: ZAMBIA Food Security Report. September 2017.

FEWSNET (2018) August 2018 Global Price Watch. August 2018.

https://www.betterevaluation.org/evaluation-options/content_analysis

FINAL EVALUATION

Page | 99

FEWSNET (2018) Early start of Season A rains and favorable forecasts bode well for sustained food security. September 2018.

FEWSNET (2018) Food security improvements driven by above-average long rains and low staple food prices. September 2018.

FEWSNET (2018) Recent harvests improve food availability of all major staples except beans. July 2018.

FEWSNET (2018) Tanzania Price Bulletin. August 2018.

FEWSNET (2018) Uganda Price Bulletin. September 2018.

Jayne (2010) Patterns and Trends in Food Staples Markets in Eastern and Southern Africa.

USDA Foreign Agricultural Service (2018) Kenya Grain and Feed Annual 2018. February 2018.

USDA Foreign Agricultural Service (2018) Tanzania Grain and Feed Annual 2018. April 2018.

WFP (2015) Rwanda 2015 Comprehensive food security and vulnerability analysis.

WFP (2016) Comprehensive Food Security and Vulnerability Analysis (CFSVA) Kenya 2016.

WFP (2017) Greater Horn of Africa Climate Risk and Food Security Atlas: Technical Summary.

WFP (2018) El Nino: Outlook 2018.

WFP (2018) Kenya country strategic plan (2018–2023). June 2018.

FINAL EVALUATION

Page | 100

Annex 13: Final Evaluation Terms of Reference

[see separate document]

Date post:	05-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times