1 Mentoring three ISIF-funded projects in evaluation and research communication August 2016 Sonal Zaveri, Vira Ramelan, Ricardo Ramirez & Dal Brodhead About the Series .....................................................................................................2 Introduction ...........................................................................................................3 U-FE .......................................................................................................................................................................... 3 ResCom ................................................................................................................................................................... 3 Why combining U-FE and ResCom makes sense? ................................................................................ 3 Overview of ISIF......................................................................................................4 Beginning the Journey: Establishing Readiness .................................................................................... 4 The partners ......................................................................................................................................................... 7 Link TB with Technology (LTT). Operation ASHA, Cambodia ....................................................... 7 A Mobile Application & Mapping Platform to for Accountability in the Health Services for Tea Garden Workers, Nazdeek, Assam, India................................................................................. 8 Cook Islands Maori Database. Cook Islands Internet Action Group. ........................................... 9 Lessons Learned......................................................................................................9 Evaluation Readiness........................................................................................................................................ 9 Research Communication Readiness ...................................................................................................... 11 Implementation .................................................................................................... 11 Users, Uses and Key Evaluation Questions (KEQs) .......................................................................... 11 Challenges with Users, Uses and KEQ .................................................................................................... 14 a) OpAsha’s journey ........................................................................................................................................ 15 b) Nazdeek’s journey ...................................................................................................................................... 17 c) Cook Island’s journey ................................................................................................................................ 19 Outcomes: use of the findings............................................................................... 20 OpAsha’s outcomes......................................................................................................................................... 20 Nazdeek’s outcomes ....................................................................................................................................... 21 The Cook Islands’ outcomes ....................................................................................................................... 23 Integrating UFE and ResCom ................................................................................. 25 Nine lessons learned ...................................................................................................................................... 27
AbouttheSeriesThis series of case studies emerged from an action-research project entitledDevelopingEvaluationandCommunicationCapacityinInformationSocietyResearch(DECI-2). ThepredecessorDECI-1project focusedonlyonevaluationmentoring inAsia. The subsequent DECI-2 project collaborated with research networks andgrantees supported by International Development Research Centre’s (IDRC)Information&NetworksProgrambetweenJuly2012and2017.ThisparticularcasesummarizesworkwiththeISIFgrantsprogrambasedatAPNICinBrisbane,Australia.The DECI-2 Team started off by helping this project develop evaluation plans andcommunicationstrategies. The initialDECI-2 roadmapconsistedofa sequenceofplanning steps in evaluation and communication, some of which were clearlycomplementary.DuringtheprecedingDECI-1project,wewitnessedhowutilization-focused evaluation (UFE) works as a decision-making framework within whichnumerousevaluationapproachescanco-exist.Thecommunicationstepsturnedouttobequitesimilarastheplanningsequencechallengedprojectmanagerstobeclearabouttheircommunicationpurposes,audiencesandexpectedchanges.As evaluation and communication were linked together, we discovered that bothprocessescreatedadecision-makingframeworkforprojectpartnerstoexpressandagree on their assumptions, expectations, and outcomes. The approach creates apressureonstakeholderstomaketheimplicit,explicitandconsequentlyhelpsteamsclarify their Theory of Change. With research projects and with experimentalinitiatives,thisprocesscantaketimeasemergentoutcomescanprovidefeedbacktocausestakeholderstoadjustprojectobjectivesandstrategies.Using thisapproach ishowwecameabout thenotionofahybriddecision-makingframeworkwhereevaluativeandcommunicativethinkingworkas twosidesof thesame coin. All thiswork to re-discover human nature: as soon as you encounterexcitingnewsyoufeelcompelledtoshareit.DECI-2was developed as an action-research project in capacity development.Wetestedmentoring as awayof providing evaluation and communication support toourpartners.RegionalmentorsbasedinAsia,Africa,andLatinAmericaprovidedthebulk of the mentoring. While our main partners were IDRC-funded researchnetworks (part of the Information & Network Program), we have also tested theapproachwithprojectsinotherfields.
3
IntroductionWe have developed an integrated approach that combines Utilization-FocusedEvaluation (U-FE) and Research Communication (ResCom) as complementaryprocessesthatcanhelpresearchprojectsincreasetheirlong-termoutcomes.
U-FEIn simple terms, U-FE is an evaluation approach proposed by Patton (2004)1thatseeks to generateuseful evaluation. In order to attain such a goal,U-FE follows aseriesofiterativestepsfromtheearlystagesofaprojectthatneedstobeevaluated.Thepurposeof the steps is to help the evaluator facilitate a process that enablesher/himtoimplementthekeyelementsoftheapproachthatincludes:• Identificationofprimaryevaluationusers;• Identificationofprimaryevaluationpurposesanduses;• Formulationofkeyevaluationquestions(KEQ)inasystematicway;• Identification of relevant/cost-effective data collection tools and analysis
processes;• Facilitationoffindingsuse.Although Patton, the original proponent of the U-FE approach, recently increasedthenumberofUF-Estepsto17,forthesakeofsimplicitytheDECI-2teampreferredtofollowtheoriginal12-stepprocess(Patton,2004).
ResComResComreferstotheuseofcommunicationstrategiesformakingresearchfindingsavailable, inatimely,relevantandusefulwaytopolicymakersasameansofmoreeffectively influencing public policy. Although therewasn’t a step-by-step ResComprocessas inthecaseofU-FE,theDECI-2teamproposedasimilar12-stepprocessfor ResCom that would cover similar topics on the communication side. Such aprocess isbased largelyon theRAPID framework (2004)2andon thecommonandcomplementary aspects of ResCom andU-FE. In the sameways thatU-FE tries tomake evaluation “useful”, ResCom focuses on “useful policy-influencingcommunication”. It isworthwhilementioning that as in the caseofU-FE, the flowbetween steps is iterative rather than linear. Table 2 summarizes the proposedResComprocesssteps.
WhycombiningU-FEandResCommakessense?Asindicatedearlier,DECI-2’scentralassumptionforcombiningU-FEandResComisthat sucha combinationcanhelpprograms increase their long-termoutcomes.AsdescribedontheDECI-2website,fromapracticeperspectiveDECI-2combinesU-FEandResCombecause:• Theyshareanumberofcommonplanningsteps(situationalanalysis,stakeholder
• The emphasis on “use” in UFE is comparable with the emphasis on targetedmessagesincommunicationplanning.
• The emphasis on “facilitating use” in UFE, where the evaluators ensure theevaluationfindingsgetutilized(asopposedtobeinglefttochance),remindsusthat communication activities and products need follow-up to heighten theireffectiveness.
• The integration of evaluation and communication processes ensures that wefocusoncommunicationobjectivesthatarerealistic,andmeasurableintermsofreachandshort-termoutcomes.
Despite these complementarities, there are also some significant differences thatcanmakeitdifficulttoutilizetheU-FE–ResComcombination.ThemaindifferenceisthatwhileResComhasaveryspecificpurpose–usingcommunicationto influencepolicy,U-FEisquiteflexibleintermsofitspurposeanduse.Inthisregard,itiseasiertofindtherequiredprojectreadinesslevelforconductingU-FEthanforconductingResCom.AnotherpracticaldifferenceisthatU-FErequireslesstechnicalknowledgefor peoplewhowant to learn how to use it. It does not require a background inevaluation to learn how to conduct U-FE. In contrast, ResCom is quite difficult toimplement for someonewho does not have a background in communication andwhodoesnotunderstandthedynamicsofinfluencingpolicy.
OverviewofISIFThe Information Society Innovation Fund Asia (ISIF Asia) is a grants and awardsprogram aimed at stimulating creative solutions to ICT development needs in theAsiaPacificregion.ISIFofferedgrantsandawardsonacompetitivebasistogroupsinterested in ICTaction-research.The ISIFprogramishostedbyAPNIC inBrisbane,Australia; an organization that partnered with DECI-1. That process enabled theorganization to apply andbenefit fromUtilization Focused Evaluation (UFE). SonalZaveripreparedacasestudyonthatexperiencein2011.ISIFtheninvitedDECI-2topartneragain,thistimetomentoraselectionoftheirgranteesintheAsia&Pacificregioninbothevaluationandcommunication.
BeginningtheJourney:EstablishingReadinessDECI-2 and ISIF invited expressions of interest formentoring inUFE and ResearchCommunication (ResCom) from twelve successful ISIF grantees in the 2014-15fundingprogram.Thefirststepwastoverifytheirlevelof“readiness”.Theoriginalsteps of UFE call for a review of readiness at both the organizational and theevaluators levels. Our experience with DECI-1 confirmed the importance of thesestepsandinDECI-2weextendedthisreadinessreviewtoalsocovertheResComside.The readiness assessment took place before launching anymentoring in UFE and
5
ResCom. The DECI team visited Brisbane to meet with the senior managementbehindtheISIFproject.Subsequently,threeISIFgranteeswereselected,fromIndia,CambodiaandtheCookIslands.
“Readiness” meant that senior decision makers in the host organizations werewillingandabletocommitstaff,timeandsomeresourcestowardscapacitybuildinginboththeevaluationandcommunicationareas.Further,itmeantthattheywerecommittedtodesigningandusingevaluationsaswelldevelopingacommunicationstrategy and associatedmaterialsas part of thework expected from the researchgrant.WealsomadeitclearthatthecombinationofUFEandResComwasnewtous, and that we would be adapting this approach with them as part of our ownresearch agenda. Since the projects had limited resources, we underlined ourcommitmenttomakingtheprocessusefultothem.Organizations were expected to identify staff, “mentees” with relevant skills whowere interested in learning from thementoring in evaluation and communication.Such“readiness”implicitlymeantthatseniordecisionmakersunderstoodthevalueofUFEandResComandtheopportunityprovidedbythementoringprocesstobuildorganizationalcapacitiesinevaluationandcommunication.Welearnedthatatbestwecouldcapture“readiness”astheintenttocommittotheprocessoflearning,butthat it was very difficult for thementees to fully understand how andwhat theirinvolvement in this process would be. In other words, there was a leap of faithrequiredatthestart.TheveryfactthatDECI-2wasintroducedasanaction-researchprojecthelpedconveythenotionof learningandadapting,whichallowedpartners
toassumealearningapproachmorereadily.TheDECImentorshadanumberof“readinessdialogues”usingemailandskypewiththosegranteeswhoshowed interest inUFEandResCommentoring.WeexplainedwhatUFEandResComwas,howitwouldbenefittheorganization,howmentoringwouldtakeplace–(emails,skypes,webinarsandatleastonefacetofacemeeting),andwhattypeofcommitmentfromtheorganizationwasnecessary.The‘readinessdialogues’ enabled the mentors to engage senior decision makers, gauge theircommitmentandtoassessthepresenceofbasicanalyticskillsandinterestinstaff(forUFE)andcommunicationexperience(forResCom).TheavailabilityofsomeskillsisanecessaryandimportantreadinesscomponentforUFE and ResCom. Evaluation and communication may not be priority issues forseniordecisionmakers,buttheUFEandResComprocessesaresuccessfulonlyiftheorganization commitsnotonly to learning,butalso to systematically following thenumerousstepsoutlinedinbothUFEandResCom.3Inotherwords,thekeyquestionwas:wouldtheselectedorganizationsbewillingtolearnandstaythecourseoverseveralmonthsofmentoring?Having been through the UFE process in DECI-1, the ISIF grant manager was achampionforUFE(andtheaddedcomponentofResCom)andplayedanimportantrole inpromoting thevalue of thementoringwithpotential grantees. “Readiness”needstobeindicatedbydonorsorgrantmanagersastheroadaheadinlearningtoimplementUFEandResComisnotsmooth.Thisisthecasebecausementeessetthepaceoftheirlearning,oftengoingbackandforthonwhatneedstobeevaluatedandcommunicated, forwhom andwhat purpose. The patience and trust of the grantmanager or donor in such individualized hand holding capacity building process iscritical.Thisintenttoengagewastranslatedintopracticalsteps:a) By organizations – the senior decision makers in each of the 3 grantee
organizations sent a formal expression of interest to receive mentoring,committed to allocate amodest, realistic budget to implement the evaluationandcommunicationplans.Theyidentifiedpointpersonsintheirorganizationstobe ‘mentees’, individuals who would receive the one on one support frommentorsandtranslatethelearningintoactionintheirownorganizations.
b) Bymentors:Throughaseriesofemailexchangesandskypeinterviews,thethreegrantee organizations and staff were vetted on their “readiness” to receivementoringinUFEandResCom.
Three2014ISIFgrantrecipients–OperationAsha(Cambodia),Nazdeek(India)andCook Islands Maori Database (Cook Islands, South Pacific Ocean) – fulfilled thereadiness criteria andwere selected to receive additionalmentoring in UtilizationFocusedEvaluation(UFE)andResearchCommunication(ResCom)byateamofDECI-3Weworkedwiththeoriginal12stepsofUFE(notthe17describedin:Patton,M.Q.2012.Essentialsofutilization-focusedevaluation.Sage)alongwith12comparablestepsinResCom,see.http://evaluationandcommunicationinpractice.net/about/why-we-combine-them
7
2 regionalmentors (Asian) inUFE and ResCom, backstopped by the twoDECI – 2projectleadsinCanada.
Thepartners
LinkTBwithTechnology(LTT).OperationASHA,CambodiaOperationASHA(OpAsha) isanNGOwithapresenceinIndiaandCambodiathatprovidestuberculosis(TB)careto the most disadvantaged populations. In Cambodia,OpAsha works closely with the government toimplement theTBprogram ineightprovinces, reachingout to about 20% of the population. Cambodia andSouth Africa are among the High Burden Countriesaccording toWorld Health Organization (WHO, 2014)4.On average, each undiagnosed/untreated patientpotentially infects 12 others, leading to a geometricincrease in infections. In2014,aboutone thirdofTBcases inCambodia remainedundetected.Thereisanurgentneedtoscaleupdetectionofthesehiddenpatientsto limit potential outbreaks in the community. With grants from ISIF, OpAshadevelopedamobileapplication(App)thatcouldpotentiallybeagamechangerinTBdetectioninCambodia:itbridgesthegapbetweentheGovernment’sTBprogrammeand undiagnosed TB patients. The contact tracing App’s geographic mappingfunction also allows mobile field supervisors to identify and react quickly tocommunityhotspotswithhighTBprevalencetopreventoutbreaks.Fieldsupervisorsvisitfamiliesofexistingpatients,factorieswherepatientsworkandalso go door-to-door in areas they serve. They use the App to educate thecommunity on symptoms of TB, ask them to answer a basic questionnaire, andsubsequentlyfacilitatesputumtestinganddiagnosisofsuspects.TheAppservesto
connect patients and form a wirelesslink with the microscopy centers sothatnopatientislosttofollow-up.ThisAppiseasytouseandcanbeadaptedtofitanylanguageorculturalcontext.With the App, OpAsha has increasedaccuracyofTBdetectionby12%.Photo:DECI-2mentorsViraRamelan–left- and Sonal Zaveri –middle visitOperationAsha.
AMobileApplication&MappingPlatformtoforAccountabilityintheHealthServicesforTeaGardenWorkers,Nazdeek,Assam,India.Tea garden workers in Assam have insufficient access to health facilities andessential services. Existing facilities are severely underequipped and understaffed,and many villages are located in underserved and remote areas. As a result, teagardenworkers,many of them from the indigenous (“Adivasi") community, sufferhigh rates of maternal and infant mortality with minimal access to legal andadvocacyresourcestoaddressviolations.Nazdeek5, a human rights organization, was interested in piloting a nine-monthproject that tracks, maps, and receives real-time incident reporting. This processtakesplacethroughaSMSmobileandmappingplatformoncasesofmaternalandinfantmortalityinonedistrict(Sonitpurpopulationis1,924,110),anareawithhighAdivasipopulations(approx.600,000).NazdeekpartneredwithaninternationalNGObased in New York, the International Center for Advocates Against Discrimination(ICAAD),tobuildaSMSmobileandmappingplatformandcollaboratedwithPajhra,alocalactivistorganization,toimplementtheApp.
Pajhra has worked for many years in the community around Tezpur, Assampromoting the rights of indigenous labourers working in the tea gardens. For theNazdeek project, they have one Project Coordinator who works with two Block(government administrative unit) Coordinators. In each block, there are about 20womenvolunteersenrolledintheuseofmobiletechnology.Fortywomenvolunteersweregivenmobilephonestoreporthealthrightsviolationsthat have been pre-coded by type.Women have to text the code the violations,whicharereceivedatthePajhraofficeandimmediatelyverifiedbyastaffthroughacall back. Sometimes the Pajhra team does site verification. The violations arepopulatedonamap,whichinformsthelocationandtypeofviolation.
5NazdeekisahighprofilehumanrightsNGO.Theyhavegoodanalyticalandcommunicationrelatedskills. Four of the team members are non-profit human rights lawyers and also have somecommunicationexperiencewithpublishingopinion-editorials(Guardian,DailyBeast,HuffingtonPostblogs)andorganizationnewsletters,publishingYouTubevideos(seeYouTubechannelsforICAADandNazdeek), and being interviewed on radio and television (Al-Jazeera, NDTV, local-ABC, local-Fox,BelgianTV).
9
CookIslandsMaoriDatabase.CookIslandsInternetActionGroup.TheCookIslandslanguageisontheUNESCO’sendangeredlanguageslist.Theaimofthe project is to develop a database of Cook Islands Maori Words, their Englishtranslations and example usages in a sentence in both English and Maori. Thedatabase ismade available in various forms -web,mobile, email, SMS and socialnetwork applications where users can search for words in either English or CookIslandsMaori,andsubscribetodailyrandomwordsserviceorcontributenewwordsto the database. The intention is for the language not only to be preserved andretained, but it can also evolve as native speakers add new words for modernconcepts to an easily accessible online Cook Islands Maori Dictionary. Otherobjectivesofthegrantwere:toengagetheDiasporainthecollection,organizationanddisseminationoflanguageinformation;toutilizeandbuildonthetechnicalskillsalreadyavailableincountryforITProjectManagement,topromotewebandmobileapplicationdevelopmentandprovideteachingplansforteacherstoutilizetheCookIslandsMaoriDatabaseintheirclassrooms.
LessonsLearned
EvaluationReadinessThe “readiness dialogues” had tentatively outlined that there would be acommitment of time, resources and personnel. However, it was challenging foranyoneembarkingonthisjourneytotrulyunderstandtherolesofthementors,thementees and accept that uncertainty.Mentees were used to a more directiveapproach in capacity building, to be students rather than learners, expectingmentors toprovide all the answers. Facilitatedbut exploratory learning at one’sownpacewasanunfamiliarprocess.We opted to start the readiness reviewwith emphasis onUFE. OpAsha’s CountryDirector’sexpectationsaboutUFEatthebeginningoftheprojectwereverydifferentto what was to unfold during the mentoring process. The Director expected toparticipate inanevaluation“how to” courseonwhat isevaluation,how tocollectdata, analysedatausing “fancy”analysis techniquesand soon. TheUFE12 stepshad been shared through documents and webinars, but it took a while for theCountryDirectortounderstandthatthementoringandlearningwasa“process”andthat there would be a great deal of back and forth across the various UFE andResComsteps.Initially, the DECI-2 mentors expected to have one evaluator mentee and onecommunicationmentee.However,theNGOsweresmallwithpersonnelresponsiblefor multiple tasks. This meant that often, one mentee doubled up to fulfil bothevaluationandcommunicationfunctions.FortheDECI-2team,thementeereferredto the contact person for evaluation and/or communication mentoring. Anassociated challengewas thenew terminology: inUFE thosewhoownanddesigntheevaluationare“primary intendedusers”of theevaluation.However, this term
10
can easily be confused with the project end-users (the intended beneficiaries).Furthermore, the evaluation primary user need not be thementee (more on thisbelow). Thetablebelowsummarizesthepersonsandrolesthatwefinallysettleduponineachproject:OpAsha,Cambodia(*) Nazdeek,India(**) IAG,CookIslands(***)
clearly warns against the ‘evaluator mentee’ being the ‘User’ stating that oneshouldavoidthis ‘evaluationtrap’.However, inthecaseofOpAsha,thementeehadtoalsotakeonthemantleofapartuser.InthecaseoftheCookIslands,theboth mentee and User were from the same organization but roles becameblurred.
severaliterations.TheMenteewasfromNazdeekandahumanrightslawyerwhohadbothanalyticalandcommunicationrelatedskills.TheUserwasidentifiedassomeone from the grassroots implementing organization, Pajhra. Nazdeek wasverysurethattheywantedPajhratodevelopevaluationandcommunicationskillsand they believed thementoring process in UFE and ResComwas an excellentwaytodoso.
(***) “Seeing as we only had two people on the team, it was easy sharing the
responsibilities.ThecommunicationdesignatewasAnobecausehewasthewriterofthemajorityofthetechnicalreport.I[Maureen]wasabletodotheanalysisofthe survey that we used to evaluate the project and identify who would beappropriate users of our database and application too. “ Report, Learning fromtheUFEandResComProcess,CookIslands.
Overall, the mentees background was largely technical (software, legal) so theygainedagreaterappreciationofcommunicationandevaluationfromthisstudy.The selected organizations were aware that additional timewould be needed forUFEandResCom.OneateammemberfromtheCook Islandsnotedthattheextraworkloaddidimpactonthetimetheycouldgivetotheoriginalgrant(fromISIF)anddelayedtheirreportingresponsibilitiesforboththegrantmanagerandthementors.The team strongly felt that the time needed for the ‘additional’ UFE and ResComtaskswas underestimated. This however did not deter from their valuing theUFEandResComlearningexperience,whichtheyfeltbuildtheircapacities.
11
ResearchCommunicationReadinessWe have found that the communication side of a project often emergesspontaneously and intuitively as a set of unplanned activities. Communicationproducts can be stand-alone materials or media that are produced with limitedknowledge of audience needs, or theirmedia preferences. In this instance, eachproject had a communication component in its design; though much of it wasneither explicit nor strategic at the start. Production of communicationmaterialstendedtobeagoalperse,ratherthanameanstosupportachievementofawell-definedobjective.Asaresult,communicationrolewasreducedastechnicalmaterialproduction.Wefoundthattherewasanarrowunderstandingof“communication”.Itwasoftenassociated mainly with dissemination campaigns. Thus it was a bit difficult andconfusing for project staff to be asked to define the intent or purpose and theobjectivesofcommunication.Theywerehesitant tocarryouta“proper”audienceresearch, although they did have some knowledge about their audiences’preferences. With the absence of clear communication purposes, objectives andaudiences’ preferences,made it difficult for the projects tomeasurewhether theintended change was achieved. At the start, it was not clear to them how theformulation of the intent and objectives of communication, in combination withaudienceresearchwouldbringaboutbenefitsandnotbeaburden.From a communication perspective, thismeant that their readiness needed someextrasupport.Sinceweworkedwithsmallprojectsteams,andinsomecaseswithasingleperson in chargeof bothUFEandResCom, this readiness challengeneededattention, with attention not to overwhelm them. What we began to discoverhowever,wasthatthetwocomponentsworkedwelltogether.Allprojectshadaninnovation to test through field experimentation; they could easily identify thosepartnerswhowouldbeinterestedif itworked(audiences).Theycouldalsoidentifythe evidence that each audience would require as proof of concept around eachinnovation. In short, for these projects having evidence and sharing it could beunderstoodasatooltoimproveperformance.
Implementation
Users,UsesandKeyEvaluationQuestions(KEQs)AllthreeorganizationswentthroughvariousiterationsofidentifyingUsers,UsesandKEQsandmostoftheinitialdiscussionstodefinethesewereintense.ThefollowingexamplefromOpAsha illustrateshowtheprojectcameupwithseveraloptionsforUsers,UsesandKEQs.OpAshaFirstApproximationUSER: WHO (World Health Organization in Cambodia)
USE: May publish findings in public health journals to influence public policy on
KEQ: To evaluate the TB algorithms that result in high TB yield, based on Cambodia’s
12
approach to case finding
epidemiology
OpAshaSecondApproximationUSER: CENAT (Government of Cambodia Center for Leprosy and Tuberculosis)
USE: Use our findings to influence and to substantiate CENAT’s policy direction to implement electronic monitoring and evaluation system and electronic medical record
KEQ: To evaluate whether the use of technology has an impact on the TB detection in areas of intervention
OpAshaFinaldesignUSERS: OpAsha’s Country Director and the Program Director (medical officer) jointly planned to use the findings.
USE 1: Internal- To develop a feedback mechanism to enhance the scale-up of the next pilot project, which integrates the eDetection App funded by ISIF grant with two other Apps for roll out in Daunkeo Operational District Takeo Province (~200,000 people). USE 2: External- To disseminate findings to donors, government and potential adopters to increase funding or uptake of the use of the technology to deliver TB care.
KEQ 1: How has the App contributed to TB care in terms of screening, case notification compared to conventional methods? KEQ 2: What are the Field Supervisors’ challenges and benefits in using the App for TB care and service delivery? KEQ 3: What are the lessons learnt for scaling up?
As it turnedout, theUsewas immediate (withResCom inputs) at the ICT4D2015Conference in May 2015 where the App was demonstrated, and the evidencegathered from the studywas sharedwith an international audience and potentialfunders.In India,NazdeekandPajhraalsowentthroughseveral iterationstodecideontheUsers,UsesandKEQs.Nazdeek&PajhraFirstApproximationUSER: Local activists including Nazdeek, PAJHRA
USE: To collect data to advocate and litigate govt.
KEQ
Nazdeek&PajhraSecondApproximationUSER: Community members and local leaders
USE: to create awareness for access to maternal health services
KEQ
Nazdeek&PajhraThirdApproximation USERS: Local Government and Courts
USE: for policy level choices to improve health services
KEQ
Nazdeek&PajhraFinaldesignUSER: Project Coordinator, USE: To learn about KEQ 1: What are the factors that
13
PAJHRA
implementation to improve the program, so that Pajhra and Nazdeek are better informed to source the next round of funding.
affect the participation of volunteers (women who use the mobile phones to report violation of health rights)? KEQ 2: How do the ASHAs (community health workers), women whose rights have been violated, community leaders, health facility staff and community perceive the changes that the project has brought? With reference to knowledge about health rights violations, better access to health services, better quality of services. What has worked and what has not? What more can be done?
In2015theCookIslandscelebratedtheir50thanniversaryofself-rule.TheirversionoftheMaoriLanguagewasontheUNESCOendangeredlanguageslist.Theconcernremains that the language will not survive into the 21st century. Hence, it firstappearedtobeanexcellentopportunitytochooseaUserfromthegovernmentanduse the information to engagedonorswhowereplanning to come to kick off theanniversary celebrations in early 2015. Numerous meetings were held with theDeputyPrimeMinister’sSecretaryandPersonalAssistant,theMinistryofCulture–including one in the presence of the DECI-2 mentors. However, there was astalemate in the elections and after waiting for the political results for severalmonths,theCookIslandsteamhadtorevisetheirUser,UsesandKEQ.Itistotheircreditthattheywereabletodosoontheirown–discardtheoldplansand create new ones, a testimony of the skills acquired through the mentoringprocess.AninterimplanwasmadetargetingtheMinistryofEducationandworkingwithteacherswhowouldusetheAppinmainstreamMaorieducation.Howeverthisplanalsocouldnotbeusedbecauseofthecontinuingstalemateinelectingthenewgovernment.Aswiththeothertwoorganizations,oneoftheseniorteammembersoftheCookIslandsInternetActionGroupbecamethementeewhereasthesoftwaredeveloperbecame theUser. Theurgentneedwas to continuedeveloping theApp forwhichthefundswereneeded.Onceagain,theorganizationwasabletoplantheUsesandKEQ.CookIslandsIAGFirstApproximationUSERS: Deputy Prime Minister’s Secretary and Personal Assistant, the Ministry of Culture
USE:
KEQ: To
CookIslandsIAGSecondApproximationUSER: Ministry of Education representative
USE: To approach government and donors for funding and credibility would be high because results were obtained from mentoring through an IDRC funded project and supported by APNIC/ISIF.
KEQ 1: Is it (website/mobile app) a useful tool? KEQ 2: If so, who is it useful tor? (*)
(*) In term of: Demographics (age, gender, location); actual numbers; preferredmedium for access (Android, iOS, Web, Book, other); and Language Fluency forMaori/English.AsasidebenefittheUFEandResComstudywouldprovidebetterinsightsaboutthetargetusersto:
Inotherwords,oneteammemberbecamethementeeforevaluationandResCom,shedeveloped the survey,andprovidedmaterials to the localnewspaper; in turn,shewasassistedbytheUsertogetthe‘surveyout’ontheinternetplatform.
ChallengeswithUsers,UsesandKEQIdentifyingUsers,linkingthemtoUseandKEQprovedtobechallengingforallthreeorganizations.E-mailexchanges,Skypesandwebinarsprovidedthe‘howto’–whatare thecriteria to identifyUser,howdoyouhelp theUser identify theUseof theevaluation and how to develop KEQ to support Use. However, this theoreticalunderstanding using remotementoringwas inadequate to finalize User, Uses andKEQ. As theOpAsha Country Director for Cambodia stated, she: “…got stuck onsomecomponentsofUsersandUses–Itriedsohardtopinitdown.WharetheKEWs– askedmuyself 10,000 times.Once thatwas sorted out, everything quickly fell inplace.”A face-to-facemeetingprovedtobecritical inbreakingthedeadlockof identifyingUsers,UsesandKEQ.Thementors’budgetpermittedonlyoneface-to-facemeetingandithadtobecarefullytimedsothatorganizationswere‘ready’tomovesmoothlythroughseveral stepsandderivemaximumbenefit fromthe intensiveoneononementoring. Thementors were aware of the outputs expected from the grantees.Theyhadreadrelevantdocumentssuchasthegrantapplications,summaryreportsand sought clarifications from the mentees by e-mail and Skype. Initially it wasdifficult for mentors to understand why the selection of Users, Uses and KEQsproved to be so difficultwhen the grant applicationswith expected outputswereavailable.The face-to-face meetings with each project enabled mentors to understand thecomplexityoftheprojects,whattheyweretryingtoachieve;andtheirconstraintsas
15
well as the capacities of the staff.While the project documents explained details(suchaswhat theApp intended toachieve), the face-to-facemeetingenabled thementors tounderstand thecontext inwhich theorganizationswereworking, staffcapacity and interrelationships as well as the political environment in which theyworked.InUFEthisrepresentsastepknowassituationalanalysis.TheidentificationofUsers,UsesandKEQstriggereda‘projectaudit’andaninformalunravellingof theunderlyingTheoryofChange. It focusedonbasic issuessuchas:what resources was the organization investing in the project, what were theunderlying assumptions for selection of the activities, what change was expectedinitially and more long-term and how the change would impact the targetpopulation?EachorganizationwasintroducinganAppthathadculturalandpoliticalovertones and it was important to understand the context in which the Appwasbeingintroducedandtoassessiftheproposedchangewaslikelytobesustainable.TheseconsiderationswereimportantforUFEandResComasfindingsweremeanttopromotesomechangeandinfluenceothers.Intwooftheprojects(OpAsha,CambodiaandMaoriDatabase,CookIsland),theUFEand ResCom mentoring process went closely together. Although some Skypediscussions and emails exchange were done separately, both UFE and ResCommentorswereabletovisitandworktogetherwiththeprojectteamineachlocation.This face-to-face mentoring – which turned to be the most effective mentoringmethod. In contrast, the mentoring process for Nazdeek project was slightlydifferent.OnlytheUFEmentorwasabletovisittheproject,astheResCommentorwas not able to do the face-to-face visit to India. This meant that most ResCommentoring process with Nazdeek and Pajhra team was done through emails andSkypediscussions,followedbyoneadditional,face-to-facemeetinginBrisbane6.
a)OpAsha’sjourneyOpAsha Cambodia had difficulty identifying the User. UFE suggests that the Usermust also identifyUseand link that to theKEQnecessary toprovide findings thatsupport Use. This discussion of Use and KEQ that can help to identify the mostappropriateUser.Forexample,OpAshahadvariousdialogueswithaninternationalhealthexpert fromWHOwhowasembedded in theMinistryofHealth, to identifyUse and KEQ. OpAsha was looking for a champion within the government topromotetheire-detectionApp(withevidenceonitsefficacythroughtheUFEstudy)asaviablealternativetoexistingsolutionsforTBcasefinding inthecommunity.Aseries of KEQ were developed with the expert such as how the technology hadimproved detection, community’s response to the new technology and fieldsupervisor’s performance. Yet, while discussing Use, the User suggested writingjournal articles on this pilot. Clearly, this was not OpAsha’s expectation regardingUse(infactitsoundedlikeacommunicationobjective).Agreatdealoftimewaslostinengagingwith thispotentialUser– theexpertwasverybusyandappointmentsweredifficulttoget,severalmeetingswereneededtosensitizeaboutUFE,explain6Forphotosofthemeetingsee:https://www.flickr.com/photos/gabrielvaldezcadena/albums/72157647395892848
16
roleoftheUser,encouragetheexperttobecomeapotentialUser,identifyKEQsandso on. During the mentor’s face-to-face visit, it was apparent that OpAsha’s e-detectionAppwasapilotandthegovernmentwouldentertainitswiderapplicationonly when the reach and evidence was substantial. Clearly, the expert User andOpAsha’sexpectationofwhattheUsercouldUseitforwasunrealistic.OpAsha is a small NGO with a Country Director, a medical officer in charge ofprograms, and a large field staff. The Country Director was the UFE and ResCommentee since she had the education, experience, good knowledge of English andanalyticability.ThemedicalofficerwasselectedtobeaUsersincehewasawareoftheTBdiagnosisandtreatmentandcouldprovideinputfortheKEQ.However,thekeyUserwasmore likelyby theMenteebecauseofherposition (CD)and rapportwiththegovernment.TheMenteewasalsotheUser’sboss,anditwasdecidedthatboththeMenteeandtheMedicalOfficerwouldbejointUsers.Thisdecisionmeantthat there was greater chance of institutional learning. This decision in hindsightproved to be very useful as the Medical Officer was able to participate in asubsequentconference,ICT4DinSingapore,todiscussfindingsandcontributetothedevelopmentofcommunicationmaterials.InthewordsoftheCountryDirector,therole of “User” was the “hook” which helped the medical officer go “fishing” indifferentfora–thisUserwentfrombeinginvolvedtoengagedintheprocess.The ResCom component benefitted from this process. Identifying USE, USER, andKEQledtoanin-depthdiscussiononabroaderorganizationalaswellaspoliticalandsocial contexts. This had some similarities with situational analysis, one the firstimportant steps in Research Communication. However, some elaboration oncommunication-relatedmattersor“communicationaudit”wasalsodone.Itincludedan assessment of the existing practices of project communication, the extent towhich communication products or activities met the current needs of theorganization, communication capacity among project team members, resourcesallocatedforcommunicationproducts,etc.In OpAsha the institutional capacity of communication was lacking. The CountryDirectorwassolelyresponsibleforallprojectcommunicationimplementation-dueto the limited capacity of the existing staff. The communication products that theprojectproducedinthepastincludedtrainingmaterialsforfieldcoordinators,powerpointpresentation,reports,andvideo.Theproductionofcommunicationmaterialstendedtobespontaneous,asanoperationalframeworkforcommunicationmaterialdevelopment did not exist. The lack of clear communication purposes, objectivesandtargetaudiencemadeitdifficulttomeasuretheeffectivenessofcommunicationmaterialstheyproduced.ThediscussionofUSER,USEandKEQduringtheface-to-facevisithelpedtoshowthelink between UFE and ResCom. For instance, the second Evaluation Use [“todisseminatefindingsandevidencetodonors,governmentandpotentialadopterstoprovidemorefundingoruptakeoftheuseofthetechnologytodeliverTB”]requiresan explicit and structured communication strategy and plan. The OpAshamenteeconfirmedthat the intentofcommunicationshouldbe tosupport their immediate
17
need of using evidence to promote the App. So in this case the evaluation USEoverlappedwithoneoftheResearchCommunicationpurposes.As explained earlier, the 2015 ICT4D Conference in Singapore was seen as animmediate opportunity forOpAsha to demonstrate theApp andpromote findingsfromthestudytointernationalaudienceandpotentialdonors.ItwasthendecidedthatOpAshawouldpreparetwocommunicationmaterials,powerpointpresentationand poster to be shown at the conference. The content and structure of thosematerials were discussed and reviewed together with mentee and mentors. Thiskind of peer-review prior to the production or demonstration of communicationmaterialswasneverbeeninitiatedbefore.
b)Nazdeek’sjourneyNazdeekhadchallenges indecidingwhat toevaluateandhowtouse the findings.ThecontextwashighlycomplexandtheSkypecallsprovedtobeinconclusiveanditwasdifficult for thementor tounderstand thebottlenecks. Thementor’s visit totheprojectsiteallowedhertoappreciatethesituation.ShewasabletounderstandthepartnershipbetweenNazdeek -ahighprofilehumanrightsNGO-andPajhra -acommittedfield levelNGOfighting for therightsof themarginalized.Nazdeekhadfiled two human rights violations in the state’s high court with documentaryevidence and the atmosphere was very tense when the mentor visited Tezpur,Assam. In addition, the USP 7 of the project, which was greater communityinvolvementinreportinghumanrightsviolation,wasitselfunderthreat.Whilethegeo-mappingofhealthrightsviolationswastakingplace,theteagardenownersandgovernmentwereunawareofit.Thewomenvolunteerswereafraidtoreporthealthrights violations as some of them were employed by the government (as nurse,nurseryteacher,etc).DiscussionswiththePajhrastaffindicatedthatsomewomenvolunteerswerenottextingtheinfringements,theyfacedproblemsinselectingtheright code, and did not seem to bemotivated.Manywomen had never reportedwhileotherswereveryactive (textingstarted inMay2014). Itwasalsoclear thatwithout good community level data, Nazdeek would not be able to confront thegovernment. The face-to-face discussions enabled a better understanding of theprojectunderreview,thechallengesbeingfacedandthecontext inwhichtheAppwasbeingused.Several options were explored for Users and Uses. One option for User was thegovernmentbuthavingunderstoodthepoliticalundertonesofNazdeekandPajhra’swork,andthefactthattheydidnothavemuchofaworkingrelationshipwiththem,ruledoutthispossibility.OtheroptionsforUserswere:theco-founderofNazdeek,ahuman rights lawyer, located at their Delhi office, and the Program Director ofPajhra.TheUsewouldbetogatherdatatocommunicatetopotentialfundersastheprojectwasendinginNovember2014(theysubsequentlyreceivedanextensionof3months from ISIF).Thementee lawyer fromNazdeek felt thatwritingproposals tofunderswasnotaproblembuttherewasanurgentneedtoaddressimplementationchallengesparticularly in community involvement, theirproclaimedUSP.For these7uniquesellingproposition
18
reasons,itwasdecidedthatthePajhraProjectCoordinator(ISIFgrant)wouldbetheUser.TheUsewastolearnabout implementationtoimprovetheprogram,sothatPajhraandNazdeekwerebetterinformedtosourcethenextroundoffunding.ItappearedthatPajhraandNazdeekwereonacollisioncoursewiththegovernment.It was critical for Nazdeek to understand that keeping a door open forcommunication was important for the realization of maternal rights. It must benotedthatAssamisaborderstateintheNorthEastofIndiaandagitationpoliticsiscommon. Authorities are less likely to listen and proactively assist in reviewingmaternalhealthviolationsiftheorganizationsareidentifiedwithagitationpolitics.TheUFEmentorassistedbothNazdeekandPajhratoreviewtheirtheoryofchangeandinthediscussionsaboutKEQ,UsersandUses–aprogramaudithappenedquitenaturally. This was immensely useful to the team. On return, this is what thementeeemailed: Itwasgreat tomeet youandget somuchworkdone in suchashorttime.Althoughitwasallcompletelynewtous,wearenowintherightplacetogettheUFEmoving.(August8,2014)The result of the situational analysis also shed light on an opportunity for acommunication intervention. Having known that Nazdeek was on collision coursewiththegovernment,DECI-2teamstronglysuggestedNazdeekandPajhratofocustheir project communication on building a good communication with thegovernmentinordertogettheirattentionontheurgentneedtostopviolationsofhealthrightofwomenlaboursatteagarden.Theunderlyingassumptionwaswhileagovernmentmightbepartoftheproblembuttheyarealsopartofthesolution,andthatchange–inthiscaseissupportfromthegovernment,willlesslikelytooccuriforganizationshadanopenandstrongconfrontationalpoliticalstand.Nazdeek and Pajhra agreed to use more collaborative approaches to engage thegovernment. It was decided by the project that Pajhra led the communicationactivitieswiththegovernmentespeciallyasNazdeekwouldhavebeenperceivedasantagonistic.Thismeantthatanexplicitcommunicationstrategyneededtobebuilttosupportthiswork.Astartingplacewasaudienceanalysis,whichallowedtheteamtoconnectwithgovernmentasafutureaudience,andbeginestablishingabasisforcollaboration.The communicationoutcomes set out included “expect to see”, “like to see”, and“love to see” outcomes. The case in Nazdeek and Pajhra shows a considerabledifferentwiththecaseinOpAshaCambodiaandCookIslands,asdirectlinkbetweenUFEandResComwasnotstrong.Following the communication strategy designed with the guidance from the DECImentors,Nazdeek,PajhraandICAADreleasedthereport,NoTimetoLose:FightingMaternal and InfantMortality throughCommunityReportingand sentout a joint-pressreleaseaboutthereport.Thepressreleasementionedthatcommitmentfromthegovernmentwasgainedwiththeplantoestablish“CitizenGrievanceForum”attheBlock level toaddressmaternalhealthviolations.Ashortvideowasalsomade
19
anddistributedhighlightingstoriesonmaternalmortalitycasesfromthegroundandtestimoniesonhowtheAppisusedasatooltoreportcases.TheNazdeek team reported thatmedia coveragewas good, around 4-5 local andnationalnewspaperscoveredthepressconference, includingTheHindustanTimes,and thekindof coveragewas indeedpositive. More recently,TimeMagazineandtheGuardian,coveredthestoryoftheuseofAppasatoolforcommunityreportingtoreducematernalandinfantmortalityinAssam.A Nazdeek laywer comments: “We've basically been following the strategy wedesigned during our last call … The objective of the communication strategy, (i.e.building a relation with the Government to have regular meetings between localauthoritiesandparticipants)wasmet,andwe'reveryhappyabout that.Nowwe'llneedtounderstandwhetherauthoritieswillactuallycommittoholdingthemeetingsandwhether they will follow up on the individual cases presented by participantsduringthemeetings”.Nazdeekfacilitatedoutreachtomedia,whilePajhratookaleadingroleinprovidinginformationtojournalistsfromTimeMagazineandtheGuardianonwhathavebeenhappeningonthefield.ExecutiveDirectorofPajhratestifiedthatfindingsobtainedfromtheevaluationprocesswereveryusefulasconvincingevidencestoshowtothemedia.ThePajhramenteecametorealizethelinkbetweenUFEandResComwhereresearch-basedevidenceistakenupandusedforcommunicationpurposes.
c)CookIsland’sjourneyOneoftheCookIslandsinitialchallengeswasdiscriminatingbetweenauserandabeneficiary,especiallyastraditionalevaluations looksattheresultsand impactsoftheprojectofthemainbeneficiary.However,oncetheywereabletoidentifyotherUsers, including themselves, there was better understanding about how theevaluationdatacouldbehelpfultotheproject.Theywereabletoleveragethehugeamountofwork thathadgone into thedatabaseproject and the impacts it couldhaveonanumberofusersoutsideofthoseoriginallyidentified.FromtheoutsetoneofthekeyUsersidentifiedwasthegovernment.Thepurposeofthe database project was to build an application that would preserve the CookIslandsMaori language that facesa threatofbecomingextinctdue to lackofuse.ThementeewassurethatatleastonepoliticiancouldbeenlistedastheUser,whowould vouch for theproject as itwas focusingon thedwindlingnumberof fluentMaoriLanguagespeakers.However,thepoliticalsituationremaineduncertain.Thegovernmentwasinitiallypreparingforanelectionandevenayearafterithadbeenrun, therewas indecisionas to its finaloutcome. Thismeant that therewas littlesupport fromMinisterswhowereunsure if theyretainedtheirportfolios.SimilarlywithouttheMinisters,evenSecretaries(theHeadsofMinistries)wereunsureabouttheirpositions.ItthereforemeantthattheCookIslandsgrouphadtolookoutsideofgovernmentforotherUsers.
20
In line with the role of Evaluation Uses in the UFE process, the Purpose ofcommunicationisimportanttodefine.Itthiscase,itwastogetthebuyinfromtheMinistryofCulturalDevelopmentandtheCabinettoallocatesomefundstosupportpreservationofMaoriLanguageusingtheApp;andtohelptheMinistryofCulturalDevelopmenttoconvinceandgetfundingfromthedonorstosupportpreservationofMaori Language. Themain target audience identified included: theSecretaryofthe Ministry of Cultural Development and the Cabinet as their primary targetaudience;andDonorsastheirsecondarytargetaudience.AlthoughtheCook Islandsteamneededtodoaudienceresearch inorder toverifyinformation needed andmedia preferences, some ideas emergedwith regards tocommunicationmaterials, includingvideopresentation that contained testimonialsfromtheusersoftheApp,factsheets,andseriesoftestimonialposters.
Outcomes:useofthefindingsAll three organizations reported benefits and unexpected outcomes from theUFEandResComprocess.
OpAsha’soutcomes1)OpAsha’smentee/user benefited greatly frombeingmentored – she knew thedatasowellthatshecouldposition itasneededtotargetkey influencersandwasable to use her ResCom knowledge and skills for marketing and developingcommunicationmaterials.InvolvingthemedicalofficerasajointUserhasresultedinmore debate and analysis within the organization. The greatest benefit was at apersonal level – in understanding the project better and building a set of skills inevaluationandcommunicationthatcouldnotbeforgotteneasily.“Iwentthroughtheprocess,sothedataispartofme...the team understands how to use evidenceand has built their capacity…they have touched,felt and learned from the data.” (OpAsha,CountryDirector)ThereweresomeunexpectedfindingsfromtheUFEstudy–thenumberofpeoplescreened using the App (with the tablet)was fewer than the conventional paper-and-pencilmethodbecausefieldworkershadtoimmediatelykeyininformationas
well as spend timeexplaining about the tablet and theApp inthe community. This has resulted in plans for communitysensitization.This findingwas important fora smoother scale-up. Other findings were positive as diagnosis was moresystematic, more people were diagnosed and case detectionrateswerebetter.Most important, themedicalrecordkeepingwasaccurateaswasthe followupwithpatients.Geomappingwasanaddedadvantage.
21
OpAshahasused the informationboth internallyandexternally. Internalusehasresulted in combining the e-detection app (which was funded by the ISIF grant)seamlessly with eAlert and eCompliance apps to provide a comprehensivecommunityledTBpreventionandtreatmentservice.ThefindingshavealsohelpedOpAshatomarketitsworkandenabledascale-uptootherdistrictsinCambodia.The data generated through theUFE study is being used as a baselinewithin theorganization.Externaluseofthedataincludesthegovernment’srequesttopresenttheAppanditsfindingsatitsnextTechnicalWorkingGroup(TWG)meetingandthereisinterestin OpAsha’s extending the reach of the App. OpAsha feels confident regardingrepeatingthesurveythattheyappliedthroughtheUFEandaddingonquestionsforitsnextroundofanalysis.OpAsharealizedthroughtheUFEandResCommentoringprocess, that thegovernmentwouldbe interested in theirworkonly if evidencewasavailablefromthepilotphaseandOpAshascaledupandextendeditsreach.OpAsha isnowrealistic in itsexpectations fromthegovernmentandhasamatureunderstandingofhowtoplantheirnextsteps.
Nazdeek’soutcomesNazdeekandPajhrahadbeendocumentingandgeomappinghealthrightsviolationson line without informing the government. The initial plan was to confront thegovernmentwiththeevidence.However,itwasonlywhenadiscussionaboutwhat
neededtobeevaluatedandwhatneededtobecommunicatedtookplacethattheideatobuildpartnershipsandstrengthenadvocacywiththegovernmenttookroot.The UFE gathered information about volunteers profile, enquired about theirmotivationandconstraintsinreporting.Thedataindicatedthatthewomendidnotperceive the violations as against human rights. All their life they had seen suchviolations and had accepted them as routine or their fate. What emerged wasexactlywhatPajhraasahumanrightsorganizationhadbeenfightingforyears–theneed to empower tribal men and women who for generations had experiencedexploitation.Pajhrahad,intheapplicationoftheApp,completelymissedoutWHYthetribalwomenwerenotreportinglapses.Nazdeek and Pajhra collected information regarding the factors affecting theparticipation of volunteers using a guided questionnaire (because of the limitedliteracyof volunteers) and FGDswith volunteers and community.Almost anequalnumberofvolunteerslivedinthevillagesandtheteagardensbutthosewholivedinvillages were the ”high reporters”. Most “High reporters” were students andteachers,asopposedtonothousewivesorgovernmentfieldnursesorgovernmentemployednurseryteachers.Thosewhodidnotreportdidsobecausetheywerenotsure of the code they selected. Surprisingly, the government field nurses did notvolunteer information regarding lapses in the tea gardens whereas those in thevillagesandtownswereabletorespondtodemandforinformation.“Lowreporters”in the tea gardens hadmore informal communicationwithwomenwho accessedhealth services. ASHAs (government field nurses) command respect and haveinfluence in the tea gardens and tend to blame the pregnant (tribal) women forkeepingpoorhealthornottakingthemedicinesandthepregnantwomen.Forexample,Pajhrareportedthatthegovernmentnurseryteacherdidnotreceivenutritionfortheunder5s inhercarefor6months inteagardensandthevillages,but did not report it. In villages, volunteers are more educated and can accessASHAswhowillprovideaccurateinformationonhealthcarelapses.ButthecontextisverydifferentintheteagardenswherethereareculturalbarriersforteagardenworkerstoaccessthehighercasteASHAsandherrecords.Thesurveydataraisedquestionsabouttheinterrelationshipsofthevolunteerswiththe community, health careworkers and the historical structural inequalities thatsubtly impacted the responses for an App that was technologically fine. Theunderlying problem for Pajhrawas the issue of entitlements and how confidentwomenaretoclaimtheirrights.Unlessthiswasresolved,theAppwasunlikelytoreporttherealnumbersofhealthrightsviolations.Another importantoutcomewasthatalthoughtherewasastrongmobileplatformavailabletoreporthealthviolations-anditwascommonknowledgethattherewerehealthviolations-thevolunteerswerenotreportingtheminthenumbersrequired.Pajhrahadassumedthatthevolunteersselectedwereeithernotcompetentornotmotivated or had a conflict of interest (since some were employed by thegovernment as teachers and field nurses) and itwas perhaps necessary to recruit
23
newpersonnel.Infactthefirstinterpretationofthefindingswasthatitwastheteagarden worker women (tribal) who were the least responsive although theirinteractionwithotherwomenwasthehighest.Theplanwastoaskthemtoleave.Thevolunteersneededempowermentsessions–toknowandexercisetheirrightsastribalsandwomenandtobeconfidentinreportinglapsesinhealthcare.Havingminimal education, thewomen also neededmore interactiveways to understandandreportcodesthatrepresentedspecificviolations.Finally,thewomenneededtounderstandtheircollectivestrengthasadvocatesandtoshareandlearnfromeachother. Follow up meetings with the relevant government authorities to redresshealth rights violations needed to be introduced and strengthened, not just thereportingoflapses.Womenneededtoseechangehappening.Perhapsthereportingwas important froma legalperspectivebutcommunityactionanddialoguewiththegovernmentwasthepriorityatthecommunitylevel.
anotheradvantagebecausetheybecamethecontributorstothedevelopmentoftheMaori database. There was overwhelming support for the project and itscontinuation. The target audience identified the predominant use of and theusefulnessoftheonlineaccessofthedatabase.
The User used various avenues to communicate abouttheprojectfindingsvialocalmedia,aregionalITemaillist(PICISOC.org) and taking advantage of requests from ITconference organizers. Presenting the project as theseevents enabled them to extend the project out to asmanyotherpotentialusersaspossible.
The local media, particularly the Cook Islands News reported on the work of theCook Islands Internet Action Group. Themedia has been proactive in providingarticlesthathavepromotedtheworkthathasbeendoneonthisprojectandintheachievementsalongtheway.Thishasraisedtheprofilenotonlyoftheprojectbutalso of the technical partners who have become involved. This has resulted inemploymentopportunitiesthatwereearliernotasforthcoming. TheCookIslandsteamproposestousethesurveyfindingstoputpublicpressureongovernmentviathemediatoallocatemoreresourcesintermsoflanguagepreservation.The Maori Database, Cook Islands were unable to find at least one politicalchampiontosupporttheirwork.Butwithineachofthesectorstheyfoundindividualchampionsfortheircause.AlthoughtheMinistryofEducationdidnotgivethemthesupport,individualteacherseagerlyofferedtoassisttheprojectbyhelpingtouploadthe dictionaries, and then later to produce the classroom teaching and audioresources to be used alongside the database in Maori Language classes of thenational secondary schools. Along the way, they learned more about their ownlanguageandwhyitwasimportanttoproducethistool.TheUser(plusthesoftwareexpertof theCook Islands InternetActionGroup)made theandroid appavailableofflineandfreeforCookIslandsstudents,anditisproposedthatanotherappwillbedevelopedtoutilizetheteachingresourcesmoreeffectively.TheUserandsoftwaredeveloperwasawarded“BestDigitalInnovatorinthePacific”for the database development for promoting a native language, by the Vanuatu
Government–testimonythattheprojecthadsucceededingettingthemessageoutacrosstheregion.The intendedoutcomewas thedevelopment of education resources and enablingteachers to use the database app as a teaching and learning tool. However, anunintentionaloutcomethatemergedwasthatanotherAppisbeingdesignedtousethese teaching resources,whichwill utilize theoriginal database, but canbeusedalongsidetheaudioresourcesasadditionaltoolsforMaoriLanguagelearners.Another intended outcome was to enable government officials to experience thework that the Cook Islands teamwas capable of, and to offer them employment.This occurred due tomedia coverage,which highlighted the achievementsmainlyabroad. An unintended outcome was an increased demand for the softwaredeveloper’s skills, acknowledged by a large regional organization (Pacific IslandsForumFisheriesAgency,FFA)andjoboffersintheSolomonIslands.
AResComuser is the localmedia,particularlytheCook IslandsNewswhichhasbyfarthewidestdistributionandinfluenceofallthemediaoutletsintheCookIslands.Themediahasbeenproactiveinprovidingarticlesthathavepromotedtheworkthathasbeendoneonthis.Thishasraisedtheprofilenotonlyoftheprojectbutofthetechnical partners who have become involved, resulting in employmentopportunitiesthatwereearliernotasforthcoming.
An unexpected proposewas to use the survey findings to put public pressure ongovernment via the media to allocate more resources in terms of languagepreservation.Inthefirst6weeksofitsgoingonsale,theAppleappwasgeneratingfundsmainlyfromNewZealandandAustraliabutalsofromelsewhere.
IntegratingUFEandResComDECI-2 combined UFE and ResCom for the first time as two parallel mentoringstreams. Organizations received opportunities to identify what they wanted toevaluate and in parallel an opportunity towork on their communication strategy.Organizationsmentored for ResComhad the option to design the communicationstrategy, identify theaudienceanddevelopcommunicationmaterials forwhateverpurposetheydeemedimportant.Theyhadtheoptiontochoose(ornot)tocombineResComwithUFE,anddevelopcommunicationsforUFEdissemination.Thementorswere open to both possibilities. As DECI-2 was a research project interested intracing the conditions where organizations would see the ‘natural fit’ of UFE andResCom and had the opportunity to create a coordinated response of USE andINFLUENCEoftheirstudyfindings.Because UFE and ResComwere considered to be parallel processes, thementorsfollowedtheirownmentoringschedulewith therespectiveorganizations.TheUFEmentor scheduled the face-to-facemeetingwithNazdeek independently,whereastheResCommentorprovidedinputslaterandthroughSkypecalls.BecausetheCook
26
Islandsweregeographicallyfarawayandtravelwascomplicated,itwasplannedthatbothmentorswouldvisittogether,butcouldmentorindependently(asforNazdeek).OurexperienceandfeedbackfromourmenteesintheCookIslandsindicatedthatifUFE‘led’theprocess,theResComstepsfittedperfectlywithintheoverallevaluationplansoftheorganizations.IdentifyingUSE,USERSandKEQsprovidedanentrypointforsituationalanalysis,whichisanimportantstepinResearchCommunication.ThementeescouldseetheconnectionbetweenUFEandResComandtheirplanningforboth was simpler, quicker and relevant. It was a win-win for both mentors andmentees. This idea of twinningmentoring of UFE and ResComwas replicated forOpAsha with the same success. The OpAsha mentee reported that UFE withoutResComand vice versawas ‘fluff’ and had no substance in application. Lessonslearnedfromthisexperiencehave ledtothedevelopmentofahybridapproachtointegrateUFEandResCom.This graphic below summarises how the Maori Database project interpreted thecoordinated impact of UFE and ResCom. Their communication strategy involvingmediaandconferencepresentationshadraisedawarenessanduserswerestartingtopurchase theappsonline.At the same time, it raisedawarenessof the skillsofoneofthelocaltalents.TheUserandsoftwaredeveloperwasacknowledged:hehasbeensnowedunderwithworkoflate,andreceivedanofferofajobintheSolomonIslandstowherehewillbegoingveryshortly.DuringtheISIFgrant,thesamepersonhad lost his job and his skills were not recognized. The graphic explains how thedatabase will use principles of ResCom and UFE to be a sustainable resourceprovidingincometostrengthenworkonthedatabaseinthefuture.
Questions,openeddoors for course-correction ineachof the threeprojects.Adiscussion onwhat the projects were trying to accomplish and how, enabledoutcomestobearticulatedclearlyleadingtoaninformalprojectaudit.
2) Complementary face-to-face & remote mentoring: Webinars were meant to
build capacitiesusing low-cost technologywith the recorded sessions availablefor refreshing and clarifying knowledge. Thewebinarswere planned in such away thateach teamhad todo somehomework thatwouldbuildonwhatwaslearnedineachwebinarandbridgetothenextwebinar. Itwashopedthatthisgraded,blendedlearningwouldprovideanimpetustotheapplicationofUFEandResCom concepts. We learned that while the webinars were used to refreshconcepts,andPowerPointsslideswereusedtosensitizeimportantstakeholders,they could not substitute for the dialogue and discussion needed in a face-to-facemeetingthathelpedtoclarifydoubtsofbothmentorsandmentees.Forthementors,themostimportantadvantageobtainedfromtheface-to-facemeetingwas an understanding of the context in which the projects worked and anassessmentofthestaffcapacityavailableintheorganization.Bothwerecriticaltoexecute theUFEandResComsince capacitieshad tobebuilt internally andusewaspossibleonlywithinacertaincontext.
inparallel,buttheyappearedtobebestunderstoodwhencapacitieswerebuiltintandem.Inotherwords,havingworkedontheUFEUSERS,USESandKEQs–itwasverynaturalandeasyfororganizationstodevelopacommunicationstrategy,identifytheaudienceandpurposeofthecommunicationmaterials.Theprocesswas smoother and less intimidating when mentors worked together insupporting the mentees in UFE and ResCom. The synergy between UFE andResComenhancedthevalueofboth.Asonementeeputit–“YoucannotdoUFEwithout ResCom and you cannot do ResCom without UFE.” The timing wasimportant,illustratedthroughanexperiencewithanorganizationwhereResComsupportcamemuchlaterthanUFE.Theorganizationcouldnotdecidewhattheywantedtoworkonandsuchanexperiencecanbefrustrating.
4) Evaluation USERs with many hats: While working with this grassroots
organization, it became evident that there was limited staff capacity and thattheyplayedmultipleroles.Also,managerialstaffwasleanandhencetherewereveryfewoptionsforaddingevaluatoranduserresponsibilities.Insuchcases,thepractice of separating User and Mentee/evaluator/communication specialistroleswasnotpossible.
6) Just-in-time mentoring: Mentoring based upon the need, pacing it so that
organizations learn when they are ready was critical in embedding andexperientially learningUFEandResCom in this case.This typeof individualizedmentoring capacity building was very different from the workshop approachwhereprimarilyskillbuildinghasbeenprovided.
and facilitate the UFE and ResCom process. The process is entirely driven byorganizationsandalthoughattractiveintheory,itcanbeintimidatinginpractice.Therearenogoalposts.Mentorskillsare importanttounderstandthecontextand political environment in which grassroots organizations work. ThisunderstandinginturnguidesorganizationstoimplementUFEorResComthatispractical,relevantandcompletedwithinadesignatedperiod,saysixmonths.
8) Building capacities and demystifying: The mentoring that was offered built
institutional capacity and commitment. Organizations expressed readiness,selected evaluators or communicators to receive thementoring; and providedtime, resources and personnel to complete the tasks. Thementoring not onlybuiltskillsinUFEandResCom,butdemystifiedbothandopenedthedoortouseevidence for learningandevaluation.Theprocessbuilt institutional confidenceandenhancedevaluativethinking.Organizationswhowerementoredwereabletousetheevidenceconstructivelyandidentifiednewareasofworkthatwouldbenefitfromfocuseddatagatheringandanalysis.
capacitybuildingforevaluationandcommunicationskills,but itdidrequiretheinvestmentof timeand resources. Inexpensivemeansor shortcuts tocapacitybuilding,however,donotgivethesameresults.Thementoringprocesswascosteffective in that only one face-to-facemeetingwas scheduled, and the resultswere a useful evaluation and communication product, aswell as a number ofnewskillslearned.
