MyChild Solution in Afghanistan: An External Evaluation - Transfer of Work Processes to Existing Health System
Timothy Anderson
Tall inn University, 2018
2
Table of Contents
Glossary&Abbreviations.....................................................................................................................................3
Abstract .............................................................................................................................................................4
Introduction.........................................................................................................................................................5
Background..............................................................................................................................................................5
ResearchQuestions..................................................................................................................................................5
EvaluationStructureandFramework.......................................................................................................................5
Methods..............................................................................................................................................................9
StudySetting&Design.............................................................................................................................................9
StudyParticipants....................................................................................................................................................9
DataCollection.......................................................................................................................................................10
DataAnalysis..........................................................................................................................................................10EthicalConsiderations............................................................................................................................................11
Results...............................................................................................................................................................12
Conclusion.........................................................................................................................................................20
Limitations.........................................................................................................................................................20
References.........................................................................................................................................................21
3
Glossary & Abbreviations
Fixedsession:Animmunisationsessionthatoccurswithinahealthcentre/clinic.
Outreachsession:Animmunisationsessionthatoccursoutsideofahealthcentre/clinic.Duringoutreachsessions,healthworkerswillgenerallytraveltoruralorunderservedpartsofthedistrict.
EPI:TheExpandedProgrammeon Immunisationdevelopedby theWorldHealthOrganisation in 1974withthegoalofreducingchildmorbidityandmortalitybymakingimmunisationservicesavailableforallchildren.
EPIDashboard:Anonlineplatformwhereclinics reportdetailsand indicators fromtheir immunisationsessionsarevisualised.Allplanned,held,andmissedsessionsaredisplayedontheEPIDashboard.OnlydataforclinicsthatareusingMyChildSolutionisdisplayedintheEPIDashboard.
MCC:MyChildCard,thepaperhome-basedchildhealthcardusedwithinMyChildSolution.
SPF:Smart Paper Forms are the forms/vouchers are used to capture health information. Theseforms/voucherscanbescannedanddigitised.
SCA: The Swedish Committee for Afghanistan, a non-governmental organization responsible forhealthcareprovisioninLaghmanProvince,Afghanistan.
4
Abstract
Background:ShifohasrecentlyimplementedMyChildSolution,ahealthinformationmanagementsystemdesignedforchildhealth,withinMehterlamDistrictinAfghanistan.Thisreport,writtenbyanexternalevaluator,willprovideananalysisofwhich,andtowhatextent,workprocessesrelatedtoMyChildSolutionarebeingtransferredtospecificfacilityandmunicipallevelsofthehealthsystemstructuresinAfghanistan.
Objective:ThisevaluationfirstoutlinestheStandardOperatingProceduresneededforthetransferofMyChildSolutiontoMehterlamDistrict.Thisstudythenseekstodiscoverwhatessentialprocesseshavebeentransferred,andtoassesstheaccuracyandsustainabilityofthesetransferredprocesses.
Methods:Thisisaqualitativestudy,focusedonsemi-structuredinterviewsconductedwithhealthworkersatseveralclinicswithintheMehterlamDistrict.Interviewsareanalysedusinganarrativeanalysisapproach.Tosupportinterviews,researchaccesswasgrantedtosupportingdocumentsfromShifo.Thesedocumentsincludereceipts,printingorders,stockcards,datacompletenessreports,andverificationlogs.
Results:TheresultsofthisexternalevaluationindicatethatasignificantportionofprocessesessentialtoMyChildSolution-95%-havebeentransferredtothelocallevelinMehterlamDistrict.Inmostcaseswheretheseprocesseshavebeentransferred,theyarebeingperformedinawaythatisbothaccurateandsustainable.AllremainingprocessesareplannedtobefullytransferredtoMehterlambytheendof2018.
5
Introduction MyChild Solution, a health informationmanagement system designed for child health, developed byShifoFoundation,hasbeenimplementedandevaluatedinAfghanistan(MehterlamDistrict).Onecriticalstep to fully integratingMyChildSolutionwithin thecurrenthealthsystemstructure is the transferofessential tasks and routines to the local level. After training health workers in this new healthdelivery/data collection system, project’s goal is to gradually transfer processes connected with theinterventionandallowMyChildSolutiontobesustainableatthefacilityanddistrictlevels.Thisreport,writtenbyanexternalevaluator,willprovideananalysisofwhich,andtowhatextent,workprocessesrelated to MyChild Solution are being transferred to specific facility and district levelsof the healthsystemstructuresinAfghanistan.
Background MyChildSolutionwas introduced inAfghanistan inOctober2016withinMehterlamDistrict, located inLaghman Province. MyChild Solution was designed as a replacement for the previous healthmanagement information system (HMIS), whichwas primarily paper-based and involved a significantburden of administrative paperwork for health workers.MyChild Solution was intended to ease thisburden of paperwork, combining a paper child health card (MyChild Card) and a digital componentbasedonSmartPaperTechnology.
HealthworkersenterinformationabouttheservicesprovidedtoeachchildonSmartPapervouchersinMyChildCards,justasitwouldbenormallydonewithregularpaperusingballpointpens.Attheendofthe immunisation session, the nurse compiles all the vouchers from the day into a 'session bundle'.Followingthis, thevouchersarescannedatthescanningstationsituatedatMehterlamDistrict,whichhaselectricityandinternetconnectivity.Thedataisuploadedontoasecureserverwhereanelectronicmedicalrecordiscreatedforeachchild(Andersonetal2017).
While the introductionofMyChild Solutionhasbeen shown to reduce administration time for healthworkers,andhasreceivedpositivefeedback(seeAndersonetal2017), there isaneedtoensurethatthisnewhealthinformationmanagementsystemhasbeenimplementedproperlyandsustainably.
Research Questions 1.WhataretherequiredworkprocessesassociatedwithoperationsofMyChildSolutioninMehterlamDistrict(Afghanistan)?
2.InAfghanistan,howmanyoftherequiredworkprocessesforMyChildSolutionhavebeensuccessfullytransitionedtothelocalhealthsystemstructures?
3.Towhatdegreeare theseprocessesbeingperformed inamanner that is accurateand sustainablewithinthelocalhealthsystem?
Evaluation Structure and Framework Themajorprocessesandsub-processes listedherehavebeenadapted fromadocumentonStandardOperating Procedures provided by Shifo. The Standard Operating Procedures document details thestandardoperatingproceduresforMyChildSolution,includingalltheworkprocessesthatareessentialto the full functioning ofMyChild Solution. Each of these tasks has been categorized into five broad
6
types(ormajorprocesses).Thetablesbelowlistthemajorprocesses(andtheirrelevantsub-processes)thatneedtobeverifiedinAfghanistan.
1.Majorprocess:Procurementandsupplymanagement
Thismajorprocess involves thenegotiation,purchase,andconsistent supplyofallmaterialsused forMyChildSolution.
Table1
Sub-processes
ProcurementofMyChildCards/SmartPaperForms(MCC/SPF)
Purchasinginternetservices
Purchasingequipment(scanners)
SupplyandstockmanagementofMyChildCards(MCC)/SmartPaperForms(SPF)atdistrictlevel
SupplyandstockmanagementofMCC/SPFathealthcentrelevel
2.Majorprocess:Dataentry
Thismajorprocessinvolvesthedailyentryofpatient(child)dataontoMyChildCardsandSmartPaperForms,aswellastheirdeliverytoscanningstations.Dataentryismostoftendonebyavaccinator.
Table2
Sub-processes
DataentryusingMCC/SPF
DeliveringMCCVouchers/SPFtodesignatedscanningstations
Continuousfeedbackandimprovementsondataentrytools
7
3.Majorprocess:Scanningoperations
AfterdataentryandthedeliveryofSmartPaperForms,theFormsneedtobescannedandarchivedatscanningstationsatthedistrictlevel.
Table3
Sub-processes
ReceivingSmartPaperFormsandvouchersfromhealthcentres
ScanningandarchivingSmartPaperFormsandvouchers
Operationalmaintenanceofrequiredscanningstations
4.Majorprocess:Dataprocessing
Afterbeingscanned,patientdata fromSmartPaperForms isprocessedandsaveddigitallywithin thelocal health system.While someof thismajor process is automatic, there ismanualwork needed toverifyanydatathatcouldnotbeautomaticallyrecognisedduetopoorhandwriting.
Table4
Sub-processes
Automaticimageprocessingofhandwrittentextandcheckmarks
Dataverification
Reportingontechnicalerrorsandsystemimprovement
5.Majorprocess:Monitoringandevaluation
Thismajorprocess involvesmonitoringofdataqualityusingarangeof indicatorsandreportsthatareproduced by the system. This process ensures that reports are complete and timely. Any errors orinconsistenciescanbedetected,andfeedbackisprovidedtothehealthworkers.
8
Table5
Sub-processes
Ensuringdatacompleteness
Timelyreporting
Analysingdatarecordingerrors
Printingandhandingoverpaper-basedreportstohealthcentres
Findingpersonaldataofachildfromtheelectronicregisterwhenparentslosechildhealthcards
ProvidingfacilitybasedKeyPerformanceIndicatorsviaSMStohealthworkers
9
Methods Study Setting & Design This external evaluation is conducted for clinics inMehterlamDistrict, Afghanistan.MyChild Solutioncovers141healthservicedeliverypoints inMehterlam,amongwhich,thereare8fixedhealthservicedeliverypoints,105outreachhealthservicedeliverypointsand28mobilehealthservicedeliverypoints.
This is a qualitative study, focused on semi-structured interviews conducted with health workers atseveral clinics within theMehterlam District. To support interviews, research access was granted tosupportingdocumentsfromShifo.Thesedocumentsincludereceipts,printingorders,invoices,monthlyreporting forms, data completeness reports, and verification logs.In addition, supportingmedia files(videosandphotosofdailyoperationwithinclinics)wereprovidedbySCA.
Study Participants Thefirstpointofcontact isaMyChildSolution(MCS)SeniorOfficerworkinginSCAMehterlamLiaisonOffice.MCSSeniorOfficerresponsiblefor immunisation informationmanagementandcoordinationattheprovincial level for facilities thatuseMyChild system.Afteran initial SkypecallonApril23rd, theevaluator receivedcontactsof studyparticipantsthatcouldprovidedetailsofhoweachworkprocesshasbeentransferredandisbeingperformedatthelocallevel.Toprotecttheirprivacy,allparticipantsarereferredtobytheirtitlesratherthantheirnames.Table6belowprovidesalistofthesecontacts.
Allhealthworkersselectedforthisstudysatisfythefollowingcriteria:
§ They are recommended by an MCS Senior Officer as representative of MyChild SolutionprocesseswithinMehterlamDistrict;AND
§ TheyareresponsibleforMyChildSolutionand/orimmunisationsuppliesinMehterlamDistrict;OR
§ Theyhavebeenengagedinroutineimmunisationatahealthcentreforatleast1month
Table6
MajorProcess Relevant ContactPerson(s)
Clinicplacement Interviewdate
Procurement and supplymanagement
MCSSeniorOfficer Mehterlam LiaisonOffice
April23rd
Dataentry Vaccinator(1)
Vaccinator(2)
Vaccinator(3)
Vaccinator(4)
MehterlamCHC(1,2)
BabasahibBHC(3)
ShamatiBHC(4)
May9th(1)
May9th(2)
May16th(3)
May16th(4)
10
Scanning MCSSeniorOfficer Mehterlam LiaisonOffice
April23rd
Dataprocessing MCSSeniorOfficer Mehterlam LiaisonOffice
April23rd
Monitoring andevaluation
MCSSeniorOfficer Mehterlam LiaisonOffice
April23rd
Data Collection Giventhe inability to travel toAfghanistan (duetosecurity limitations) for interviewsandparticipant-observation, semi-structured interviews were conducted over Skype. Open-ended questions (seesamples below) were asked regarding each work process from the respective health workers, withspecific attention paid to where there have been inconsistencies or challenges encountered in thetransferprocessofMyChildSolution. In somecases, intervieweeswereasked tovisuallydemonstratehowparticularprocessesareperformed(e.g.,fordataentry).Allconversationswererecordedonvideotomaketranscriptionandsubsequentanalysismorereliable.Eachinterviewrangedfrom20minutesto70minutes,withanaveragetimeofapproximately40minutes.AllinterviewswereconductedinEnglishintheMehterlamDistrictOffice(wheretherewasthemostconsistentinternetconnectionavailable).
Open-endedinterviewquestionsthatwereaskedtoallparticipantsinclude:
Whatisyourrolewithintheclinic/officeandwithinMyChildSolution?
Howlonghaveyoubeenworkingwithinyourrole?
Whatdaily/weekly/monthlytasksareyouresponsiblefor?
Doyouhaveanyfeedbackontheprocessesyouworkwith?
HavetherebeenanyproblemsadaptingtoMyChildSolution?
Ifneeded,wouldyoubeabletoteachanewhealthworkerthestepsyouperform?
Data Analysis InterviewswereconstructedwiththeintentionofbothconfirmingtheprocessescarriedoutbystaffinMehterlam and discovering if these tasks could be passed on or taught to potential incomingstaff.Regardingtheresearchquestions,accurateandsustainableprocessesaredefinedasthefollowing:
1. Accuracy:Processesareperformed inamannerthatencompassesallstepsandsub-processesessential for the functioningofMyChild Solution (seeEvaluation Structure and Frameworkformoredetails).
2. Sustainability: Processes are well-understood by health workers and receive consistentfeedback.Stafffeelcomfortableperformingtheseprocessesontheirownandteachingthemtoothers.
11
Regardingaccuracy, the aim is to determine the extent to which each major process has beentransferredtothelocalhealthsystemstructureinMehterlam.Withintheresults,foreachmajorprocessand sub-process, a percentage is assigned. This percentage indicates whether the process has beenevaluated as fully transferred (100%), not transferred at all (0%), or partially transferred. Ideally, allprocesses under evaluation should be fully transferred toMehterlamDistrict; 100% is the goal in allcategories.
Regardingsustainability, staff were asked questions to ascertain how well they understand MyChildSolutionandhowcomfortabletheywouldfeelexplaining/teachingtheirroletoauditorsandnewstaff.
Interviews were analysed using a narrative analysis approach (see Chase 2005). Each interview wasviewedmultipletimes,astheevaluator lookedforthe‘story’ofthe interviewee’sdailyroutine.Thesenarratives, taken together, were used to give a holistic picture of howMyChild Solution operates inMehterlamonadailybasis.WithintheResults,theevaluatorhasaimedtorepresenttheintervieweeintheirownwords,usingthesameterminologythattheyemployintheirdescriptions.Timestampswereusedineachvideotocreatereferencepointsthatcouldbeeasilyreturnedtoduringtheanalysis.
Ethical Considerations Priortotheinterviews,allobjectivesofthisevaluationwereexplainedtoparticipants.Allintervieweeswereinformedthatparticipationinthisevaluationwasvoluntary.Theywerealsoassuredthattheyhadtherighttowithdraworrefusetoanswerquestionsatanystageoftheinterview.Recordedinterviews,transcriptions, andnoteswereonlyaccessible to theexternalevaluator. Toensure confidentiality,nospecific details of financial transactions (e.g., from receipts or order transcripts) have been includedhere.
12
Results Theresultsofthisevaluation,includingabreakdownofthetransitionstatusforeverymajorprocessandsubprocess,areavailableinaseriesoftablesbelow.Theevaluator'sunderstandingoftheaccuracyandsustainabilityofeachmajorprocessisalsoincludedinthetextbeloweachtable.Thedescriptionsofthesub-processesalsoserveasananswertothefirstresearchquestioninthistext:whataretherequiredworkprocessesassociatedwithoperationsofMyChild Solution inMehterlamDistrict (Afghanistan)?Amoregeneralsummaryoftheresultsisincludedattheendofthissection.
1. Procurement & Supply Management (100% transferred) Table7
Process Description TransitionStatus SupportingDocument(s)
Results
ProcurementofMCC/SPF
NegotiatingandcontractingprintingcompaniestosupplyMyChildCardsandSmartPaperFormstoclinics
Transferred(100%) ReceiptsfromtheKabulofficeoftheSCAconfirmingpurchaseofMCC/SPFthroughout2017
Supportingdocumentsindicatesuccessfultransfer
Purchasinginternetservices
Negotiatingandcontractinganinternetserviceproviderforscanningstations
Transferred(100%) AcontractbetweentheSCAandAfghanTelecomconfirminginternetservicesinMehterlamDistrictOffice
Supportingdocumentsindicatesuccessfultransfer
Purchasingreplacementequipment(scanners)
NegotiatingandcontractingSmartPaperscannersandreplacementrollers
N/A N/A Scannersareestimatedtolastfor7years,whilerollersneedtobereplacedannually.
SupplyandstockmanagementofMCC/SPFsatdistrictlevel
MonitoringstockandsecuringdeliveriesofMyChildCardsandSmartPaperformstoindividualclinics
Transferred(100%) ReceiptsfromtheKabulofficeoftheSCAconfirmingpurchaseanddeliveryofMCC/SPF
Supportingdocumentsandinterviewindicatesuccessfultransfer
13
throughout2017
SupplyandstockmanagementofMCC/SPFsathealthcentrelevel
MonitoringstockandsecuringdeliveriesofMyChildCardsandSmartPaperformstoindividualclinics
Transferred(100%) ReceiptsfromtheKabulofficeoftheSCAconfirmingpurchaseanddeliveryofMCC/SPFthroughout2017
Supportingdocumentsandinterviewindicatesuccessfultransfer
Healthcare inMehterlamDistrict,Afghanistan is deliveredby the SwedishCommittee forAfghanistan(SCA), which has its head office in Kabul. The Kabul office is responsible for several aspects ofprocurement for Mehterlam, namely printing of MCC/SPF, distribution of MCC/SPF, and purchasinginternet services. TheMCS SeniorOfficer is an employeeof the SCA, primarily placed at the regional(Laghman) and district-level (SCA Mehterlam liaison/coordination) offices, where he is responsibleforimmunisation informationmanagement and coordination in facilities that useMyChild Solution attheProvinciallevel.
At the time, he is not facing any printing shortages, but the printing company tends to be latewithsupplies.Thecompanyoftenclaimstheywillsupplyitinonemonth,butoftendeliveriescantakeupto3months.However,evenwiththesedelays,stock isstill sufficient,andthecardsareprintedwell.HeputsvouchersandcardsinStockCards,suppliesaredistributedtohealthfacilitiesonceamonth.Healthfacilities are using them properly, and these facilities contact the MCS Senior Officer when moresuppliesareneeded.
2. Data entry (100% transferred)
Table8
Process Description Transitionstatus SupportingDocument(s)
Results
DataentryusingMCC/SPF
RecordingpersonaldetailsofchildrenontoMyChildCards;recordingdailyandmonthlyvaccineuse
Transferred(100%) Datacompletenessreportsgeneratedbythesystem;linkstoEPIDashboard
Supportingdocumentsandinterviewindicatesuccessfultransfer
DeliveringMCCVouchers/SPFto
Bringingsmartpaperformsandvouchers
Transferred(100%) Datacompletenessreportsgenerated
Supportingdocumentsand
14
designatedscanningstations
forscanningtothescanningstationattheendofthemonth
bythesystem;linkstoEPIDashboard
interviewindicatesuccessfultransfer
Continuousfeedbackandimprovementsondataentrytools
SendingfeedbacktoShiforegardingtheuseofMyChildCardsandSmartPaperForms
Transferred(100%) N/A Interviewindicatessuccessfultransfer
Dataentry inMehterlam is completedby vaccinators. For this evaluation, interviewswere conductedwith4vaccinators fromthreedifferentclinics.Thesevaccinators fill in registration,givevaccines,andattendbothfixedandoutreachsessions.Vaccinatorsshowedhowtheyfilledouteachpaper(proofofID,registrationpaper,parents’names,dateofbirth,etc).Registrationvouchersarethenputinthedatabox and kept for one week. Feedback from the health workers is very positive; the burden ofadministration is greatly reduced in comparison to theprevioushealth information system. Individual(5-10min)andgroupcounselling(30min)tasksareperformed.
However, the vaccinators had also encountered several challenges within the data entry processes.Primarily,mothersaresometimesafraidtogivemobilenumbersorhusbands’workinformationoutforsecurity reasons. Sometimes parents also do not receive SMS due to poormobile network coverage.Anotherproblemisthat,becauseregistrationinformationisconfidentialandonlyavailabletotheMCSSeniorOfficer, patient data cannot always be retrieved by vaccinators themselves. If theMCS SeniorOfficerisnotintheoffice(e.g.,onSaturdays),thensometimespatientswhohavelostachildhealthcardcannotbehelped.This lackofcoordinationinworkinghoursbetweenvaccinatorsandtheMCSSeniorOfficer can occasionally be frustrating for the vaccinators. Often, mothers must travel a long waythroughdangerousareastogettotheclinic-itisthereforeveryproblematiciftheirvisittotheclinicisnotsuccessful.
3. Scanning operations (100% transferred)
Table9
Process Description Transitionstatus SupportingDocument(s)
Results
ReceivingSmartPaperFormsandvouchersfromhealthcentres
ReceivingandorganizingMyChildCardvouchersforscanning
Transferred(100%) DatacompletenessreportsgeneratedbyShifo;linkstoEPIDashboard
Interviewandsupportingdocumentsindicatesuccessful
15
transfer
ScanningandarchivingSmartPaperFormsandvouchers
RunningSmartPaperFormsthroughthescannerandensuringthattheirfilesarereceivedandprocessedbytheofficelaptop
Transferred(100%) Datacompletenessreportsgeneratedbythesystem;linkstoEPIDashboard
Interviewandsupportingdocumentsindicatesuccessfultransfer
Operationalmaintenanceofrequiredscanningstations
Cleaningandchangingscannerrollersonceayear;clearingpaperjams;ensuringlaptopfunctionality;internetmaintenance
Transferred(100%) N/A Interviewindicatessuccessfultransfer
The MCS Senior Officer is responsible for all scanning, and he receives session vouchers from eachfacility at the end of each week. Once a week, the MCS Senior Officer does voucher scanning,synchronizing,andverificationofalldata.Theprocessissosimplethathecancoverallhealthfacilitiesbyhimself.Afteroneweek,theyarewrappedwitharubberbandandtheMCSSeniorOfficercollectsthem.Hebringsthemintohisofficeandscansallofthem.
Regardingequipment,twoscannershavebeenreceivedfromtheMyChildproject.As longastheyaremaintainedproperly(e.g.,replacingrollersannually),thesescannersshouldworkforupto7years.Shifodeliversnewpartsforthescanners.
4. Data processing (83% transferred)
Table10
Process Description Transitionstatus SupportingDocument(s)
Results
Automaticimageprocessingofhandwrittentextandcheckmarks
Handledbysoftware(SmartPaperTechnologyEngine)
Transferred(automatic)(100%)
N/A ThisishandledautomaticallybySmartPaperTechnologyEngine
16
Dataverification Checkingandcorrectingforflaggederrors/marks/badhandwritinginvouchersscannedbySmartPaperTechnologyEngine.Thisinvolvestwosteps:primaryverificationandmasterverification
Partiallytransferred(50%)
VerificationlogsfromOctober2017-present
PrimarydataverificationisdonebytheMCSSeniorOfficer,whichwasconfirmedviainterviewandsupportingdocuments.However,masterverification(asecondstepwhereanyfurtherflaggederrorsarecorrected)isplannedtobetransferredbyendofyear2018.Atpresent,masterverificationisstillperformedbyShifostaff,inordertoanalyseerrorsandimprovemachinelearningalgorithmsandvalidationrules.
Reportingontechnicalerrorsandsystemimprovement
SendingfeedbacktoShiforegardinganytechnicalproblemswithsoftwareandsuggestionsforimprovement
Transferred(100%) N/A InterviewwiththeMCSSeniorOfficerindicatessuccessfultransfer
Afterscanning, theMCSSeniorOfficerperformstheprocessofprimarydataverification,checking fordataentryerrorsinMyChildCards.AnextensiveverificationlogfileconfirmsthattheMCSSeniorOfficerhasbeenprocessingdocumentsandverifying fields flagged for reviewsinceOctober2017.Hehasnocomplaintswiththeprocessandisveryfamiliarandcomfortablewithdataverification.Heisconfidenthecouldteachthesestepstoanyincominghealthworker.
17
TheonlycomplaintbroughtupbytheMCSSeniorOfficerwaswiththeslowinternetconnectioninhisdistrict office – it takes a long time for scanned vouchers to synchronize. Some tasks are delayed orencounterproblemsduetothispoorinternetconnectivity.Newinternetoptionswouldbeideal.
Dataverificationisonlyhalfwaytransferred(50%),sincethecrucialsecondstepofMasterVerificationisstilldonebyShifo.However,MasterVerificationisoncoursetobetransferredtoMehterlamDistrictbytheendofyear2018,atwhichpointdataverificationshouldbe100%transferred.
5. Monitoring and evaluation (90% transferred)
Table11
Process Description Transitionstatus SupportingDocument(s)
Results
Ensuringdatacompleteness
Measuredbycomparingthenumberofimmunisationsessions(bothfixedandoutreach)capturedinelectronicreportstothenumberofsessionsthatwereplannedtobeheldaccordingtofacilityschedules.Inthepossiblecaseofamissedsession,theresponsiblepersonmustcallthehealthfacilityandclarifywhetherthesessiontookplaceorwhyitwascancelledormissed.
Transferred(100%) AccesstoEPIDashboardanddatacompletenessreportsshowingthatmissingdataforfixed/outreachsessionsarebeingnotedandreported
Supportingdocumentsandinterviewindicatesuccessfultransfer
Timelyreporting EnsuringthatallelectronicHMISreportsareavailabletothedistricthealthofficesbythe5thofeverymonth.TheEPIDashboardandemail
Transferred(100%)
LogsconfirmingthatreportsfromMehterlamhavebeenreportedontime
18
areusedasplatformsforsharingreports.
Analysingdatarecordingerrors
Ensuringthatanyrecordingerrorsmadebyhealthworkersareanalysedandthatanyrelatedfeedbackispassedontohealthworkers
Transferred(100%)
Accesstoanoperationsissueregisterwhichconfirmsthatrecordingerrorsarebeingloggedandanalysed
Supportingdocumentsandinterviewindicatesuccessfultransfer
Printingandhandingoverpaper-basedreportstohealthcentres
Afterelectronicreportsaregenerated,thereportsareprintedandsharedwithfacilitiesthatdonothaveaccesstoelectronicreports
Transferred(100%) N/A Interviewindicatessuccessfultransfer
Extractingpersonaldataofaclientfromelectronicregisterwhenparentslosechildhealthcards
Properlyretrievinganyneededpersonaldatafromtheclinic'ssystemwhenaparent'sMyChildCardhasbeenlost
Transferred(100%) N/A Interviewindicatessuccessfultransfer
ProvidingfacilitybasedKeyPerformanceIndicatorsviaSMStohealthworkers
ThesystemcangeneratenewKeyPerformanceIndicators(KPIs)thatfacilitatemonitoringandevaluationofEPIperformanceatthefacilitylevel.KPIsaresentbySMStohealthworkersmonthly.KPIsareplottedandfolloweduponthemonitoringboardsbyhealthworkerseverymonth.
Partiallytransferred(38%)
N/A InterviewdataandcommunicationwithShifoindicatethatthissubprocessiscurrentlybeingpilotedin2clinicsoutofthe8fixedfacilities(38%)totallyoperatinginMehterlamDistrict
19
TheMCSSeniorOfficergoestohealthfacilitieseverymonthtosupervisetheMyChildworkprocess.Hesharesactivities,teachesthemhowtofillincards,detachvouchers,howtoadministervaccines,etc.Heis very comfortable training new staff. The MCS Senior Officer also checks monthly activity reports,registry reports, follow-up lists, and tally reports for any discrepancies. Any findings are sharedwithhealthworkers.
In addition to these steps, the MCS Senior Officer analyses monthly reports, checks coverage of allvaccines,andevaluateswastage.Someissuesarerelatedtohealthworkererrorandothers(rarely)tosystemerror.SystemerrorsaresharedwithShifo.TheMCSSeniorOfficerandShifoare incontactatleastonceaweektodiscussfeedbackandanyproblemsthatmayhavecomeup.
Otherproblemsaresharedwiththehealthworkersthemselves.Otherwise,allactivitiesarerecordedinEPI Dashboard. All fixed and outreach sessions conducted per week are recorded. Any missed orcancelledsessionsarementioned.Theseprocessesareallgoingwell.Itisasignificantimprovementtotheprevioussystem,wheremonthlyevaluationwasmuchmoretediousandtime-consumingandhadtobedonemanually.Allreportscannowbegeneratedelectronically.
Overall Transfer In total, this study estimates that 95% of processes essential for the proper functioning of MyChildSolution have been transferred to the existing health system structure in Mehterlam District inAfghanistan(seeTable12).
Table12
MajorProcess PercentageTransferred
Procurement&supplymanagement 100%
Dataentry 100%
Scanningoperations 100%
Dataprocessing 83%
Monitoring&evaluation 90%
TotalforMyChildSolution 95%
20
Conclusion The results of this external evaluation indicate that a significant portion of processes essential toMyChildSolution -95%-havebeentransferredtothe local level inMehterlamDistrict. Inmostcaseswheretheseprocesseshavebeentransferred,theyarebeingperformedinawaythatisbothaccurateand sustainable. All interviewees stressed that theywere comfortable in their roles, appreciated theeaseandconvenienceofworkingwithMyChildSolution,andexpressedtheirabilitytoteachincomingstaff these processes if needed. Only two sub-processes -Data verification and Providing KeyPerformanceIndicators-werenotentirelytransferredtoMehterlamDistrict.AllthesesubprocessesareplannedtobefullytransferredtolocallevelinMehterlamDistrictbytheendof2018.
Limitations Despite the generally positive trend of these findings, there are several limitations of this study thatshouldbeconsideredbyreaders.Theseare:
1. Duetosecurityconstraints,all interviewswereconductedthroughSkypeandSlack.Theabilitytoobserveworkprocesses inactionwasgreatlydiminishedoveravideocall (ascompared tobeingon-site),anditwasoccasionallydifficulttogetasenseofhowMyChildSolutionhadbeenintegratedwithinday-to-dayactionsandroutines.
2. Challengesincommunicationandtranslation.AlthoughallinterviewswereconductedinEnglish,in most cases, English was not the native language of interviewees. In all interviews withvaccinators, theMCS Senior Officer had to act as a translator. This opens the possibility formisunderstandingsandmistranslationsthatmighthaveaffectedresults.
3. The lownumberof interviewees.Outof 8 fixed facilities inMehterlamDistrict usingMyChildSolution,theevaluatorwasonlyableto interviewhealthworkersfrom3fixedfacilities.Whilenot ideal, time constraintsmade it difficult to conduct amore expansive study and acquire amorerepresentativesampleofhealthworkersfromacrossMehterlam.
Theselimitationshighlighttheneedforafurtheron-siteevaluationofworkprocesstransitionsatalaterdatein2020or2021,whenthesolutionisscaleduptothewholeLaghmanProvince.
21
References Chase,SE.(2005)."NarrativeInquiry:Multiplelenses,approachesandvoices."InNKDenzin&YSLincoln(Eds)The SageHandbookofQualitative Research(3rd Edition) pp. 651-679. ThousandOaks, CA: SagePublications.
Anderson,T.,Eshonkhojaeva,S.,Safi,H.,Hamidi,S.,Mikaelsson,M.(2017).“ImprovingChildHealthcarein Low Resource Settings: A Pilot Study of MyChild Solution in Afghanistan”. Shifo.https://shifo.org/doc/MyChildEfficiencyEvaluationAfghanistan.pdf/