Evaluationof2019EvaluationLabWorkshopMelissaBinder,EvaluationLabDirectorLaurenHutchison,MPPGraduateAssistantMay22,2019
Thisreportprovidesanevaluationofthe4thAnnualEvaluationLabWorkshopheldonApril12,2019,basedonobservationsoftheEvaluation-SquaredTeam(Eval^2)andparticipantfeedbackprovidedviaanelectronicsurvey.
Eval^2,socalledbecauseitevaluatestheEvaluationLab,beganmeetinginJanuary2019toplantheWorkshopandtooverseetheannualevaluationoftheStudent-OrganizationPartnershipprogram.ThePartnershipisourcoreprogram;itcreatesevaluationteamscomposedofgraduatestudents,organizationstaffandanexperiencedUNMorcommunityevaluator.StudentstakeagraduatecourseinevaluationscienceintheFallandcompleteacompanionproject-basedcourseintheSpring.
The2019Eval^2memberswere:
• EvaluationLabDirector,MelissaBinder• AssociateDirector,SoniaBettez• TeamLead,ClaudiaDíazFuentes• Fellow,FelicityFonseca• SeniorFellow,AudreyCooper
ThecornerstoneoftheannualWorkshopisapostersessionwherestudentsdescribetheevaluationprojectsconductedduringtheyear.TheTeamsolicitedinputfromtheEvaluationLabclassforotherWorkshopcontentanddecidedtoincludethreeoftheirideas:trainingonsurveydesign,apanelonhoworganizationshavebenefitedfromevaluation,andasessiontoconsider“LifeaftertheEvaluationLab”tohelppartnerorganizationsplanforconductingevaluationsontheirown.TheTeamalsoaddeda30-minuteintroductiontotheLab,toaddressthesuggestionfrompreviousyears’participantsthattheWorkshopincludemoreinformationabouthowitoperates,andinvitedKimStraus,ManagerforBrindleFoundationaskeynotespeaker.(Seeappendixforthefinalagenda.)
TheCenterforNonprofitExcellence(CNPE)co-sponsoredtheWorkshop.CNPEpromotedtheeventonitswebsiteandprovidedtheconferenceroomattheUnitedWay,logisticalsupportandcoffee.
Theintendedaudiencewaspartnerorganizationsandunaffiliatednonprofits.Partnerorganizationstaffregisteredforfree;unaffiliatednonprofitspaid$30perperson.
GoalsfortheWorkshopwereto:
1.Createexcitementaboutevaluation2.Demonstratethatevaluationisdo-able3.PromoteSummerInstituteforthosewhowantcomprehensivetraining
ThepromotionalflyerfortheWorkshopincludedthefollowinglanguage,whichwehopedwouldaccuratelyportraytheevent:
2
Yesyoucan!Accessible&Hands-OnEvaluation2019EvaluationLabWorkshop
FeaturingKeynoteSpeakerKimStraus,FoundationManager,BrindleFoundation
• LearnabouttheUNMEvaluationLabapproachandprogramsthathelporganizationsuseevaluationtoolstoimprovetheireffectiveness.
• Seehowstudentspartneredwithorganizationtodesignparticipantsurveys,exploreddatabasesolutions,measuredintangibleoutcomesandusedevaluationtodevelopnewprograms.
• Participateinahands-ontrainingforcreatingmeaningfulsurveys.
Inretrospect,weshouldnothaveadvertised“ahands-on”trainingforsurveys,sincethatimpliesthatparticipantswouldbedevelopingsurveysonthespot,andseveralparticipantsexpresseddisappointmentthatthetrainingwasnotmore“hands-on.”
ItisdifficulttomanageexpectationsfortheWorkshop,andthisyear,intryingtodoabetterjobofexplainingwhatwedo,wemayhaveinadvertentlycreatedanatmospherewhereunaffiliatedparticipantsfeltlikeoutsiders.OneparticipantlatersharedwithCNPEthattheyweren’tsuretheyweresupposedtobethere,sincetheeventwasaboutorganizationsalreadyinvolvedwiththeLab.Asurveyrespondentfoundthatweusedjargonthattheydidn’tunderstand.
Inresponsetothisfeedback,Eval^2brainstormedalternativeformatsfortheend-of-yeargathering:
• Wecouldinterleavethepostersessionwithcontentrelatedtoposters,sothereismoreexplanation.
• Wecouldmovefroma“Workshop”toa“Reception,”anddoawaywiththetrainingsotheintentionisclear.
Ithinkweneedtobemorefocusedonmessage.Forexample,wewantedtoshowthat“Yesyoucan!”Soweshouldhavesomehowtalkedabouthowwemadetheprojectshappen...
Mostparticipantsdefinitelyengagedaswehadhopedtheywould.Feedbackfromthesurveywaspositive,andallrespondentssaidtheywouldrecommendtheWorkshoptoacolleague.
Organizationrepresentativeshadmorepositivefeelingsaboutevaluationaftertheworkshop,comparedwiththeirfeelingsbefore.
InresponsetoaquestionaboutsomethingtheylearnedintheWorkshop,participantssaidtheylearnedaboutlogicmodels,differentevaluationprocessesusedindifferentorganizations,datapresentationanduse,andspecificsforsurveydesign.
Eval^2notedhowtruncatedthesurveyresponseswere,comparedwithlastyear’sfocusgroupdata,whichfeltmuchricher.
3
SurveyQuestionsandResponses
35surveyscompleted:34onlineand1paper
Q1.Iama: 23oftherespondentswererepresentativesofnonprofitorganizations(65.71%).
9individuals(25.71%)answeredstudent.4individuals(11.43%)answeredOther.
Q2.Whatword(orwords)describes(ordescribe)yourthoughtsandfeelingsaboutevaluationthismorning,beforeyouattendedtheworkshop?[Selectallthatapply.]
OverhalftherespondentssaidtheywereinterestedinEvaluationpriortotheworkshop.
Mostoftheselectedanswersshowthatthecrowdwaslookingforwardtobeingabletoansweranyquestionstheyhadaboutevaluation.Themostselectedanswerswere:Interested,excited,nervous,understanding,andquestions.
Q3.Whatword(orwords)describes(ordescribe)yourthoughtsandfeelingsaboutevaluationnow?[Selectallthatapply.]
Overhalftherespondentsansweredtheywereinterestedinevaluationafterthesession.
Thetopsixresponsesshowedthatattendeeswalkedawayfromthesessionveryhappyandwithmoreunderstanding.Thetop6answerswere:Interested,Pleased,Understanding,Excited,Energetic,andConfident.
RespondentschosemorepositivewordsforevaluationaftertheWorkshop.(Seefigure1.)
Q4.Whatdidyoulearntodaythatyoudidn'tknowbefore?
Organizationrepresentativesreportedthattheylearned...• abouttheEvaluationLab• aboutsurveydesignandimplementation,includingthatiPhonescan
scanQRcodes;inthewordsofrespondents:“Matchingsurveyresponsestoprogrammission.Codingsurveyresponsestofindthemes,”“Usingnationalquestionstocomparetonationalsurveys.”
• aboutdatapresentationanduse• newideasforevaluation;onerespondentwrote,“Ilearnedabout
thedifferentevaluationprocessesfromdifferentagencies.”• whatalogicmodelisandhowtopresentit• frompeers,vianetworking
(Seefigure2forallresponses.)
“Ilearnedwhatlogicmodelswereandtheimportantsteps/tipstocreatingsurveysthatIdidn’tthinkaboutbefore.”
4
Figure1.Organizationrepresentatives'choiceofwordstodescribeevaluationafterandbeforeWorkshop(fromquestions2&3)
Note:Thisfiguresummarizestheresponseof23organizationrepresentatives.Respondentscouldchoosemorethanoneword,andtheycouldalsoaddtheirownwords.
1
1
1
1
1
1
1
1
3
6
2
2
5
7
1
2
3
6
4
17
1
1
1
2
2
2
4
4
6
7
9
11
11
16
0 2 4 6 8 10 12 14 16 18
Boring
Cautious
Doubtful
Manageable
Necessary
Obligatory
Opportune
Scared
Unaware
Unsure
Abitmoreinformed
Motivated
Unsurewhetherboardwillwanttodoasurvey
Nervous
Neutral
Stressed
Overwhelmed
Questions
Confident
Energetic
Pleased
Excited
Understanding
Interested
After Before
5
Figure2.WhatrespondentssaidtheylearnedintheWorkshop(fromquestion4)
Q5.Whichsessiondidyoulikethebest?
MostorganizationrespondentslikedthePosterSessionortheSurveysTrainingbest,butallsessionswerelikedbestbyatleast2people.(Seefigure3.)
6
Figure3.Sessionsthatorganizationrepresentativeslikedbest
Note:Thechoiceswere:HowtheEvaluationLabWorks,PosterSession,MeaningfulSurveys,KeynoteSpeakerandPanelontheValueofEvaluation.
Q6.WhatwouldyouchangeaboutthesessionsoranyotheraspectoftheWorkshop?
Responsesincluded:• Nothingtochange• Makesessions/activitiesmore“hands-on”• Logisticalchangestopostersession• Timesuggestions(i.e.morebreaks,sessionstoolong)• Moreexplanations/info(i.e.explainacronyms,logicmodels,etc.).• Wouldlikeadifferentlocation/layout• MoreFood• Wouldlikehandouts
7
Figure4.WhatrespondentswouldchangeabouttheWorkshop
Q7.WouldyourecommendtheWorkshoptoacolleague?
Allrespondentsanswered“yes”tothisquestion.
Q8.Overall,howwouldyouratetheWorkshop?(Movetheslidertomatchyourrating.)
Thisquestionappearedasaslider,withapoopemojiattheleftend,aneutralfaceatthecenter,andasmilingemojiwithheartsattherightend.Adjacenttothesliderisaboxthatrecordsascore.Thesliderwassetinthemiddle,at50.(Seefigure5.)
8
Figure5.ScreenshotofthesliderforratingtheWorkshopoverall
Theaverageratingfortheworkshopwas86/100.TheworkshopgetsaB.
Thesliderappearstobeagoodwaytoassesssatisfaction.Duringthesurveytraining,anaudiencememberinitiatedadiscussionaboutcustomerservicepollsexpecting“Excellent”scores.Thereisanexpectationthatnotgivingsomeonetopscoresmeansthereissomethingwrong.Eitheraudiencememberstookthattoheartingradingtheworkshop,ortheslidergivesamoreintuitivewayofchoosingaranking.ABseemsnotsogreat,butvisually,youcanseethatit’sdefinitelyalmostallthewayto100.Mostoftheparticipantsgaveascoreof80ormore;fourwerelesssatisfied.(Seefigure6.)
Figure6.Distributionoforganizationparticipantratings
Q9.Isthereanythingelseyou’dliketoshareabouttheWorkshop?
Responsesincluded• praiseaboutvalueofworkshopandfood;onerespondentwrote,“This
wasgreat!Gooduseofmytime.”• specificlearnings;onerespondentwrote,“[T]herearemanytypesof
methodswithinquantitativeandqualitativeumbrellasandfindingtherightmethodsthatreflectthevaluesofyourorganizationisessential.”
• adesirethattherewouldhavebeenamorediverseattendance,morefood,andmoreinteractiveactivitiesafterlunch
• interestinthesummersession.
2
98
2 2
100 90+ 80+ 70+ 50-67
“Therearemanytypesofmethodswithinquantitativeandqualitativeumbrellasandfindingtherightmethodsthatreflectthevaluesofyourorganizationisessential.”
9
10
UNM Evaluation Lab Annual Workshop
Yesyoucan!AccessibleandHands-OnEvaluation
April 12, 2019
10:00am-3:30 or 4:30pm
United Way Conference Room – 2nd
Floor, 2340 Alamo SE, Albuquerque
AGENDA
10:00 am Welcome & Introductions – Sonia Bettez
10:15 am Evaluation Lab: The Musical (How the Evaluation Lab works)
– Melissa Binder and Audrey Cooper with help from students, team leads and
participating organizations
• Student-Organization Partnership
• Direct training at Summer Institute & Learning Community
10:45 am Poster session – Students
The 2018-2019 Evaluation Lab: Community evaluation projects
12:15 pm Lunch and Keynote Speaker Kim Straus, Foundation Manager, Brindle
Foundation – A funder’s perspective on evaluation
1:20 pm The Value of Evaluation – Lauren Hutchison, Facilitator
Panel of organizations involved in Student training and Summer Institute
programs
• Guiovanna Aguirre, Centro Sávila
• Eileen Goode, New Mexico Primary Care Association
• Juliann Salinas, Enlace Comunitario
• Amanda Santiago, Centro Sávila (Summer Institute participant)
2:25 pm Creating Meaningful Surveys – Claudia Díaz Fuentes & Melissa Binder
3:30 pm Close public session
3:35 pm Life after the Evaluation Lab – Brainstorming session for Evaluation Lab
organizations
4:30 pm Adjourn
6:30pm Join us for a celebratory party at Melissa’s house – BYOB – 724 Lafayette Drive
NE, 505-250-4131
