Evaluationof2019EvaluationLabWorkshopMelissaBinder,EvaluationLabDirectorLaurenHutchison,MPPGraduateAssistantMay22,2019

Thisreportprovidesanevaluationofthe4thAnnualEvaluationLabWorkshopheldonApril12,2019,basedonobservationsoftheEvaluation-SquaredTeam(Eval^2)andparticipantfeedbackprovidedviaanelectronicsurvey.

Eval^2,socalledbecauseitevaluatestheEvaluationLab,beganmeetinginJanuary2019toplantheWorkshopandtooverseetheannualevaluationoftheStudent-OrganizationPartnershipprogram.ThePartnershipisourcoreprogram;itcreatesevaluationteamscomposedofgraduatestudents,organizationstaffandanexperiencedUNMorcommunityevaluator.StudentstakeagraduatecourseinevaluationscienceintheFallandcompleteacompanionproject-basedcourseintheSpring.

The2019Eval^2memberswere:

• EvaluationLabDirector,MelissaBinder• AssociateDirector,SoniaBettez• TeamLead,ClaudiaDíazFuentes• Fellow,FelicityFonseca• SeniorFellow,AudreyCooper

ThecornerstoneoftheannualWorkshopisapostersessionwherestudentsdescribetheevaluationprojectsconductedduringtheyear.TheTeamsolicitedinputfromtheEvaluationLabclassforotherWorkshopcontentanddecidedtoincludethreeoftheirideas:trainingonsurveydesign,apanelonhoworganizationshavebenefitedfromevaluation,andasessiontoconsider“LifeaftertheEvaluationLab”tohelppartnerorganizationsplanforconductingevaluationsontheirown.TheTeamalsoaddeda30-minuteintroductiontotheLab,toaddressthesuggestionfrompreviousyears’participantsthattheWorkshopincludemoreinformationabouthowitoperates,andinvitedKimStraus,ManagerforBrindleFoundationaskeynotespeaker.(Seeappendixforthefinalagenda.)

TheCenterforNonprofitExcellence(CNPE)co-sponsoredtheWorkshop.CNPEpromotedtheeventonitswebsiteandprovidedtheconferenceroomattheUnitedWay,logisticalsupportandcoffee.

Theintendedaudiencewaspartnerorganizationsandunaffiliatednonprofits.Partnerorganizationstaffregisteredforfree;unaffiliatednonprofitspaid$30perperson.

GoalsfortheWorkshopwereto:

1.Createexcitementaboutevaluation2.Demonstratethatevaluationisdo-able3.PromoteSummerInstituteforthosewhowantcomprehensivetraining

ThepromotionalflyerfortheWorkshopincludedthefollowinglanguage,whichwehopedwouldaccuratelyportraytheevent:

2

Yesyoucan!Accessible&Hands-OnEvaluation2019EvaluationLabWorkshop

FeaturingKeynoteSpeakerKimStraus,FoundationManager,BrindleFoundation

• LearnabouttheUNMEvaluationLabapproachandprogramsthathelporganizationsuseevaluationtoolstoimprovetheireffectiveness.

• Seehowstudentspartneredwithorganizationtodesignparticipantsurveys,exploreddatabasesolutions,measuredintangibleoutcomesandusedevaluationtodevelopnewprograms.

• Participateinahands-ontrainingforcreatingmeaningfulsurveys.

Inretrospect,weshouldnothaveadvertised“ahands-on”trainingforsurveys,sincethatimpliesthatparticipantswouldbedevelopingsurveysonthespot,andseveralparticipantsexpresseddisappointmentthatthetrainingwasnotmore“hands-on.”

ItisdifficulttomanageexpectationsfortheWorkshop,andthisyear,intryingtodoabetterjobofexplainingwhatwedo,wemayhaveinadvertentlycreatedanatmospherewhereunaffiliatedparticipantsfeltlikeoutsiders.OneparticipantlatersharedwithCNPEthattheyweren’tsuretheyweresupposedtobethere,sincetheeventwasaboutorganizationsalreadyinvolvedwiththeLab.Asurveyrespondentfoundthatweusedjargonthattheydidn’tunderstand.

Inresponsetothisfeedback,Eval^2brainstormedalternativeformatsfortheend-of-yeargathering:

• Wecouldinterleavethepostersessionwithcontentrelatedtoposters,sothereismoreexplanation.

• Wecouldmovefroma“Workshop”toa“Reception,”anddoawaywiththetrainingsotheintentionisclear.

Ithinkweneedtobemorefocusedonmessage.Forexample,wewantedtoshowthat“Yesyoucan!”Soweshouldhavesomehowtalkedabouthowwemadetheprojectshappen...

Mostparticipantsdefinitelyengagedaswehadhopedtheywould.Feedbackfromthesurveywaspositive,andallrespondentssaidtheywouldrecommendtheWorkshoptoacolleague.

Organizationrepresentativeshadmorepositivefeelingsaboutevaluationaftertheworkshop,comparedwiththeirfeelingsbefore.

InresponsetoaquestionaboutsomethingtheylearnedintheWorkshop,participantssaidtheylearnedaboutlogicmodels,differentevaluationprocessesusedindifferentorganizations,datapresentationanduse,andspecificsforsurveydesign.

Eval^2notedhowtruncatedthesurveyresponseswere,comparedwithlastyear’sfocusgroupdata,whichfeltmuchricher.

3

SurveyQuestionsandResponses

35surveyscompleted:34onlineand1paper

Q1.Iama: 23oftherespondentswererepresentativesofnonprofitorganizations(65.71%).

9individuals(25.71%)answeredstudent.4individuals(11.43%)answeredOther.

Q2.Whatword(orwords)describes(ordescribe)yourthoughtsandfeelingsaboutevaluationthismorning,beforeyouattendedtheworkshop?[Selectallthatapply.]

OverhalftherespondentssaidtheywereinterestedinEvaluationpriortotheworkshop.

Mostoftheselectedanswersshowthatthecrowdwaslookingforwardtobeingabletoansweranyquestionstheyhadaboutevaluation.Themostselectedanswerswere:Interested,excited,nervous,understanding,andquestions.

Q3.Whatword(orwords)describes(ordescribe)yourthoughtsandfeelingsaboutevaluationnow?[Selectallthatapply.]

Overhalftherespondentsansweredtheywereinterestedinevaluationafterthesession.

Thetopsixresponsesshowedthatattendeeswalkedawayfromthesessionveryhappyandwithmoreunderstanding.Thetop6answerswere:Interested,Pleased,Understanding,Excited,Energetic,andConfident.

RespondentschosemorepositivewordsforevaluationaftertheWorkshop.(Seefigure1.)

Q4.Whatdidyoulearntodaythatyoudidn'tknowbefore?

Organizationrepresentativesreportedthattheylearned...• abouttheEvaluationLab• aboutsurveydesignandimplementation,includingthatiPhonescan

scanQRcodes;inthewordsofrespondents:“Matchingsurveyresponsestoprogrammission.Codingsurveyresponsestofindthemes,”“Usingnationalquestionstocomparetonationalsurveys.”

• aboutdatapresentationanduse• newideasforevaluation;onerespondentwrote,“Ilearnedabout

thedifferentevaluationprocessesfromdifferentagencies.”• whatalogicmodelisandhowtopresentit• frompeers,vianetworking

(Seefigure2forallresponses.)

“Ilearnedwhatlogicmodelswereandtheimportantsteps/tipstocreatingsurveysthatIdidn’tthinkaboutbefore.”

4

Figure1.Organizationrepresentatives'choiceofwordstodescribeevaluationafterandbeforeWorkshop(fromquestions2&3)

Note:Thisfiguresummarizestheresponseof23organizationrepresentatives.Respondentscouldchoosemorethanoneword,andtheycouldalsoaddtheirownwords.

1

1

1

1

1

1

1

1

3

6

2

2

5

7

1

2

3

6

4

17

1

1

1

2

2

2

4

4

6

7

9

11

11

16

0 2 4 6 8 10 12 14 16 18

Boring

Cautious

Doubtful

Manageable

Necessary

Obligatory

Opportune

Scared

Unaware

Unsure

Abitmoreinformed

Motivated

Unsurewhetherboardwillwanttodoasurvey

Nervous

Neutral

Stressed

Overwhelmed

Questions

Confident

Energetic

Pleased

Excited

Understanding

Interested

After Before

5

Figure2.WhatrespondentssaidtheylearnedintheWorkshop(fromquestion4)

Q5.Whichsessiondidyoulikethebest?

MostorganizationrespondentslikedthePosterSessionortheSurveysTrainingbest,butallsessionswerelikedbestbyatleast2people.(Seefigure3.)

6

Figure3.Sessionsthatorganizationrepresentativeslikedbest

Note:Thechoiceswere:HowtheEvaluationLabWorks,PosterSession,MeaningfulSurveys,KeynoteSpeakerandPanelontheValueofEvaluation.

Q6.WhatwouldyouchangeaboutthesessionsoranyotheraspectoftheWorkshop?

Responsesincluded:• Nothingtochange• Makesessions/activitiesmore“hands-on”• Logisticalchangestopostersession• Timesuggestions(i.e.morebreaks,sessionstoolong)• Moreexplanations/info(i.e.explainacronyms,logicmodels,etc.).• Wouldlikeadifferentlocation/layout• MoreFood• Wouldlikehandouts

7

Figure4.WhatrespondentswouldchangeabouttheWorkshop

Q7.WouldyourecommendtheWorkshoptoacolleague?

Allrespondentsanswered“yes”tothisquestion.

Q8.Overall,howwouldyouratetheWorkshop?(Movetheslidertomatchyourrating.)

Thisquestionappearedasaslider,withapoopemojiattheleftend,aneutralfaceatthecenter,andasmilingemojiwithheartsattherightend.Adjacenttothesliderisaboxthatrecordsascore.Thesliderwassetinthemiddle,at50.(Seefigure5.)

8

Figure5.ScreenshotofthesliderforratingtheWorkshopoverall

Theaverageratingfortheworkshopwas86/100.TheworkshopgetsaB.

Thesliderappearstobeagoodwaytoassesssatisfaction.Duringthesurveytraining,anaudiencememberinitiatedadiscussionaboutcustomerservicepollsexpecting“Excellent”scores.Thereisanexpectationthatnotgivingsomeonetopscoresmeansthereissomethingwrong.Eitheraudiencememberstookthattoheartingradingtheworkshop,ortheslidergivesamoreintuitivewayofchoosingaranking.ABseemsnotsogreat,butvisually,youcanseethatit’sdefinitelyalmostallthewayto100.Mostoftheparticipantsgaveascoreof80ormore;fourwerelesssatisfied.(Seefigure6.)

Figure6.Distributionoforganizationparticipantratings

Q9.Isthereanythingelseyou’dliketoshareabouttheWorkshop?

Responsesincluded• praiseaboutvalueofworkshopandfood;onerespondentwrote,“This

wasgreat!Gooduseofmytime.”• specificlearnings;onerespondentwrote,“[T]herearemanytypesof

methodswithinquantitativeandqualitativeumbrellasandfindingtherightmethodsthatreflectthevaluesofyourorganizationisessential.”

• adesirethattherewouldhavebeenamorediverseattendance,morefood,andmoreinteractiveactivitiesafterlunch

• interestinthesummersession.

2

98

2 2

100 90+ 80+ 70+ 50-67

“Therearemanytypesofmethodswithinquantitativeandqualitativeumbrellasandfindingtherightmethodsthatreflectthevaluesofyourorganizationisessential.”

9

10

UNM Evaluation Lab Annual Workshop

Yesyoucan!AccessibleandHands-OnEvaluation

April 12, 2019

10:00am-3:30 or 4:30pm

United Way Conference Room – 2nd

Floor, 2340 Alamo SE, Albuquerque

AGENDA

10:00 am Welcome & Introductions – Sonia Bettez

10:15 am Evaluation Lab: The Musical (How the Evaluation Lab works)

– Melissa Binder and Audrey Cooper with help from students, team leads and

participating organizations

• Student-Organization Partnership

• Direct training at Summer Institute & Learning Community

10:45 am Poster session – Students

The 2018-2019 Evaluation Lab: Community evaluation projects

12:15 pm Lunch and Keynote Speaker Kim Straus, Foundation Manager, Brindle

Foundation – A funder’s perspective on evaluation

1:20 pm The Value of Evaluation – Lauren Hutchison, Facilitator

Panel of organizations involved in Student training and Summer Institute

programs

• Guiovanna Aguirre, Centro Sávila

• Eileen Goode, New Mexico Primary Care Association

• Juliann Salinas, Enlace Comunitario

• Amanda Santiago, Centro Sávila (Summer Institute participant)

2:25 pm Creating Meaningful Surveys – Claudia Díaz Fuentes & Melissa Binder

3:30 pm Close public session

3:35 pm Life after the Evaluation Lab – Brainstorming session for Evaluation Lab

organizations

4:30 pm Adjourn

6:30pm Join us for a celebratory party at Melissa’s house – BYOB – 724 Lafayette Drive

NE, 505-250-4131