SAMS—Social Assistance Management System Chapter 3 Section 3.12 Chapter 3 • VFM Section 3.12 471 Ministry of Community and Social Services 1.0 Background 1.1 Social Assistance in Ontario Social assistance helps people who are in need because they are unemployed and/or have dis- abilities. It provides: • financial aid; • health benefits; • access to basic education; and • counselling, training and workshops to help people find and keep a job. The overall objective of social assistance is to help people become as self-sufficient as possible. Programs are funded and administered by the prov- ince and municipalities. To help improve the administration and delivery of social assistance, the Ministry of Community and Social Services (Ministry) replaced its old information technology system in November 2014 with a new system known as the Social Assistance Management System (SAMS). About 11,000 ministry and municipal personnel have to rely on SAMS to help them deliver social assistance to approximately 900,000 of the most vulnerable members of society across Ontario. These personnel rely on SAMS to, among other things: • determine an applicant’s eligibility for social assistance; • calculate and distribute about $6.6 billion in annual social benefit payments; • automatically generate letters that are mailed to people to inform them about their social assistance eligibility or about changes to their social benefits; and • generate reports that provide the information that municipalities and the Ministry need to manage social assistance programs. 1.1.1 Three Social Assistance Programs in Ontario In Ontario, three programs provide social assist- ance: Ontario Works, the Ontario Disability Sup- port Program (ODSP) and Assistance for Children with Severe Disabilities (ACSD) (because the Ministry combines information on the latter two programs in much of its reporting, we do the same in this report and include information on ACSD in our discussions of ODSP). Ontario Works The Ontario Works Act, 1997 and its regulations govern the delivery of Ontario Works. Eligible people receive Ontario Works support and services from 238 municipal offices across the province, in
Transcript
SAMS—Social Assistance Management System
Chapter 3Section 3.12
Chap
ter 3
• VF
M S
ectio
n 3.
12
471
Ministry of Community and Social Services
1.0 Background
1.1 Social Assistance in OntarioSocial assistance helps people who are in need because they are unemployed and/or have dis-abilities. It provides:
•financialaid;
•healthbenefits;
•accesstobasiceducation;and
• counselling, training and workshops to help peoplefindandkeepajob.
Theoverallobjectiveofsocialassistanceistohelppeoplebecomeasself-sufficientaspossible.Programsarefundedandadministeredbytheprov-ince and municipalities.
To help improve the administration and delivery ofsocialassistance,theMinistryofCommunityandSocialServices(Ministry)replaceditsoldinformationtechnologysysteminNovember2014with a new system known as the Social Assistance ManagementSystem(SAMS).
• calculate and distribute about $6.6 billion in annualsocialbenefitpayments;
• automatically generate letters that are mailed topeopletoinformthemabouttheirsocialassistance eligibility or about changes to their socialbenefits;and
•generatereportsthatprovidetheinformationthatmunicipalitiesandtheMinistryneedtomanage social assistance programs.
Ontario Works The Ontario Works Act, 1997 and its regulations governthedeliveryofOntarioWorks.EligiblepeoplereceiveOntarioWorkssupportandservicesfrom238municipalofficesacrosstheprovince,in
2015 Annual Report of the Office of the Auditor General of Ontario472
Also,apersonneedstotrytofindandkeepajob,andparticipateinactivitiesdesignedtohelphim or her do so (such as workshops and programs thathelpthepersonfinishhighschool).
RecipientsofOntarioWorksfinancialaidreceivebasic-needsandshelterallowances.Otherfinancialassistance is provided to eligible clients through specifictypesofbenefits,usuallyprovidedmonthly(examples include the Pregnancy and Breast-feedingNutritionalAllowanceandtheSpecialDietAllowance).Clientsarealsoreimbursedforcertainexpenses, such as employment-related expenses. Benefitsaretaxable;reimbursementsarenot.
Ontario Disability Support Program (ODSP)The Ontario Disability Support Program Act, 1997 anditsregulationsgovernthedeliveryofODSP.EligiblepeoplereceiveODSPsupportandservicesfrom45Ministryofficesthroughouttheprovince.
TobeeligibleforODSP,apersonmusthaveasubstantial physical or mental impairment that has lastedforatleastoneyear.Theimpairmentmustrestrictthepersonfromatleastonedailylivingactivity.Also,theperson’snetassets,apartfromtheir home and motor vehicle, must be worth less than$5,000ifthepersonissingleand$7,500iftheperson has a spouse.
LikeOntarioWorksclients,ODSPclientsmayalsoreceiveanumberofotherspecificbenefitsandmaybereimbursedforspecificexpenses.Forexample,theyreceivecompensationformobil-ity devices, hearing aids and any other devices
About11,000front-linepersonnel,mostofwhomarecalledcaseworkers,provideafullrangeofOntarioWorksandODSPservices.CaseworkershavetorelyonSAMSonadailybasistohelpthemprovide these services, which we describe later in thissection.TheirabilitytoprovideOntarioWorksandODSPishighlydependentonhowwellSAMSfunctionsandsupportsthem.Inotherwords,forcaseworkerstohavesufficienttimetohelptheirclients,SAMSshould:
• accurately calculate and distribute social benefitpayments;and
• automatically generate letters accurately informingclientsoftheireligibilityandtheamounts to which they are entitled, plus other documents.
Atthesametime,SAMSshouldhavecontrolstoensurethatclients’informationisprotectedandthattheamountofriskoffraudandabuseofsocial-assistance programs is as low as possible.
SAMSshouldalsobeeasytouse—thatis,itshould be designed with caseworkers’ needs in mind.Necessaryfeaturesinclude,forexample,areminderforcaseworkersofupcomingdailytasksthey need to accomplish so that they can ensure theirclientsareadequatelylookedafter.
Ontario Works Caseworkers PeopleseekinghelpfromOntarioWorkscanapplyonline,inpersonatanOntarioWorksofficeorbyphone.
473SAMS—Social Assistance Management System
Chap
ter 3
• VF
M S
ectio
n 3.
12
TheOntarioWorkscaseworker’sresponsibilitiesbegin when an applicant makes contact to schedule an in-person meeting. At that meeting, the case-workerbeginstheprocessofdeterminingiftheapplicantqualifiesforassistance.Iftheapplicantdoesqualifyandbecomesaclient,thecaseworkercontinues to meet regularly with him or her to help theclientfindandkeepajob.Thecaseworkerwillalso:
Caseworkershaveanumberofotherrespon-sibilities.Forexample,theyattendhearingswhenclients dispute their entitlements and they recover overpaymentsfromformerclients.
ODSP CaseworkersPeopleseekinghelpfromODSPalsocanapplyonline,inpersonatanODSPofficeorbyphone.These individuals have physical or mental impair-ments that, to varying degrees, can impact their abilitytobeinvolvedintheworkforce.Someareabletowork;somewithmoresevereimpairmentsfinditdifficultorimpossibletowork.
AnODSPcaseworker’srolevariesdependingonthe particular impairments his or her clients have. Ifaqualifiedapplicant’simpairmentisnotsevere,thecaseworkerwillmeetmorefrequentlywiththeclient and develop a structured employment plan. Caseworkersmaymeetlessregularlywithclientswho have more severe impairments, as these clients receivebenefitpaymentsonaregularschedulethroughtheirenrolmentinODSP.MostODSPcase-workersthereforehavehighercaseloadsthanmostOntarioWorkscaseworkers.
1.2 Management of Social Assistance Information1.2.1 Prior Information-management System Used Between 2002 and 2014
Theinformation-managementsystemusedbetween2002and2014wascalledServiceDeliveryModelTechnology(“previoussystem”).Weaud-itedtheOntarioWorksprogramin2002(seeour2002 Annual Report)andnotedthefollowingwithregard to the previous system that had been imple-mented at that time:
•Caseworkersreportedthatthesystemwasnoteasytouse—ithadnotbeendesignedwiththeir needs in mind.
• It could not accurately determine client eligi-bilityandbenefitamounts.
• It could not generate certain reports toprovidetheinformationneededfordecision-makers.
• It did not include adequate controls against fraud.
WhenweauditedODSPforour2004 Annual Report,wefoundtheMinistryhadmademanychanges to the previous system to ensure that it producedconsistentandcorrectinformation.How-ever,wefoundthattheprevioussystemstill“lackedkey internal controls, still did not meet certain key informationneedsofministryusersandrecipientsofdisabilitysupportpayments,andcontinuedtogenerateerrorsandomitinformationforreasonsthatcouldnotbeexplained.”
2015 Annual Report of the Office of the Auditor General of Ontario474
Chap
ter 3
• VF
M S
ectio
n 3.
12
WeagainauditedbothOntarioWorksandODSPforour2009 Annual Report, and reported that“despiteimprovementstotheMinistry’sServiceDeliveryModelTechnologyinformationsystemsinceitsrolloutin2002—manyofwhichwere intended to enhance reliability as well as the completenessandaccuracyofitsinformation—thesystem continues to have reliability concerns and knowndeficiencies.”
1.2.2 New Information-management System Implemented in 2014
Approval and Development of SAMS In2009,theOntariogovernmentembarkedonaninitiative to modernize aging computer technology acrossthegovernment.TheMinistryidentifiedtheprevious social assistance system as high risk and apriorityformodernization.Itnotedthat,amongotherthings,thesystem’sdesignwasnoteffective.Inaddition,thesystemitselfwasbasedonoutdatedtechnology and so could not adequately support the businessandpolicychangescomingintoeffectforsocial assistance.
TheMinistry’sbusinesscasestatedthatthemosteconomicalandeffectivewayto“modernize”theprevious system would be to replace it with a new “commercialoff-the-shelf”system.Inotherwords,itwouldlookforacommerciallyavailablesystemthat it could buy as-is, and then customize to meet its business needs.
TheMinistrysetupacompetitionforcom-mercialoff-the-shelfsystems,andtheCuramCaseManagementSystemwoninDecember2009.Thissoftwarecostsignificantlylessthanestimatedin the business case, and as a result, the overall projectbudgetwasreducedto$164.9millioninFebruary2010.
• Interfaces (connections with other com-puter systems)—thiswasdonebyIBMconsultants.
Inadditiontothesoftware,theMinistrypur-chased hardware, such as servers to store data, and central processing units to process the data, both fromIBM.
Launch StrategyLatein2010,theMinistrydecidedthata“big-bang”launchwouldbethebestwaytoimplementSAMS.Thismeantthat,overnight,SAMSwouldcom-pletely replace the previous system.
Suchanimplementationisrisky.Forinstance,iffromthemomentoflaunchSAMSdoesnotwork,orcaseworkers do not know how to use it, vulnerable clientswhodependonbenefitsmightnotreceivethe money they need to meet basic living expenses suchasfoodandshelter.TheMinistrywasawareofthissignificantriskbutplannedtominimizeitbythoroughlytestingSAMStoensureitworkedcor-rectly and by training caseworkers in advance.
Since the previous system would no longer be in usewhenSAMSwaslaunched,thismeantthatthevastamountofdataintheprevioussystemwouldhavetobetransferredintoSAMS.TominimizethedisruptiontoOntarioWorksandODSP,thetransferwouldhavetooccurinasbriefatimeaspossible.
Deciding When to “Go Live”Planningwhentolaunchaninformationsystem,andthendecidingifthesystemisreallyreadyto“golive,”arecrucialdecisions.Launchingasystemthatisnotreadycancreatehavocfortheservicedeliverythatthesystemwasdesignedtofacilitateand improve. This risk was especially worrisome in thecaseofSAMSbecausethathavocwouldaffectthelivesofover900,000ofthemostvulnerablemembersofsociety.
The launch date was changed several times becauseofdelaysandissues.InOctober2014,workingtowardalaunchdateofNovember2014,theMinistryfollowedaformalprocesstoassesswhether:
•SAMShadbeensufficientlytested;
•hardwarehadbeenconfigured,softwarewasworking correctly, and all data was ready to be transferred;and
• resources were in place to support casework-ersandmanageSAMSafterlaunch.
Whenwecompletedouraudit,theMinistrywasstillintheprocessoftryingtofixthedefectsandgetSAMSworkingproperly.Atthatpoint,theMinistryinformedusthatthiswillcostaboutanadditional$52million(someofwhichhasalreadybeenspent)onSAMSsincelaunch,inadditiontothe$238millionspentbeforelaunch,foratotalofabout$290million.AstheMinistrydoesnotantici-pateSAMSwillbecomefullystableuntilspring2016,thefinalcostofSAMSwillremainunknownuntil that time.
•thedevelopmentandimplementationoftheSocialAssistanceManagementSystem(SAMS)wasplannedandmanagedeco-nomically,effectivelyandefficiently,andin compliance with applicable policies and requirements;and
2015 Annual Report of the Office of the Auditor General of Ontario476
Chap
ter 3
• VF
M S
ectio
n 3.
12
Inaddition,ourauditincludedareviewofrel-evant audit reports issued by the province’s Internal AuditDivision.Thesereports,thelastofwhichwasissuedinNovember2013,werehelpfulindetermin-ingthescopeandextentofourauditwork.(Wedis-cusstheinvolvementoftheInternalAuditDivisioninSAMSinSection 4.4.5).
TheSocialAssistanceManagementSystem(SAMS)wasnotproperlypilotedorfullytestedduring its development. Tests that were done yielded results that were below expectations. The Ministrylaunchedanywaybecauseitconsideredtherisksofdelayingthelaunchgreaterthantherisksoflaunchingasystemthatwasnotfullyready.Further,thedecisiontolaunchwasbasedonincompleteandinaccurateinformationaboutSAMS’readiness.
TheMinistrylaunchedSAMSinawaythatmakes it impossible to return to the previous system. SincetheMinistrymustsalvageSAMS,itiscrucialthatitprioritizetheallocationofresourcestofixingit.Untiltheissuesareresolved,itremainsunknownwhetherSAMSwillperformbetterthantheprevi-oussystem.Moreimportantlywithoutacorrectlyfunctioningsystem,caseworkerscannotprovideadequatesocialservicetoover900,000clients.
• The Ministry had yet to identify many defects, and was not fully testing its software upgrades that fix defects—AsofJuly31,2015,therewere771seriousdefectsoutstandinginSAMS.Thisnumberisnotcomplete,however,becausemanydefectshadyettobeidentified.Furthermore,theMin-istryhadnotmadefixingdefectsapriority.Specifically:
• TheMinistryhadabacklogofabout11,500callsfromthehelpdeskthatithadnotyetreviewed. There was also an additional backlogforprocessingcallstootherhelplines.CallerstohelplinesbringpotentialnewdefectsinSAMStotheattentionoftheMinistry.TheMinistryalsohadabackloginreviewing439problemsidentifiedthroughthesecalls,mostofwhichcouldendupasdefectsneedingtobefixed.
• TheMinistryhadinstalledsoftwareupgradestofixdefectsbutwasnotfullytesting them. This was partly because it didnotknowhowtotestthem—justpriortolaunch,theMinistrydidnotrenewcon-tracts with certain consultants who would havebeenthemosteffectiveintestingthefixes.InJuly2015,theMinistryhiredeightnewconsultantstoworkonfixesbutestimated it would take about six months forthesenewconsultantstoreachthesamelevelofknowledgeastheconsultantswhohad been let go.
• SAMS is still not functioning properly—Untilmostoftheseriousdefectsareidentifiedandfixed,andsoftwareupgradesareproperlytested,SAMSwillcontinuetogenerateerrors.Untildefectsaredealtwith,problemswillpersist,andSAMSwillremaindifficulttouse,will continue to generate incorrect eligibil-itydeterminationsandbenefitpayments,will continue to generate inaccurate reports thattheMinistryandmunicipalitiesneedtoproperlymanageOntarioWorksandOntarioDisabilitySupportProgram,andwilllackcontrolsforreducingtheriskoffraud.Inaddi-tion, caseworkers will continue to have to use time-consuming“workarounds”todealwiththese problems.
• The Executive Committee assumed sig-nificant risk when it decided to launch SAMS—TheExecutiveCommitteeunderstoodthatSAMSdidnotmeetthelaunchcriteriadevelopedbytheMinistryandassumedtherisk that this entailed. It also understood that thefollowingotherrequirementsforlaunch
(notincludedinitslaunchcriteria)hadnotbeen met and also assumed the risk that this entailed:
• Pilottestingwithdataconvertedfromtheprevious system was never conducted, so it wasnotknownifSAMSwouldworkasfullyintended when launched.
• SixteenpercentofSAMS’functionswerenottested,andthefailurerateoffunctionsthat were tested was one in eight.
• Onlysomeofthegovernment-mandatedpayment testing was conducted, and many seriouspayment-relateddefectswerefoundafterlaunch.AccordingtotheOfficeoftheProvincialController,SAMSistheonlycomputer system ever connected to the government’s accounting system without passing the government-mandated pay-ment testing.
• The Executive Committee was not aware of the full extent of SAMS’ pre-launch issues—WhiletheExecutiveCommitteeknowinglyassumedrisksofSAMSnotmeet-ing the launch criteria and other require-ments,itsdecisiontolaunchSAMSwasnotbasedoncompleteinformationbecausetheprojectteamdidnottelltheCommitteeaboutthefollowingwithrespecttoSAMS’readiness:
• Questionable shift in roles and report-ing relationships, lack of Internal Audit involvement, in critical period up to SAMS’ launch—Inthesixmonthsbeforelaunch,the testing team’s reporting relationship was abruptly changed and started to report totheBusinessProjectDirectorinsteadof
2015 Annual Report of the Office of the Auditor General of Ontario478
Chap
ter 3
• VF
M S
ectio
n 3.
12
theTechnicalProjectDirector,asithadbeendoing.TheBusinessProjectDirectorhad no IT background and limited technical expertise.Asaresult,theTechnicalProjectDirector’sexpertisesurroundingtestingwasnotconsideredwhenSAMS’readinesswasassessed and the decision to go live was made. Duringthissametime,theOntarioInternalAuditDivision(InternalAudit)proposedanauditofSAMS’readinessfourmonthsbeforelaunch.However,asInternalAuditandSAMS’projectleadscouldnotagreeonthescopeoftheaudit,itwasnotdone.TheMinistryalsostatedthatanauditofSAMS’readinesswasunnecessary,giventheexpertiseoftheIBMconsultantspreparingitforlaunch.
• IBM was unable to correctly convert data from the previous system on time, and this delayed SAMS’ launch—OneprojectrequirementforSAMSwasthatallclientdata in the previous system, going back two years,betransferredintoSAMS.TheMinistrychoseIBMforthetaskofconvertingthedataintoaformatSAMScoulduse.IBMfailedto meet its deadline on three occasions, and theMinistryextendedthedeadlinesthreetimes.ItistruethattheMinistryreviseditsrequirementsforSAMSonseveraloccasions,whileIBMwasstilldoingitswork,andthisposedchallengesforthedata-conversionprocess.Inanycase,becauseofthedelays,thereneverwasaneffectivepilotofSAMSusingtheconverteddata;theMinistryhadtopush back the launch date three times, and theprojectbudgetroseto$242millionfrom$202.3million.IBMfinallydeliveredthedatainApril2014andatlaunch,therewereabout114,000errorsinthedatathatcausedSAMStogenerateincorrectresultsforclienteligibil-ityandbenefitpayments.
• Ministry should have overseen consultants; instead, consultants oversaw other con-sultants through most of SAMS’ develop-ment—TheMinistrydidnotproperlyoversee
CuramandIBMconsultants.Itreliedontheconsultants not only to design and develop mostofSAMS,buttoalsooverseetheirownwork.Consultantsbilledanaveragehourlyrateof$190.Theywereoverseenbyotherconsultants who were paid daily rates as highas$2,000.Manyconsultantstookmuchlonger than anticipated to complete their work,andinsomeinstancesbilledfortimespentonfixingerrorsintheirownwork.TheMinistry’sbudgetforCuram’sconsultantsmorethandoubled,from$14millionintheoriginalbudgetto$32millionatlaunch.Thevagueness in consultants’ time reporting, andthelackofindependentoversightduringmuchoftheproject,madeitdifficulttoassesshowefficientlyconsultantswereworking.
• Ministry training of staff inadequate—TheMinistryprovidedonlineSAMSusertrainingbetweenJanuaryandMay2014.Caseworkerstold us that the training program repeatedly shut down without warning and had many errors.Overhalfofthecaseworkerswhocompletedasurveyattheendoftheprogramsaidtheydidnotfeelconfidenttheywouldbeabletousethesystemforcomplexreal-lifesituations, and one-third said they did not feelconfidenttheycouldprocessthedataforevery-daytasks.Afterlaunch,about80%ofOntarioWorksandOntarioDisabilitySupportProgramofficesreportedthatcaseworkershad to deal with many problems sparked by SAMS,andthatthereweresignificantissueswithstaffmorale.Almostone-quarteroftheseofficesreportedthattheywere“unabletocon-tinueoperationswithoutadditionalsupport.”
Thisreportcontainsfiverecommendations,con-sistingof12actions,toaddressthefindingsnotedduring this audit.
OVERALL MINISTRY RESPONSE
TheMinistryvaluestheworkoftheAuditorGeneral and appreciates the advice on how to
479SAMS—Social Assistance Management System
Chap
ter 3
• VF
M S
ectio
n 3.
12
improvetheSocialAssistanceManagementSystem(SAMS)andfutureimplementationsoftechnologysolutions.Weagreewithalltherecommendations and will implement them as outlinedinthefollowingresponses.
There was a critical need to replace the Ser-viceDeliveryModelTechnology(SDMT),theold computer system that managed social assist-ance.AstheAuditorpointedoutin2009,SDMThadsecurityandcontrolissues.The14-yearoldsystemwasunstable,athighriskoffailureandhad to be replaced.
TheMinistryacknowledgesthattheimple-mentationofSAMSwasmorechallengingthananticipated.Wecontinuetomakeprogressinaddressing technical issues and improving the system.Inaddition,frontlinestaffanddeliverypartners are working hard to ensure that social assistance clients continue to be well served. Throughoutimplementation,theMinistryfocusedonensuringclientsreceivedthesup-port to which they are entitled. As impacts to productivitywereanticipated,theMinistryimplemented workload mitigation strategies which put on hold certain activities with less direct client impact to ensure social assistance clientscontinuedtobeserved.Wearegratefultofront-linestaffformaintaininghighqualitycustomer service.
Weappreciatetheopportunitytorespondtothe Auditor General’s recommendations and the MinistryiscommittedtoensuringthatSAMSprovidesthenecessaryfoundationtotrans-formandmodernizesocialassistanceservicedelivery.Wewillcontinueworkingcloselywith
municipal delivery partners on the ongoing improvementofSAMS.
4.0 Detailed Audit Observations
TheSocialAssistanceManagementSystem(SAMS)launchedinNovember2014hadseriousdefectsandwasnotfullyfunctional.Section 4.1 describes theconditionofSAMSatthetimewecompletedour audit. Section 4.2 describes the progress the MinistryofCommunityandSocialServices(Min-istry)hadmadeafterlaunchtofixit.Section 4.3 explainswhatwentwrongthroughoutprojectdevelopment. Section 4.4 presents our concerns withthedecisiontolaunchSAMSintheconditionit was in.
4.1 SAMS Defective at Time of Launch Atthetimeitwaslaunched,SAMScontainedabout2,400seriousdefectsthatcausedmanydifferenttypesoferrorsinclients’eligibilityforbenefitsandthepaymentstheyreceived.Someoftheseerrorsweredifficultforcaseworkerstoidentifyandcor-rect.Thesedefectsweremainlyduetopoorlypro-grammedsoftwareandincorrectlyconverteddata.Figure 1showsthebreakdownofknowndefects.
Giventhatseveralhundreddefectsremainedin the system when we completed our audit, webelievethatSAMSwillcontinuetocalculateincorrectbenefitamounts.Theseerrorsaddeduptoabout$140millionwhenwecompletedouraudit,and each subsequent calculation error will increase this total.
Atthetimeofouraudit,wenotedthatcase-workers had tried to intercept and manually correct
2015 Annual Report of the Office of the Auditor General of Ontario480
4.1.1 Defective System Caused Difficult-to-identify Errors
Atthecompletionofouraudit,severalhundreddefectshadnotbeenfixedandremainedoutstand-ing, as we discuss in Section 4.2.Defectscausemanydifferenttypesoferrorstoclients’benefiteligibility and payments.
Ifcaseworkersareunabletoidentifyandcor-recterrors,somemaygounnoticedformonths,ormayneverevenbeidentifiedandresolved.Socialassistance recipients are considerably vulnerable insofarasmostaredisadvantagedordisabled.Manyofthemmaybeunawareoferrorsintheirbenefitpayments,orlacktheconfidencetodisputegovernment-issued money or documents.
paidtotheclient.SAMSproceededtorecoverthis nonexistent overpayment by deducting $32fromtheclient’stotalbenefitpaymentseachmonth.Theclientfoundthemistakeandnotifiedthecaseworker.However,asstatedearlier, many clients are not able, or inclined, toreviewthedetailsoftheirpayments.
•TwofilesappearedinSAMSforthesameindi-vidual, allowing this person to receive twice thelegitimatebenefitpaymentsforthreemonths,beforethecaseworkercaughtit.
CaseworkersalsotoldusofaninstancewhereSAMSoverpaidbenefitstoaclientwithmentaldisabilitieswhodidnotrealizethebenefitwastoohigh. The individual spent the money, and did not havethemeanstorepayit.ThroughtheMinistry’sowncollectionefforts,thebankfrozetheclient’saccount, leaving the client without any money. A caseworkerhadtoworkaroundSAMSbyissuingcheques by hand to the client.
Figure 1: Serious Defects* Found in SAMS at the Completion of Our AuditSource of data: Ministry of Community and Social Services
Known Prior Found afterErrors and Problems Caused by Defects to Launch Launch TotalEligibility determination and payment amounts not correct, and other system functionalities not working
257 875 1,132
Client data converted from previous system not accurate 38 35 73Letters to clients not correct 81 221 302Reports incomplete and inaccurate 156 97 253Information sent to other computer systems through interfaces not accurate 129 150 279Overall system performance problems (including speed) 50 178 228Problems faced by clients while accessing SAMS’ online features 26 60 86Total 737 1,616 2,353
* A serious defect produces the wrong result and may require a workaround to produce the right result.
481SAMS—Social Assistance Management System
Chap
ter 3
• VF
M S
ectio
n 3.
12
4.1.2 Documents Containing Incorrect Information Mailed to Clients
TheMinistrytoldusthatallincorrectT5slipswerecorrectedbeforetheyweresenttoclients.However,wesawseveralinstancesatoursitevisitsofclientsbringingincorrectT5slipsthattheyhadreceivedtotheircaseworkersafternoticingerrors.Therecouldbeotherclientswhodidnotidentifyerrors and so would have reported incorrect informationintheir2014taxreturns.Thiscouldultimatelyaffecttheireligibilityforbenefits,theamountofbenefitstheyreceive,andthetaxtheyhave to pay.
Wealsosawanumberoferroneouslettersthatclients brought to their caseworkers that would havecausedtheclientsstressandconfusion.Wefoundlettersthatstatedthat:
•Twoclientslivingtogethereachowed$8,736because they had been overpaid (the clients infactowedonly$664each).WeincludedoneofthetwoletterstheseclientsreceivedinAppendix 1.
•AnODSPfilehadbeenputonholdandtheclient would not receive income support or otherbenefitsbecausetheclientdidnotliveinOntario(theclienthadneverleftOntario).
Caseworkersalsotoldusthatotherdocuments,suchasdrugcards,alsocontainedincorrectinfor-mation.Drugcardsprovetopharmaciesthattheholdersareeligibleformedicationcoverage.InoneinstanceSAMSprintedthenameofadeceasedchild on the parent’s drug card and the card was mailed to the parent.
4.1.3 Defective System Required Caseworker “Workarounds,” Taking Time Away from Clients
SAMSwassupposedtorelievecaseworkersofadministrative tasks so they could spend more time helpingclientsbecomeself-reliant.TheMinistrybeganmonitoringthefunctionalityofSAMSassoon as it was launched through daily calls to officesprovidingOntarioWorksandODSPservices.InthefirstmonthSAMSwasinuse,theMinistryidentifiedthefollowingissues,allofwhichresultedin caseworkers having less time to help clients:
•About80%oftheprovince’s238OntarioWorksofficesand45ODSPofficesreportedthat caseworkers had to deal with many prob-lemssparkedbySAMS,andthatthereweresignificantissueswithstaffmorale.Almostaquarteroftheseofficesreportedtheywere“unabletocontinueoperationswithoutaddi-tionalsupport.”
•Incontrast,onlyabout20%ofallofficesacross the province reported that they were abletocopewithSAMS.Wenotedthatalmostalloftheseofficeswereinsparselypopulatedareas,withrelativelyfewerclients—andthereforehadfewerSAMS’errorstodealwith.
In addition, some social-assistance services and activitieswerereduced,andimplementationofpolicy changes was delayed.
Caseworkers Had Less Time to Help Clients TheerrorsgeneratedbySAMSshiftedthemajor-ityofcaseworkers’timeandefforttoperforming“workarounds,”whentheycouldhavebeenspend-ingthattimeonprovidingthefullrangeofcase-management services to clients. A workaround is a seriesofstepstobeusedtemporarilytodealwithaSAMSproblemuntiltheproblemispermanentlyfixed.Manyworkaroundsrequireunusualstrat-egiestocoaxorforceSAMStogenerateacorrectresult.
•SAMSincorrectlyrejectedaclientforbenefits.The workaround required caseworkers to overrideSAMS’mishandlingofthefactthat,in the past, the client had been incarcerated foroneday.Thistook17steps.
Workaroundsaddressnotonlycalculationerrors,butalsodefectsinSAMS’functions,suchasreminders.Theprevioussystemhada“tasks”functionthatthecaseworkercouldsetuptoreceiveremindersofwhichcasesrequiredactionseachday.SAMS’equivalentofthisfeaturewasdefective,generatingseveralhundredsofsuchreminders.In addition, numerous reminders were irrelevant because they related to cases the caseworker was notresponsiblefor.
To work around this, caseworkers had to either set up their own reminders on other computer programs(Outlook),orkeepaspreadsheetfileofalltheircasesandcheckitdailytoensuretheywerekeepingupwithtaskssuchasfollowingupondocumentsrequiredtomaintainclient’sbenefiteligibilityorcheckinginwithclients.Maintainingthese duplicate task-reminder systems was time-consuming,andprovidednoefficientwaytoensureaccuracy and timeliness.
Caseworkers Met Clients Less Frequently TwoOntarioWorksofficeswevisitedtrackedappointments with their clients. In one, client appointmentsdroppedfrom612inMay2014,beforethelaunchofSAMS,to325inMay2015,afterSAMSwaslaunched.Intheother,thedropwasfrom862to500betweenthesametwomonths.CaseworkerstoldusthatthesedropsweremostlyasresultofthemhavingtospendextratimetodealwithSAMSproblems.
Social Assistance Services and Activities ReducedWhilecaseworkersfocusoncasemanagement,otherstaffwithmoreexperience(oftenformercaseworkers)deliverservicesmandatedbytheOntario Works ActorprovidedbyspecificOntarioWorksprograms.Theservicestheydeliveraredesignedtohelpclientsbecomemoreself-suffi-cient.Forexample:
TheMinistryalsoputcertainOntarioWorksandODSPactivitiesrequiredbylegislationandregula-tions on hold. Figure 2 provides details on the suspended activities.
4.1.4 Defects Forced Caseworkers to Circumvent SAMS Processes, Undermining Data Integrity
SAMSshouldautomaticallycalculateseparatebenefitamountssuchasshelterorheatingcostsandthen tally them up to issue the client’s total monthly payment.However,defectscausedSAMStoincorrectlycalculatetheseparatebenefitamounts.
483SAMS—Social Assistance Management System
Chap
ter 3
• VF
M S
ectio
n 3.
12
As caseworkers were unable to issue correct pay-ments, they had to work around the problem. Their most common approach was to manually issue a client’smonthlypaymentasan“undefinedbenefit.”Whenissuingundefinedbenefits,thereisnorecordoftheactualbenefittypesclientsarereceiving(suchasshelterorheatingcosts),ortheamounts.Ultimately,thiscircumventionofthenormalpro-cessunderminesSAMS’dataintegrity.
Originally,theMinistrycreatedthe“undefinedbenefit”asalast-resortoptionforcaseworkerstoprovideclientswiththecorrectamountofbenefits.Caseworkerstoldustheywereadvisednottoissue“undefinedbenefits”iftheycouldavoidit,butbyJuly31,2015,theyhadbeenforcedtorelyonitalmosthalfamilliontimestopayoutabout$130million.Asaresult,the$130millioncan’tbetracedbacktotheactualbenefitsclientsarereceiv-ing or their amounts.
Althoughcaseworkerswereissuingfewerundefinedbenefitsbythetimewecompletedouraudit,theoverallimpactofthisapproachonSAMS’data integrity could be irreversible.
4.1.5 SAMS Still Unable to Report Correct Information
succeedinginhelpingclientsgetandkeepjobsandbecomeself-sufficient,thisinformationshouldbevisible.WhilemanyODSPclientsarepermanentlydisabledandwillnotmoveofftheprogram,case-loadinformationhelpstheMinistrytracktrendsinenrolment to help it manage the program.
Monthly Statistics on Social Programs No Longer Available AfterSAMSwasimplemented,theMinistrystoppeditsregularpracticeofpublishingmonthlystatisticsonthenumberofclientsleavingsocialassistance.ThisinformationwouldnormallyhelptheMinistry’sdecision-makersidentifytrendsandanalyzeprogrameffectiveness.Wesoughtthisinformationtoassessthehigh-levelimpactofthereductioninsocial-assistanceservicesafterSAMS’implementation. Through comparisons to the resultsbeforeSAMS,wewantedtoseeifwecoulddetermine whether clients stayed longer on social assistancethantheywouldhaveifthefullrangeofsocial-assistanceservicestohelpthembecome
Figure 2: Ontario Works and ODSP Activities Suspended by Ministry in December 2014Source of data: Ministry of Community and Social Services
Suspended Activity*Ontario WorksUpdating clients’ employment-activity agreements and outcome plans (legislation and regulation requires these to be updated for benefits to be issued)
Recovering benefits from clients who breached their employment-activity agreements (e.g., by not looking for or keeping a job)
Verifying whether existing clients remain eligible for benefits
Ontario Disability Support ProgramVerifying whether existing clients remain eligible for benefits (this includes reconfirming their disability)
Preparing an annual Performance Development and Learning Plan
Collecting overpayments from former clients that are no longer receiving social assistance
* When we completed our audit, the Ministry informed us that it was expecting to resume these activities in fall 2015.
2015 Annual Report of the Office of the Auditor General of Ontario484
Afterseveralrequests,theMinistryprovideduswith these statistics. It indicated, however, that it has not yet completely validated the process used to extractthisinformationfromSAMS,andsohasnotyetmadethisinformationpublic.Wethereforedidnotrelyonthisinformation.
Ministry Could Not Determine Total Reimbursement Amounts to Be Paid to MunicipalitiesTheMinistryandmunicipalitiessharethecostofbenefitspaidtoOntarioWorksrecipients.ToobtainthefundstheyneedtodeliverOntarioWorks,municipalitiesusedtocompleteaclaimformeverymonth detailing the payments issued to clients. TheysubmittedtheclaimtotheMinistryandreceivedalumpsumtocovertheMinistry’sshareofthese costs. The previous system generated a num-berofreportsfromwhichmunicipalitiescouldpulltheinformationneededtocompletetheclaimform.
SinceSAMS’reportingfunctionwasdefectivefromthebeginning,municipalitiescouldnottotaltheir payments and claim reimbursements. As a result,theMinistrystoppedrequestingreimburse-mentclaimsfrommunicipalities,insteadadvancingthem the same amount each month (based on rollingaveragesoftheamountsmunicipalitieshadclaimedforreimbursementinthethreemonthsbeforeSAMSwaslaunched).Itrequestedthatmunicipalitiessimply“advisewhethertheamountsoftheadvancesaresufficientorneedadjusting.”
Municipalities Unable to Reconcile Bank Records with SAMS’ Records Tohelpidentifypotentialfraud,municipalitiesneedto reconcile the social-assistance payments their banksissuewiththerecordsofthosepaymentsinSAMS.However,sinceSAMScouldnotaccuratelyreport on social-assistance payments, this recon-ciliationcouldnotbeperformed.
OnesmallermunicipalitywevisitedattemptedtoreconcilepaymentsmadeinNovemberandDecember2014.SAMS’reportsonthesocial-assistancepaymentsofthismunicipalityforthisperiodindicatedabout$218,000lessintotalpay-ments than were indicated in the bank records. The municipality was able to manually trace about $200,000ofthisamounttopaymentsthatSAMShad registered but did not include in its reports. Theremaining$18,000couldnotbereconciledandjustified,meaningthatitcouldbeeitherlegitimatebutnottraceable,orunauthorized;itisimpossibleto know which until the amount is reconciled.
4.1.6 Policy Change to Improve Social Assistance Delayed
SAMS’problemscontributedtoadelayinimple-menting a policy change meant to improve social assistance programs.
Caseworkers Found Training Not Useful AtonepointduringthedevelopmentofSAMS,theMinistryplannedtolaunchSAMSinMay2014,andscheduledonlinetrainingforcaseworkersbetweenJanuaryandMay2014.Caseworkersinformedusthat the training program repeatedly shut down without warning and had many errors.
The training proved somewhat questionable anywaybecauseSAMShadnotyetbeenfullydeveloped, so the preliminary version used in trainingwasfardifferentfromthefinalversion.This preliminary version also did not contain any converteddatatoallowforsimulationofreal-lifescenarios.ThisleftcaseworkersunpreparedtoworkonSAMSwhenitwaslaunched.
In the months leading up to the launch, the MinistryprovidedadditionalSAMStrainingforcaseworkers. This training was optional and again, converteddataneededtosimulatereal-lifescen-arioswasnotused.TheMinistrydidnottrackhowmany caseworkers completed this training.
Help Desk Staff Had No Working Knowledge of SAMSStaffworkingthemainhelp-deskrightafterSAMSwaslaunchedhadnoworkingknowledgeofSAMSandcouldnotdirectlyhelpcaseworkers.Whilesometransferredcaseworkers’callstotheMinistry,mostjustrecordedthecaller’sinformationregard-ingtheproblemandforwardedittotheMinistry.This did not provide most caseworkers’ with a reso-lutionwhentheycalled,andknowledgeofwhethertheir problem would be addressed.
PoordesignofSAMSwascausingcaseworkerstocontinuetospendaninordinateamountoftimeprocessingtransactionsandperformingotheractivities.Wedescribeallofthesefindingsinthissection,andconcludewithananalysisofthedollarcosts incurred, by municipalities and the province, afterSAMSwaslaunched.
4.2.1 Not All Payment Errors Identified
Untilsuchtimeasseriousdefectsarefixed,SAMS-generated errors will continue to add to the current cumulativetotalofabout$140million(consistingof$89millioninpotentialoverpaymentsand$51millioninpotentialunderpayments).Thisfigureincludesonlythoseerrorsfromdefectsthathavebeenfixed;theMinistrycanonlyquantifythedollar-impacterrorofadefectonceitisfixedbecausethatiswhenSAMSautomaticallyrecalcu-latespastincorrectbenefits.
Forexample,SAMSmayincorrectlycalculatethemonthlybenefitforaclientas$570,whenitshouldbe$600.Afterthedefectisfixed,SAMSrecalculates this amount and reports that the client got$30amonthlessthanheorshewasentitledto.Suchidentifiedpotentialunderpaymentsandoverpaymentsmakeupthe$140million.
TheMinistrydesignedandimplementedmanualworkaroundsforcaseworkerssoacaseworkermayalreadyhaveidentifiedtheerrorandcircumventedthenormalSAMSprocesstoissuethecorrectpay-ment,wellbeforetheMinistryfixedthedefect.Inthis case, the client would have received the correct amountofmoney.
However,theMinistrycannotconfirmifwork-arounds were always applied by caseworkers, and asaresult,theMinistrydoesnotknowwhatpor-tionoftheapproximately$140millionhasalready
2015 Annual Report of the Office of the Auditor General of Ontario486
Chap
ter 3
• VF
M S
ectio
n 3.
12
been corrected by caseworkers. Thus, allowing SAMStoadjustthe$140millionofpastincorrectbenefitscouldreversecorrectionsmadepreviouslyby caseworkers.
ThismeansthattheMinistrymayneverknowhowmuchofthe$140millionwasmanuallycor-rected by caseworkers. It also means that people who received less money than they should have in thepast(becausetheyortheircaseworkersfailedtoidentifytheerror),maynevergetachequefortheshortfall,orlikewiseoverpayments,instanceswhere people received more than they should, may never be collected.
The$140millionwillcontinuetogrowbecauseoftheremainingdefects,anditwillremainanissueuntil caseworkers start their manual review at an unspecifieddate.
Inthefirstmonthafterlaunch,thehelpdeskreceived12,500callsfromcaseworkers.ByJuly2015,ithadreceivedalmost30,000calls.However,manycaseworkerstoldustheystoppedcallingthehelpdeskbecausetheyfounditpointless(as mentioned in Section 4.1.7,help-deskstaffhadnoworkingknowledgeofSAMS).Anydefectsencountered by caseworkers who did not phone the helpdeskarethereforeunknowntotheMinistry.
Whenwecompletedouraudit,theMinistryhadabacklogof11,500callsfromthehelpdeskthatit had not yet reviewed. There was also a backlog
4.2.3 Resources Are Not Sufficiently Dedicated to Fixing Known Defects
Atthetimeofouraudit,ittookonaverage40daystofixaseriousdefect,andonlyCuramconsultants,nottheMinistry,hadtheskillstodoso.However,according to the most recent report we reviewed on Curam’sconsultants’time,consultantswerespend-inglessthanhalfoftheirtime(44%)fixingseriousdefects.Theremaining56%wasspentdevelopingnewenhancementstoSAMS’functions,resolvingad-hocrequestsandtransferringknowledgetoministrystaff.
Defect Making SAMS Vulnerable to Fraud Not Adequately AddressedTheMinistrytoldusinAugust2015thatithadfixed52ofofthe57mostseriousdefects.However,weconfirmedthatoneofthe52,whichmadeSAMSvulnerabletofraud,stillexisted.
Thefixinstalledacontrolwherebycasework-erscouldnotmakechangestofilesnotintheircaseload.However,caseworkersdemonstratedto us that they still had unrestricted access to allofSAMS’approximately900,000clientfiles.
Ministry Not Fully Testing Fixes MinistrystaffdirectlydealingwithSAMSafteritwas implemented told us they were overwhelmed bythesheernumberofinitialdefects.InanattempttostabilizeSAMSinthefirst12weeksafterlaunch,theMinistryinstalledanaverageof10fixesadayintheformofsoftwareupgrades.However,theMinistrywasnottestingthesefixes.TheresultwasthatsomeoftheseuntestedfixescausedSAMStoissue$120,100inincorrectpaymentstoclientsthatcaseworkers were later assigned to recover.
ThemainreasontheMinistrywasnottestingitsfixespriortolaunchwasthat,duetofundingconstrains, contracts were not renewed with those consultantsbestabletotestthefixes.InJuly2015,theMinistryhiredeightnewconsultantstoreplacethepreviousones;however,onlyoneofthenewconsultants had any prior experience and know-ledgeofSAMS.TheMinistryestimateditwouldtakeaboutsixmonthsforthenewconsultantstoreachthesamelevelofproficiencyinSAMSastheprevious consultants.
In addition, much caseworker time has been taken up with dealing with the design innovations ofSAMS,manyofwhichdonotactuallysavemuchtime.Forexample,inputtingclientaddressestakesabout10minutes—fivetimeslongerthanbefore—becauseinformationhastobeinputinto38differ-ent screens.
Figure 3showstheresultsofatimestudycon-ductedbyoneOntarioWorksoffice:sevenmonthsafterSAMS’launch,itwastakingcaseworkersalmosttwiceaslongtoperformessentialdailytaskslike entering client addresses. Figure 3 does not includethefactthatcaseworkers’monthlydetailedreviewofbenefit-paymentaccuracytook20min-utesundertheprevioussystemand2anda1/2hoursunderSAMS,asthefigureonlysummarizesdaily tasks.
4.2.6 Costs to Stabilize SAMS Will Continue to Increase until Defects are Resolved
WesurveyedallOntariomunicipalitiestodeter-minethetotalofthesecosts.Abouthalfthemunici-palities responded to our survey and reported that theyincurred$12.3millioninovertimecostsasofJune30,2015.AstheMinistrydoesnotanticipate
2015 Annual Report of the Office of the Auditor General of Ontario488
To ensure that eligible individuals receive the levelofsocialassistanceandsupporttowhichthey are entitled, and to eliminate as best as possible,eligibilityandbenefitpaymenterrorsmadebytheSocialAssistanceManagementSystem(SAMS),theMinistryofCommunityandSocial Services should:
• assign adequate resources to review the backlogofinformationrelatedtopotentialdefectssothatdefectscanbeprioritizedforfixing;
• developaprocesstoreconcileallbenefitpaymenterrorsgeneratedbySAMStotheeligible amounts that clients should have received and ensure that they are corrected.
• Aftergo-live,theMinistryimplementedanaggressive release schedule to incorporate
Figure 3: Changes in Total Average Time per Caseworker to Perform Daily Essential Tasks Before and After SAMS’ LaunchSource of data: One municipal Ontario Works office visited
* Between December 2014 and March 2015, caseworkers started to perform, for the first time, detailed reviews of daily benefit payments, which accounted for the rise in hours in March 2015. This occurred because caseworkers became aware that daily-benefit-payments could contain errors caused by SAMS’ defects.
0
2
4
6
8
1
3
5
7
9
10
Pre-SAMS Apr 2015
Post-SAMS
Dec 2014 Mar 2015* May 2015 Jun 2015 Jul 2015
Hours
489SAMS—Social Assistance Management System
Chap
ter 3
• VF
M S
ectio
n 3.
12
fixesandenhancements.Morethan90%ofprioritysystemissuesidentifiedbyourstaffand delivery partners have been addressed.
• TheMinistryhasaplantoimproveincidentmanagement in the short term, including theidentificationofanyadditionaldefects.TheMinistryhasincreaseditscapacityandexpertisetoassiststaffinexpeditingtheclean-upoftheincidentticketbacklog,andto triage and resolve new incident tickets more quickly and systematically. TheMinistryhasmovedtoaquarterly
releasecycleforfixeswhichallowsforaddi-tionalengagementwithfrontlinestaffanddelivery partners to appropriately plan and carefullyconsiderandprioritizedefects/enhancements.
TheMinistryhasputadditionalpaymentcontrols in place and is undertaking a review ofitspaymentcontrolprocessestoensurethecontrolsareeffectivelytargetingissues.Any incorrect payments will be corrected and recovered where applicable.
4.3 Consultant Work Inadequate, Not Properly Overseen by Ministry IBMconsultantswereresponsibleforconvertingdatafromtheprevioussystemintoSAMS,andfordesigninganddevelopinginterfacestoenableSAMStocommunicatewithotherexternalcom-putersystems.CuramconsultantswereresponsibleforwritingcodetocustomizeCuram’soff-the-shelf systemtoministrybusinessrequirements(IBMboughtCuraminDecember2011,sofromthattimeforward,allconsultantswereIBMconsult-ants.However,forthepurposesofthisreport,wecontinuetorefertoCuramconsultantsasthosewhodevelopedSAMS,andIBMconsultantsasthosewhoconvertedold-systemdatafortransfertoSAMS,anddevelopedthenecessaryinterfaces.)Consultantsdeliveredpoor-qualityworkthatcon-tributedtodelaysanddefects.TheMinistrydidnotproperlyoverseetheworkoftheconsultants;italso
Data Conversion Critical ForSAMStocorrectlymakeeligibilitydecisionsandprocess payments, it must have the correct data on clients(forexample,correctage,incomeandfamilystatus).OneofthemostchallengingpartsoftheSAMSprojectwastransferringallthedatastoredin the previous system going back two years into Curamsoftware.
Curamsoftwarecouldnotacceptandreadthedataintheprevioussystembecauseofthewaythatdatawasformatted.ThedatathereforeneededtobeconvertedintoaformatthatCuramsoftwarecouldprocessbeforeitcouldbesuccess-fullytransferred.Forexample,theprevioussystemstored postal codes in whatever way caseworkers typedthemin—sometimeswithaspaceinthemiddle, sometimes without, and sometimes even with a dash in the middle. That postal-code data couldnotjustbemovedintoSAMS—itneededtobeconvertedintoastandard,pre-definedformatforCuramsoftware.Alldataneededtobethusconverted,andthisrequiredexpertknowledgeofCuramsoftware.
2015 Annual Report of the Office of the Auditor General of Ontario490
Chap
ter 3
• VF
M S
ectio
n 3.
12
Ministrystaffdidnothavethisknowledge,sotheMinistryissuedarequestforproposalstooutsideserviceproviders.Specifically,theMinistrysought a service provider that:
TheMinistrychoseIBMin2011inpartbecauseIBM’sproposalmettheserequirements.IBMwastobepaidonthebasisofmeetingthetermsofitscontract, as opposed to consultants being paid by thehour.ThecontractstatedthatIBMmustinformtheMinistryifitcouldnottransferclientdataandexplain why.
IBM Did Not Provide Adequate Expertise IBMdidnotdeliverconverteddatatofacilitatethepilottestofSAMSthatwasscheduledforthesummermonthsof2013.Thepilotwasakeypro-jectmilestone,andtherisksoffailingtoconductafullpilotwithconverteddataweresignificant,especiallybecausetheMinistryhaddecidedona“big-bang”approachtoimplementation.IfSAMScouldnotprocessclients’datacorrectlyfromdayone,seriousproblemswouldoccur.Conductingapilotinadvanceoflaunchwassupposedtoreducethisrisk,astheMinistrywouldbeabletoidentifyandfixanyerrorsidentifiedduringthepilot.
IBMrepresentativesandsomeMinistrystaffbothtoldusthatoneofthereasonsIBMcouldnotfinishthejobintimewaspartlyduetothefactthattheMinistrydefinedsomeofitsrequirementsmuchlaterthanexpected.Asaresult,IBMstruggledtoconvertsomedataforsoftwarethatwasstillbeingdeveloped. This in turn required consultants to changewhattheyneededtodotoconvertsomeofthe data.
OnJune17,2013,theMinistryservedarectifica-tionnoticeonIBMthatithadfailedtocomplywiththetermsofitscontract.ThenoticestatedthatIBMhadfailedtodeliverconverteddataontimeanddidnot ensure that its personnel had expertise or up-to-datecertificationwithCuramsoftware.
Figure 4: Project Timeline and Approved Budget, and Subsequent Extensions and Approved Budget Increases ($ million)Source of data: Ministry of Community and Social Services
$202.3 $196.7
$242
$171.4
$220.9
$0
$50
$100
$150
$200
$250
Feb 2010 Mar 2013 Nov 2013 May 2014 Sep 2014 Dec 2014
Original Project Timeline Revision* Extension 2Extension 1 Extension 3
* As a result of further planning, the Ministry revised the original project timeline to November 2013 and the project costs to $171.4 million.
491SAMS—Social Assistance Management System
Chap
ter 3
• VF
M S
ectio
n 3.
12
Essential Pilot Tests Could Not Be Conducted with Converted DataApilotofasubstantiallycompletedsystemwithconverteddatawouldrevealdefectsthatcouldbefixedbeforelaunch.Forexample,ifthecaseworkerspilotingthesystemfoundthatitmadeincorrectbenefit-eligibilitydecisions,calculatedthewrongbenefitpaymentamountsorcrashedwhenused,projectstaffwouldhavemonthstofixtheseprob-lems.Thepilotwouldalsoshowifthenewsystemcould do daily and monthly pay runs with the cor-rect total payment amounts, matching those being issuedbythesystemcurrentlyinuse.NoneofthesetestscouldbeconductedintheSAMSpilotbecausethe converted data was not ready.
TheMinistryscheduledasecondpilotwithconverteddataforlateron.However,becausedata-conversion continued to be delayed, this second pilot was again conducted without converted data. TheMinistrydidsometestingofitsown,butaswediscuss in Section 4.4.4, it was not enough.
SAMS’ Launch Postponed for Second Time Because of Data-conversion Delays OnFebruary18,2014,aboutayearafterIBMwasfirstsupposedtohavedeliveredconverteddata,thethenDeputyMinisterofCommunityandSocialServiceswrotetotheChairman,PresidentandChiefExecutiveOfficerofIBM.TheletterexpressedconcernswithIBM’sinabilitytoabidebythetermsofitscontractandmeetthereviseddeadlinesfordeliveringconverteddata.TheDeputyMinisterstatedthatIBM’sactionswereseverelyimpactingtheoverallprojectandhadforcedtheMinistrytodelaySAMS’launch.
Data Conversion Issues Led to Third Launch Postponement, Budget Increase InJuly2014,itwasclearthattheconverteddataIBMdeliveredinAprilcontainederrorsandthatSAMShadnumerousdefects.TheMinistryreportedthis to Treasury Board, and requested another projectextensionoffivemonthsandabudgetincreaseof$21million.Thiswasneededtoconductmore testing, cover additional resources to support theimplementationofSAMSandmakeitstableenoughforlaunch.(ThisisshownasExtension3inFigure 4.)
Data Transferred into SAMS Had Thousands of Errors after Launch Aftersixmonthsofworkonminimizingtheconverted-dataerrors,thedatawastransferredintoSAMSbetweenNovember6and9,andSAMSlaunchedonNovember11,2014.
•Inabout19,000casefiles,thedataonoutstanding overpayment balances, to be collectedfromclientsbytheMinistry,wasincorrect.
•Inabout10,000casefiles,thedataontrustees(peoplewhomanagethepaymentsofODSPclients who cannot manage their own pay-mentsbecauseofadisability)wasunread-able. As a result, caseworkers had to manually re-entertheinformationsothatpaymentscouldbeprocessedfortheaffectedclients.
2015 Annual Report of the Office of the Auditor General of Ontario492
Chap
ter 3
• VF
M S
ectio
n 3.
12
alerted to this problem by a caseworker who noticeditforsomeclients.Clientscouldlosetheireligibilityforbenefitsbecausetheserec-ordsweremissingfromtheirfiles;topreventthisfromhappening,theMinistryinstalledafixjustbeforethepayrunsotheseclientscouldstillreceivetheirbenefits.
•Inabout7,000casefiles,informationonclientswhopreviouslyreceivedbenefitsasamarried couple but then had separated was mistakenlyre-linked.WiththeerroneousinformationtransferredintoSAMS,therewereseveralinstancesofbreachofprivacywhere,forexample,aclient’saddresswasdisclosed to the client’s ex-spouse.
4.3.2 Ministry Did Not Adequately Oversee Consultants
TheMinistrywasultimatelyresponsible,asSystemIntegrator,forensuringthatallcomponentsneededforSAMSweresuccessfullydeveloped,integratedandtested.However,itreliedmostlyonCuramconsultantstodevelopSAMS—specifically,towritecodetomodifyCuram’soff-the-shelf system so that itmettheMinistry’sbusinessrequirements.Underthecontract,theMinistrywouldpayconsultantsanaveragerateof$190perhourbilled.Tomitigateagainsttheriskofconsultantsclaimingexper-tise they do not actually possess, and purposely delayingworkandmisinformingtheMinistrywhilechargingbillablehours,itwasessentialfortheMinistrytohaveastrategyforstayingontopoftheprojectandensuringitdidnotspiraloutofcontrol.TheMinistrydidnothavesuchastrategy.
Curam Consultants’ Work Mostly Overseen Directly by IBM Consultant TheCuramdevelopmentteamwasdirectlyover-seeninpartbyanIBMprojectmanager.ThisIBMconsultantwasengagedbytheMinistrybetweenMarch2010andApril2013(whenmostsoftwaredevelopmenttookplace)andwaspaidabout$2,000perday,foratotalof$1.3millionforthe
Ministry Did Not Have Adequate Controls to Assess Consultants’ Effectiveness and Efficiency TheMinistryreceivedtimesheetsfromCuramandIBM,buttheinformationtheycontainedwastoovaguefortheMinistrytoknowwhatspecificworkconsultantsdidforthehourscharged;forexample,inthesampleoftimesheetsthatwereviewed,someconsultantsstatedonlythattheywere“triagingconversiondefectsandfixingsomeofthem”or“defectfixing.”Giventhevaguenessofthisreport-ingandthelackofindependentdirectoversightduringmuchoftheproject,itwasdifficultfortheMinistrytoassesshowefficientlyconsultantswereworking.
Curam Consultants Worked Inefficiently Before Launch, Still Not Required to Report Activity to Ministry at Time of AuditInApril2013,theIBMprojectmanagerwastrans-ferredoutofthisroleandtheMinistryhiredanindependentconsultanttooverseeCuramconsult-ants’work.ByNovember2013,thisconsultanthadimprovedthewayCuramconsultants’workwasdocumentedandanalyzed.Ourreviewofdocu-mentsmaintainedbythisconsultantidentifiedthatbetweenNovember2013andMarch2014,CurambilledtheMinistry11,500hours,atanaveragerateof$190/hour,forworkthatwasestimatedwouldtakeabout10,300hours,indicatingthattheywereworkinginefficiently.
4.3.3 Curam Consultant Billings Remained High Because Ministry Staff Unable to Fix Serious Defects
WefoundthatincustomizingCuramsoftware,Curamconsultants’workcontainederrorsthatcreatedabout5,100defects.Forthemostpart,thesame consultants were the only ones who could fixthedefectstheycreated.Furthermore,asmen-tioned in Section 4.2.4,somefixesengenderedfurthererrorsthatalsoneededfixing.TheMinistrywasbilledfortheextensivetimeconsultantsspentcorrecting their own errors.
TheMinistry’sstaffof11developerswerebecomingmoreknowledgeablewhenwefinishedour audit, but we were still surprised at how slowly theywerelearningtofixevenminorproblems.Ourreviewidentifiedthatinthefirstninemonthsafterlaunch,these11staffresolvedonly257minorprob-lems.Thistranslatesintoonly2.5fixesofminorproblems per month per developer.
• assess and document the progress in achiev-ing these targets.
FromMarch2014on,theindependentconsult-antstoppedassessingifworkwasdoneefficientlyorevenontimebecausedevelopmentofSAMSwasessentially complete and consultants were mainly workingonfixingdefects.TheMinistrymadeadecisionnottoassesshowefficientlythisworkwasbeingperformed.
Atthetimeofouraudit,theMinistrytoldus it intended to resume assessing consultants’ efficiency.
RECOMMENDATION 2
To prevent unnecessary delays in bringing the SocialAssistanceManagementSystem(SAMS)tofullandeffectivefunctionality,andtoensurethattheconsultantsstillworkingonSAMSareheldaccountablefordeliveringqualityresults,theMinistryofCommunityandSocialServicesshould:
• onfutureprojects,worktowardsreducingits dependence on consultants, and ensure consultants’knowledgeistransferredtoministrystaff.
MINISTRY RESPONSE
TheMinistryagreeswiththerecommendationandwillimprovethemonitoringofconsultantsto ensure the desired results are achieved. In doingso,theMinistrywillassesstheefficiencyandeffectivenessofworkperformedbyconsult-ants, and take corrective action where necessary to ensure deliverables are provided within speci-fiedtimeframesandmeetqualitystandards.
2015 Annual Report of the Office of the Auditor General of Ontario494
Chap
ter 3
• VF
M S
ectio
n 3.
12
MINISTRY RESPONSE
TheMinistryagreeswiththerecommendationand,aspartofitsIntegratedTransitionPlan,hasensuredthataknowledgetransferstrategyis in place and that there is appropriate capacity forongoingmaintenanceandimprovementofSAMSoverthelong-term.Thisstrategyconsistsof:
• assessingstaffreadiness;
• providing the required product training and supporttools;
WealsobelievethatcreditingIBMforfreeservicesshouldbeoffsetbythefactthat,despiteissueswiththequalityofIBM’swork,theMinistryreducedIBM’sperformancewarrantyperiod(forallservicesexceptforinterfaces).Specifically,whentheprojectmisseditsMay2014launchdate(asdiscussed in Section 4.3.1),theoriginalcontractwithIBMhadtobeextendedtothenewlaunchdateofNovember11,2014.Becauseanewcontractwouldhavetobesigned,IBMhadanopportunitytoaskforanincreaseinconsultants’hourlyrates,whichtheMinistrywantedtoprevent.TheMinistrythereforereducedtheoriginalcontract’sperform-ancewarrantyperiodof150daystojust40daysinthenewcontract,expiringDecember21,2014.Inexchange,IBMdidnotrequestanincreaseinthealreadyhighhourlyratesofitsconsultants.
Theresultofthewarranty-periodreductionwasthatbetweenDecember21,2014,andApril11,2015,theMinistrypaidconsultantstofixdefectsthat would have been covered under the original termsofthewarranty.Howeverthespecificdollaramounts related to this could not be determined.
4.4 Ministry Overly Optimistic about SAMS’ Readiness for Launch InearlyNovember2014,theMinistryknewthatSAMSwasnotfunctioningasitshouldbeforaNovember11launchtobesuccessful.TheExecutiveCommitteedecidedtoproceedwiththelaunchany-way,believingthatitandprojectstaffknewenoughaboutSAMS’defects,thatcaseworkersandotherministrystaffwouldbeabletomanagetheimpactofthosedefectsafterlaunch,andthatpaymentstoclientswould,forthemostpart,beaccurate.Webelievethiswasanunreasonablyoptimisticviewpoint.
495SAMS—Social Assistance Management System
Chap
ter 3
• VF
M S
ectio
n 3.
12
Inthissection,wepresentthe“go-live”criteriathattheMinistryhaddevelopedasthefinalhurdleforSAMStomeetbeforelaunch,aswellasSAMS’knownperformanceagainstthosecriteria(Sec-tion 4.4.1);wediscussotherimportantrequire-mentsthattheMinistryknewSAMSdidnotmeet(Section 4.4.2);wereviewtherisksthattheExecutiveCommitteetoldusitweighedtodecideto launch given what it knew (Section 4.4.3);wepresentfurtherdeficienciesinSAMSthatprojectstaffdidnottelltheExecutiveCommitteeabout(Section 4.4.4);andweconsideradditionalfactorsthatcontributedtotheMinistry’sfaultyassumptionthatanyproblemsencounteredafterlaunchwouldbefairlyeasytomanage(Section 4.4.5).
Terminology for This SectionThroughoutthissection,wewillbereferringtocertain parties shown in Figure 5.Specifically:
• The Executive Committee:theproject’sover-seeing body. It made the decision to launch basedoninformationitreceivedfromprojectstaff.
Figure 5: The Key Parties That Developed and Launched SAMSSource of data: Ministry of Community and Social Services
Executive CommitteeCo-chairsDeputy Minister—Community and Social ServicesCorporate Chief Information Officer—Ontario Public Service
Other MembersAssistant Deputy Minister—Social Assistance OperationsCluster Chief Information Officer—Children, Youth and Social ServicesAssistant Deputy Minister—Social Assistance PolicyAssistant Deputy Minister—Business Planning and Corporate ServicesSAMS Executive LeadDirector—Ontario Internal Audit1
SAMS Executive Lead
Business Project Director
Various Teams
Executive Committee
Project Staff
Testing Team2 Various Other Teams
Technical Project Director
Change in reporting lines May–November, 2014
Change in reporting lines May–November, 2014
1. The Internal Audit Director played an advisory role to the Executive Committee. The Internal Audit Director was not a voting member in making the decision to launch SAMS.
2. From May to November 2014 (that is, the six months prior to SAMS’ launch), the testing team reported to the Business Project Director, not the Technical Project Director. After launch in November 2014, the testing team resumed reporting to the Technical Project Director.
2015 Annual Report of the Office of the Auditor General of Ontario496
Figure 6: SAMS’ Performance Information Reported to the Executive Committee in October 2014Source of data: Ministry of Community and Social Services
Requirement Under Go-live Criteria Reported Status1. 90% of planned user-acceptance tests done 81% of planned user-acceptance tests done1
2. 90% of user-acceptance tests produce right results 86% of user-acceptance tests produced right results2
3. 0 serious defects in test results 1 serious defect still existed
4. 100% of SAMS-generated payments traceable to previous system
98% of SAMS-generated payments traceable to previous system
5. SAMS generates 100% of payments generated by previous system
SAMS generated 99% of payments generated by previous system
6. 0 discrepancies in payment amounts between previous system and SAMS
5,500 discrepancies in payment amounts between previous system and SAMS3
7. Daily pay run takes less than 1 hour; monthly pay run takes less than 6 hours
Daily pay run took 11.5 hours; monthly pay run took 11.5 hours
8. 0 serious defects when processing pay runs 0 serious defects when processing pay runs
9. 100% of test scenarios executed 95% of test scenarios executed
10. 100% of test scenarios executed using converted data 0% of test scenarios executed using converted data
11. 100% of interface testing with third parties done 70% of interface testing with third parties done
12. No batches skipped while running all six batch groups; 0 batch groups to exceed expected time frames
Reports batches skipped while running all six batch groups; 3 batches exceeded expected time frames (taking 13 hours, 13.5 hours and 21.75 hours)
13. 100% of routine tasks take 3 seconds or less 75% of routine tasks took 3 seconds or less
14. SAMS takes under 3 seconds to reassess client eligibility after changes made to case
SAMS took between 3 and 10 seconds to reassess client eligibility after changes made to case
15. SAMS takes under 7 seconds to search cases by name and status
SAMS took 8 seconds to search cases by name and status
16. 0 data-conversion errors found in 14 fields checked Not reported 4
17. 0 client's eligibility or payment amount should be impacted by uncorrected data-conversion errors
15,824 clients' eligibility or payment amount were impacted by uncorrected data-conversion errors
18. 0 serious defects in converted data 7 serious defects in converted data
1. As explained in Section 4.4.4, our audit work determined that only 74% of planned user-acceptance tests were done, not 81% as reported to the Executive Committee.
2. As explained in Section 4.4.4, our audit work determined that less than 86% of user-acceptance tests produced right results. Testers did not track the number of times they misrepresented right results, so the percentage cannot be determined.
3. As explained in Section 4.4.4, our audit work determined that there were 52,000 discrepancies in payment amounts, not 5,500 as reported to the Executive Committee.
4. Project staff did not report the number of errors found in the 14 fields checked. Instead, it reported that “all discrepancies are identified and remedial action documented.” Our review found that almost 30,000 data-conversion errors were found in the 14 fields. The Ministry’s remedial actions were for caseworkers to manually fix these errors.
497SAMS—Social Assistance Management System
Chap
ter 3
• VF
M S
ectio
n 3.
12
wereassessed.Someofthisplannedtestingdidnottake place. In addition, the go-live criteria did not include several important requirements that should havebeenconsidered.Wedescribebothinthefol-lowing subsections.
Some Planned Testing Did Not Take PlaceNo Pilot Conducted with Converted Data; Limited “User Acceptance Testing” Identified that More Test-ing Was Needed
As discussed in Section 4.3.1,aplannedfullpilotofSAMSnevertookplacebecauseconverteddata(fromtheprevioussystemintoSAMS)wasnotavailableintime.Thus,theversionofSAMSthatwas launched was never piloted with actual data.
Useracceptancetestingshouldbeconductedwithfullytrainedcaseworkers,performingtheiractualday-to-dayactivities.FortheSAMStesting,however,thecaseworkerswerenotfullytrainedonSAMS,andtheversionofSAMStheytestedwasincomplete: it was not able to generate payments, generatereportsorinterfacewithothercomputersystems,sincethesefunctionswerenotyetreadyand had not been installed.
16% of 1,772 Business Requirements Not Tested; Test Failure Rate Was One in EightSAMSmustbeabletocorrectlyperformmyriadfunctions,includingdeterminingeligibility,cal-culating payment amounts, generating letters and reports,andinterfacingwithothercomputersys-tems.OfallthebusinessrequirementstheMinistrydefinedforSAMS,aboutone-thirdrelatedtoreportandlettergenerationandinterfacing.Projectstafftestedthosefunctionsthoroughly,butdidnotcom-pletelytestSAMS’otherfunctions.Itleft16%ofthebusinessrequirementsforthoseotherfunctionsuntested.
Theoverallaveragefailurerateintestresultswas13%,oroneineight.Figure 7 shows these results.Inotherwords,13%ofSAMS’functionswere not working as intended.
Important Criteria Not Included in Go-live CheckGo-live Criteria Did Not Specify Overall Acceptable Number of Serious Defects; There Were Hundreds at LaunchTheMinistry’slaunchstrategystated,“...thesolu-tiontobedeliverediscomplexwithsignificantrisktotheMinistryifthesolutionisimplementedwithdefects”andthatSAMSshouldbeimplemented
Figure 7: Test Results for Different SAMS FunctionsSource of data: Ministry of Community and Social Services
Function Total Tests Unsuccessful Tests Failure Rate (%)Generate a correct report 2,579 700 27
Generate a correct letter 8,574 1,384 16
Interface successfully 4,006 554 14
Other functions (e.g. eligibility determination, calculation of payment and other case management functions)
8,114 293 4
Total 23,273 2,931Overall Failure Rate 13
2015 Annual Report of the Office of the Auditor General of Ontario498
IfSAMSweretolaunchwithalotofseriousdefects,itwouldnotmatterifSAMSmetalltheother go-live criteria. Given the big-bang approach toimplementation,theimpactofalargenumberofseriousdefectswouldoutweighanyfunctionalitySAMShad.AsindicatedinFigure 6, the go-live criteriaspecifiedonlythefollowingwithrespecttoseriousdefects:
•Seriousdefectsfoundinjusttheuseraccept-ance testing should be resolved and retested (criterion3,whichwasnotmet:oneseriousdefectstillexistedatthego-livecheck).
Infact,projectstaffknewofmanymoreseriousdefectsthanwhatwasreportedtotheExecutiveCommittee.WediscussthisinSection 4.4.4 and Section 4.4.5. The point here, however, is that, not-withstandingthedefectstheExecutiveCommitteedidnotknowabout,theExecutiveCommitteemadethe decision to go live with the understanding thattherewere418seriousdefectsandthatwork-aroundshadbeenpreparedforonly217ofthem.
Government-mandated Payment Testing Not Part of Go-live Criteria: Incomplete Tests Resulted in 28 Ser-ious Defects After LaunchSince2005,thegovernmenthasmandatedthatthecomputersystemforanyprogramthatprovides
payments must pass certain tests to ensure that paymentsandchequestubsareaccurate.SAMSwasnotinastableenoughconditiontopassallofthesetests, and only some testing was conducted.
This requirement was triggered by an incident thatoccurredin2004.About27,000OntarioChildCareSupplementchequesforNovember2004wereprinted with the name, address and Social Insur-anceNumberofthewrongrecipient.Topreventsomethinglikethisfromhappeningagain,thegovernment now requires that any computer system that issues direct deposits and cheques through the government’s payment processing system (the Inte-gratedFinancialInformationSystem,orIFIS)mustundergo mandatory testing.
SAMSinterfaceswithIFISforODSPpayments,so it should have undergone the mandatory testing. AccordingtotheOfficeoftheProvincialController,SAMSistheonlycomputersystemeverconnectedtoIFISthathasnotdoneso.
Thegovernment’sEnterpriseFinancialServicesandSystemsDivision(EFSS),whoperformsthistest,didalimitedtesttoensurethatSAMS’pay-mentfilewouldnotcrashIFISandthatIFISwouldissuethepayments.However,EFSScouldnotper-formalloftherequiredtests.Asaresult,significantrisks, such as the risk that a payment could go to the wrong person, remained untested.
WhenSAMSwaslaunched,thefirstpayrunessentiallyrepresentedacompletetestcycle.Dur-ingthispayrun,28seriousdefectswerefound,mostofwhichtheMinistrylabelledasthehighestseverity that had broad system-wide impacts and resulted in some clients’ not getting paid, duplicate payments being issued, and noticeable errors in printed cheque stubs.
WealsonotedthatafterlaunchingSAMS,whenmajorsoftwareupgradesareinstalled,SAMSis required to undergo the same government-mandated payment testing. This is because softwareupgradesriskmakingunwantedchangestopaymentfileswhichcancreateerrorsandcomplications while processing payments. This riskcanonlybemitigatediftheMinistryconducts
ToensurethattheSocialAssistanceManage-mentSystem(SAMS)reachesthehighlevelofperformanceintendedandthatitfunctionsincompliance with government requirements, theMinistryofCommunityandSocialServicesshouldensurethatSAMSundergoesandpassesall government-mandated payment testing.
MINISTRY RESPONSE
TheMinistryagreeswiththerecommendationandrecognizesthatthestrategytakenforthiselementofpaymenttestingrepresentedacon-densed approach to address key risk.
TheMinistrywillworkwiththeOfficeoftheProvincialController,TreasuryBoardSecretar-iatandtheMinistryofGovernmentandCon-sumerServicestoensurethatcurrentandfutureSAMSchangesrequiringgovernment-mandatedpaymenttestingwilladheretothefullend-to-endprocess.Wewillsharelessonslearnedinthis regard across government.
4.4.3 Executive Committee’s Rationale for Launching an Unready SAMS
TheExecutiveCommitteetoldusthatitconsideredthefollowingrisksifitdelayedthelaunchofSAMSto be greater than launching a system that was not fullyready:
• The next available launch date would have beenspring2015.Thiswouldbetheearli-esttimethatafour-dayshutdownoftheexistingsystemcouldbescheduledfordatatransferwithminimaldisruptiontoOntarioWorksandODSPservices.However,MinistrycontractswithODSPcaseworkersweredue
fornegotiationinspring2015.Therewasthereforeariskthatthelaunchwouldbepre-empted by labour negotiations and would havetobedelayedevenfurther.
•TheMinistrytrainedcaseworkersonSAMSinMay2014,sixmonthsbeforetheplannedlaunch.AlthoughtheMinistryprovidedaddi-tional, optional online training in the months followingtheoriginaltrainingandcouldcontinuetodoso,theExecutiveCommitteebelieved that delaying the launch to spring 2015posedariskthatcaseworkerswouldhaveforgottentheirtrainingandwouldfinditmuchhardertouseSAMSthanascomparedtoNovember2014.
• Pushing back the launch date would require theMinistrytoaskthegovernmentforanotherprojectextensionandmoremoney(the amount requested would be consider-able—everythree-monthextensioncostsabout$20million).BecausetheMinistryhadalready done this three times and still did not haveafullyfunctioningsystem,theybelievedthat,coupledwiththeotherfactors,therewas a possibility that the government could refuseanddecideinsteadtocanceltheprojectaltogether, cut its losses and start over again.
Project Staff Did Not Disclose All Defects to Executive CommitteeProjectstafftoldtheExecutiveCommitteethatSAMShad418seriousdefects,andthat217ofthemcouldbehandledbyjust27workarounds.However,wefoundthatSAMSactuallyhad737seriousdefects.
Project Staff Conducted Fewer Tests than Reported, Results Incorrectly Stated ProjectstafftoldtheExecutiveCommitteethat81%ofplannedtestscenarioswereexecuted.Go-livecriterion1wasfor90%oftheplannedtestscen-arios to be executed (see Figure 6).TheExecutiveCommitteefeltthatmissingthecriterionbyjust9%wasanacceptablerisk.However,ourreviewofdocumentationfoundthatonly74%ofplannedtestscenarios were executed.
only74%ofthetestscenariosintheoriginal,com-pletetestplanwereexecuted,81%ofthetestscen-arios in the revised, shorter test plan were executed.
ProjectstafftoldtheExecutiveCommitteethatSAMSproducedtherightresultin86%ofthetestscenariosexecuted.Go-livecriterion2wasfor90%oftestscenariosexecutedtoproducetheright results (see Figure 6).TheExecutiveCom-mitteefeltthatmissingthecriterionbyjust4%wasanacceptablerisk.However,welearnedininterviewing the testers that wrong results were counted as right results as long as the wrong result wascausedbyaknowndefect.Testerswerealsonotaskedtorecordhowmanyresultstheydidthisfor,sotheactualpercentageoftestscenariosproducingthe right result is not known.
Projectstafftoldusthatinthesecases,theyknew why the wrong result occurred and once the defectwasfixed,SAMSwouldproducetherightresult.ThisviewmightbedefensibleiftheMinistryhadfixedalldefectsbeforelaunchingSAMS,butitdid not.
Executive Committee Did Not Know One in Eight Interfaces Not TestedInterfacetestingwiththirdpartiesdeterminesifclientdatatransferscorrectlybetweenSAMSandexternalcomputersystems.Projectstaffdidnottest11ofthe85interfaces,oraboutoneineight,andtheExecutiveCommitteedidnotknowthis.
Oneinterfacethatwasnottestedinformscase-workersofindividualswhoareservingaprisonsentenceandshouldnotreceivebenefits.Asthisinterfacedidnotfunctionproperly,caseworkerswouldnotreceivethisinformationpromptlytostopbenefitpaymentstoincarceratedclients.TheMinistryissuedaworkaroundfortheproblemandnotifiedcaseworkersaboutitinMay2015.Bythistime,however,SAMShadcommunicatedincorrectinformationinmorethan25,000notificationsregarding incarcerated clients. As a result, there was a high risk that payments to incarcerated clientshadnotstopped.Whilesomeofthesepay-ments may have been stopped because caseworkers
501SAMS—Social Assistance Management System
Chap
ter 3
• VF
M S
ectio
n 3.
12
becameawareoftheincarcerationsthroughothermeans,wefoundseveralcasesduringoursitevisitswherethepaymentshadnotbeenstopped.Forexample,oneprisonerreceived$466.58inbenefitpaymentswhileincarceratedthatwouldtake16months to recover.
The74interfacesthatweretestedweredividedinto17testgroups.Beforelaunch,almosthalfofthe groups had either not been completely tested orhadunresolvedtransmissionerrors.Oneinter-faceenabledclientinformationtobetransferredtoexternalserviceprovidersthathelpODSPrecipientsfindjobs.SincethisinterfacehadseveraltransmissionerrorswhenSAMSwaslaunched,ser-viceprovidersfacedtheriskofnothavingtherightinformationtocontacttheseODSPclientsontime.
Executive Committee Did Not Know that Pay Runs Not Fully Tested TheMinistrytestedSAMS’abilitytoissuecorrectpayments several times prior to implementation by comparing its payments with those issued by the previoussystem.ThelasttestwasofthemonthlypayruninOctober2014,onemonthbeforelaunch.Themonthlypayrunissues616,000paymentstoclients,orabout77%ofallclientpayments.Theseareforthemanyrecurringbenefitsissuedinsetmonthly amounts.
Theremaining23%ofpaymentsareissuedinadailyrunandincludeone-timebenefitpaymentsforthings like medical supplies and advance payments. About186,000daily-runpaymentsareissuedpermonth,totallingabout$78million.
TheExecutiveCommitteethoughtbothrunshadbeenfullytestedbeforelaunch.Itdidnotknowthat only the monthly pay run - and not the daily payrun-wasfullytested,andthatprojectstaffwere,therefore,notfullyawareofwhattheresultswouldbewhenSAMSdidthefirstlivedailypayrun (although daily and monthly pay runs can issue similar payments, certain payments are issued only duringthedailypayrun).
TheExecutiveCommitteealsothoughtthemonthlypayrunhadbeenfullytested,butitwasnot.Becausesomeinterfacesandthefunctionthatsuspends payments under certain conditions were stillnotfunctioninginOctober2014,about26,000paymentsinthemonthlyrunwereexcludedfromtesting.
Figure 8showsadetailedbreakdownofthetesting and the results.
Figure 8: Breakdown of Pay Runs into Monthly and Daily Payments, and Testing Results, October 2014Prepared by the Office of the Auditor General of Ontario
Paymentsincorrect
(8%)
Paymentscorrect(88%)
Monthly-runpayments
(partially tested)(77%)
Daily-runpayments
(not tested)(23%)
Paymentsnot tested
(4%)
Breakdown of Monthly and Daily Payments Monthly Pay Run Test Results
2015 Annual Report of the Office of the Auditor General of Ontario502
Chap
ter 3
• VF
M S
ectio
n 3.
12
TheExecutiveCommitteethoughtthattestershadfoundonlyabout6,000discrepancies(thatrelatedtoSAMS’defects)betweentheactualmonthlypayrunoftheexistingsystemandSAMS’test run. It believed that case workers would be able toworkaroundthesediscrepancieswhenSAMSdiditsfirstlivepayrunandensureclientsreceivedthecorrectbenefits.
Executive Committee Did Not Know that Converted Data Was Not Fully Tested And How Many Errors It ContainedTheMinistryspecificallyidentifieddatain51screenfieldsthatshouldbecleanedupfromthe previous system so that when this data is transferredintoSAMS,itdoesnotcauseerrorsinprocessingclients’eligibilityandbenefitpayments.Ideally,converteddatashouldbecheckedinall51fields.However,theMinistrytestedconverteddatainonly14prioritizedfields.TheExecutiveCommit-tee did not know this.
• Breaches of client privacy:Whetheraclient’s address was correctly converted into SAMSwasnottested.Caseworkerstoldusofseveral instances where improperly converted data led to client addresses being disclosed to people who should not have access to this
privateinformation.Forexample,theMin-istryestimatedthatinabout7,000cases,IBMdata converters mistakenly re-linked together thefilesofpreviously-linkedindividualssuchasdivorcedcouples.WhentheirdatawastransferredintoSAMS,theex-spouses’personalinformationwasavailabletooneanother. In one example, an abusive ex-husbandgainedaccesstohisex-wife’saddresswhenheappliedforsocialassistance.
• Deletion of 78,000 income records: Whetherclients’historicalincomeinforma-tionwascorrectlyconvertedintoSAMSwasnot tested. As mentioned in Section 4.3.1, IBMdataconvertersincorrectlydeleted78,000suchrecordswhentransferringthemtoSAMSclientswhoserecordswerelostwouldnothavebeeneligibleforbenefits,andSAMSwouldautomaticallysuspendtheiraccounts.Topreventthisfromhappening,theMinistryinstalledafixjustbeforethepayrun so these clients could still receive their benefits.
4.4.5 Project Staff and Executive Committee Assumed a High Degree of Risk
As mentioned in Section 4.4.3,theExecutiveCom-mitteetoldusitthoughtinOctober2014thatitwastooriskytoaskthegovernmentforpermissiontopostponethelaunchofSAMSbecauseSAMSwasnot ready. It worried that the government might finallyputastoptotheproject,whichwasincreas-ingly over-budget and had already been repeatedly postponed.However,launchingSAMSbeforeitwas ready was also very risky. As this report has described,itresultedinthefollowingproblems,noneofwhichhadbeenfullysolvedwhenwecom-pleted our audit:
magnifiedtheriskthatprojectstaffandtheExecutiveCommitteeassumed.Inourview,theycontributedtowhatprovedtobeanunjustifiedoptimismthattheSAMSlaunchwouldgoreason-ably smoothly and that any problems would be manageable.
Roles and Responsibilities Shifted Just Prior to LaunchSixmonthsbeforelaunch,therewasashiftintheroles, responsibilities and reporting relationships ofthebusinessandtechnicalteams.ThetechnicalstafftestingthereadinessofSAMSmovedfromreportingtotheTechnicalProjectDirectortoreportingtotheBusinessProjectDirector.
TheMinistryinformedusthatthechangeintesting responsibility was made to resolve workload issues.Howeverwenotedthatboththebusinessteam and the technical team were equally occupied withpreparingforlaunch.WealsonotedthatafterthelaunchofSAMS,thetestingteamwentbacktoreportingtotheTechnicalProjectDirector.
TheresultwasthattheexpertiseoftheTech-nicalProjectDirectorwasnotconsideredwithrespect to test completion and test results when SAMS’readinesswasassessedandthedecisiontogolivewasapproved.TheBusinessProjectDirectorwhooversawthetechnicalstaffconductingthelastsixmonthsoftestingbeforelaunchdidnothavetheTechnicalProjectDirector’stechnicalexpertise.Thissix-monthperiodofchangedrolesandrespon-sibilitieswascritical,becauseinformationaboutthetestsconductedonSAMSandtheirresultswasgatheredandsharedwiththeExecutiveCommitteeduring this time to support the decision to launch SAMS.
Internal Audit Did Not Audit SAMS’ Readiness for Launch
All ministries are served by an internal audit team thatispartoftheOntarioInternalAuditDivision(InternalAudit).InternalAudit’smandateistohelpministriesachievetheirbusinessobjectivesbypro-actively consulting on, evaluating and improving risk management, controls and governance.
Internal Audit did conduct audit work on the SAMSproject,butthelastreportitissued,inNov-ember2013,wasafullyearbeforelaunch.Whenwe completed our audit that was the last audit they had conducted.
Executive Committee Accepted the High Degree of Risk Based on Risk Assessment Conducted in October 2014 Theprojectteam(withtechnicaltestingstillreportingtotheBusinessProjectDirector)andanIBMseniorexecutiveconductedariskassessmentduringthelasttwoweeksofOctober2014.Theyconcludedthattherisksoflaunchingasystemthatdidnotfullymeetthego-livecriteriashouldbeaccepted and launch should go ahead.
Seniorprojectstaffandothermembersofthe executive committee presented the results against the go-live criteria and the accompanying riskassessmenttotheco-chairsoftheExecutiveCommittee,whoseapprovalwasneededtolaunchSAMS.
2015 Annual Report of the Office of the Auditor General of Ontario504
Chap
ter 3
• VF
M S
ectio
n 3.
12
RECOMMENDATION 5
In order to improve the decision-making process usedtolaunchamajorinformationsystem,theMinistryofCommunityandSocialServicesshould:
• ensurethatthedecisiontolaunchaninfor-mation technology system is based on rel-evantcriteriaandinformationthatprovidesdecision-makers a complete and accurate statusofsystemreadiness;and
• have Internal Audit independently review keyinformationusedinassessingthesys-tem’sstateofreadinesswhilemakingthedecision to launch.
MINISTRY RESPONSE
TheMinistryagreeswiththerecommendationandwillensurethatallinformationthatisprovided to decision makers will include a com-pleteandaccuratestatusofsystemreadinessthat is independently assessed by an internal audit.Inaddition,theMinistrywillensureanincreasedroleforinternalauditinassessingthereadinessofmajorinformationsystemspriorto making a decision to launch a system. This would be in addition to the advisory role played bytheExecutiveCommitteeandinternalauditonspecificengagementsearlierintheprojectlifecycleofSAMS.
505SAMS—Social Assistance Management System
Chap
ter 3
• VF
M S
ectio
n 3.
12
Appendix—Example of a Letter with Incorrect Information Sent to an ODSP ClientSource of data: One ODSP office visited
