““Making your data a strategic asset”Making your data a strategic asset” Data Quality - The Key to Successful Analytics and CRMData Quality - The Key to Successful Analytics and CRM
Michael CollinsMichael Collins BA(Hons), DipM, MCIM, FIDMBA(Hons), DipM, MCIM, FIDM
Managing Consultant - Database Marketing CounselManaging Consultant - Database Marketing CounselVisiting University Lecturer in Database Marketing & CRMVisiting University Lecturer in Database Marketing & CRM
• The importance of understanding your core systemsThe importance of understanding your core systems • How clean should your data be?How clean should your data be? • The impact of a data auditThe impact of a data audit • Where does poor data come from? Where does poor data come from? • Methodology for improvementMethodology for improvement• Getting employees to ‘live and breath’ data qualityGetting employees to ‘live and breath’ data quality
Data as a corporate assetData as a corporate asset
• Value to the businessValue to the business
• Invest in the maintenance of an assetInvest in the maintenance of an asset
• Responsibility of all who use it, access it, are Responsibility of all who use it, access it, are involved in its acquisition, storage or involved in its acquisition, storage or maintenancemaintenance
• Rules of managementRules of management
• ValidationValidation
• SecuritySecurity
• An appreciating asset – in everyone’s interestAn appreciating asset – in everyone’s interest
Implications of Inaccurate DataImplications of Inaccurate Data
• There is no substitute for acquiring accurate data There is no substitute for acquiring accurate data - analysis tools can’t compensate for lack of data- analysis tools can’t compensate for lack of data
• The more “real time” contact we have with The more “real time” contact we have with customers, suppliers or employees, the more customers, suppliers or employees, the more accurate the data needs to be and the more accurate the data needs to be and the more devastating can be the results of inaccuracydevastating can be the results of inaccuracy
• Quality will determine how much of a guide or Quality will determine how much of a guide or ‘black & white’ analysis can be reached‘black & white’ analysis can be reached
Implications of Inaccurate DataImplications of Inaccurate Data
• Skewed campaign planningSkewed campaign planning• Improper selections for campaignsImproper selections for campaigns• Expensive product mistakesExpensive product mistakes• Non-delivery of the message (esp. E-mail)Non-delivery of the message (esp. E-mail)• Reflection of your businessReflection of your business• ‘‘Junk mail/spam’ tagJunk mail/spam’ tag• £££££ Wasted£££££ Wasted
• Score the data as part of your data strategyScore the data as part of your data strategy• Build a model that provides a level of confidenceBuild a model that provides a level of confidence• Base the model on known factorsBase the model on known factors
— SourceSource— Recency of updateRecency of update— TestingTesting
• Use the score to determine priorities for enhancement Use the score to determine priorities for enhancement and to inform the business of the level of confidenceand to inform the business of the level of confidence
• Strive to improve the level of confidenceStrive to improve the level of confidence
• Data Audit -technologyData Audit -technology• What needs fixingWhat needs fixing• What needs summarizingWhat needs summarizing• What derived data is requiredWhat derived data is required• Attrition - data use by date!Attrition - data use by date!• How do you fix and improve it How do you fix and improve it
— external enhancement, internal technologyexternal enhancement, internal technology• Data business rulesData business rules• What to do while you are fixing it!What to do while you are fixing it!• Keeping it fixed – monitor and enhance!Keeping it fixed – monitor and enhance!
Data AuditData Audit• Appraise the dataAppraise the data
— Technology for auditing the dataTechnology for auditing the data— Do fields hold what they claim to hold?Do fields hold what they claim to hold?— Is it in a usable formatIs it in a usable format
• For operations?For operations?• For analytics?For analytics?
— How extensively populated are the fields?How extensively populated are the fields?
• Ascertain the age of the data – has it passed its ‘USE BY Ascertain the age of the data – has it passed its ‘USE BY date’?date’?
• What needs to be done to make this data usable/valuableWhat needs to be done to make this data usable/valuable
Data Audit TechnologyData Audit TechnologyTools to report on the quality of data - attention is drawn to those fields that require analysis.Tools to report on the quality of data - attention is drawn to those fields that require analysis.
Against each column name Against each column name • Minimum Value and Maximum Value Minimum Value and Maximum Value • Mean, Median, ModeMean, Median, Mode• Minimum Length and Maximum Length Minimum Length and Maximum Length • Mean and Mode Length Mean and Mode Length • Defined Data Type and number/% records that conflictDefined Data Type and number/% records that conflict• % populated with valid characters (excluding spaces)% populated with valid characters (excluding spaces)• Number of unique valuesNumber of unique values
Data Report – Logistics CompanyData Report – Logistics Company
Example of some of the data irregularities identified – addresses in the name field, addresses and Example of some of the data irregularities identified – addresses in the name field, addresses and postcodes in the Town field, lower case characters, invalid postcodes etcpostcodes in the Town field, lower case characters, invalid postcodes etc
How to Fix what needs Fixing!How to Fix what needs Fixing!
• Internal processesInternal processes— Data cleansingData cleansing— CorrectionsCorrections— Use of address enhancement software Use of address enhancement software — Use of touch-pointsUse of touch-points— Use of people in the business who knowUse of people in the business who know— Source evaluationSource evaluation
Common functions of data cleansing technology Common functions of data cleansing technology • Find and ReplaceFind and Replace• Standardisation: compare values from the given Standardisation: compare values from the given
column with a column in a compiled Knowledge column with a column in a compiled Knowledge Base e.g. list of TitlesBase e.g. list of Titles
Common functions of data cleansing technology Common functions of data cleansing technology • Find and ReplaceFind and Replace• Standardisation: compare values from the given Standardisation: compare values from the given
column with a column in a compiled Knowledge column with a column in a compiled Knowledge Base e.g. Job TitlesBase e.g. Job Titles
Input StandardPersonnel Manager HR ManagerStaff Manager HR ManagerHR Manager HR ManagerHuman Resources Manager HR ManagerStaff Development Manager HR ManagerAccounts Clerk Accounts ClerkAccounts Officer Accounts ClerkAccounts Accounts ClerkAccounts Receivable Accounts ClerkPetty Cash Clerk Accounts ClerkManaging Director CEO/MDMD CEO/MDCEO CEO/MDChief Executive Officer CEO/MDMan Dir CEO/MDMng Director CEO/MDSecretary to MD CEO/MDPA to Sales Director PA/SecPersonal Assistant PA/SecSecretary ???
Common functions of data cleansing technology Common functions of data cleansing technology • Find and ReplaceFind and Replace• Standardisation: compare values from the given column Standardisation: compare values from the given column
with a column in a compiled Knowledge Base e.g. Job Titleswith a column in a compiled Knowledge Base e.g. Job Titles• Data split: Divide data in a single field into multiple fields Data split: Divide data in a single field into multiple fields
e.g. Mr John Smith to be divided into three fields of Title, e.g. Mr John Smith to be divided into three fields of Title, First Name and SurnameFirst Name and Surname
• De-duplication and Merge/purge De-duplication and Merge/purge • Case conversionCase conversion• Address technology – correction, replacement, batch and Address technology – correction, replacement, batch and
Data Business RulesData Business Rules• Business rules manage the validation process and the ongoing protection of data qualityBusiness rules manage the validation process and the ongoing protection of data quality
• Make your rules as stringent as you can to begin, then assess the volumes of rejects and Make your rules as stringent as you can to begin, then assess the volumes of rejects and adjust accordinglyadjust accordingly
• Quarantine offenders Quarantine offenders
• Impose rules on internal data acquisitionImpose rules on internal data acquisition
• Ensure they are included in the brief for any external data capture resourcesEnsure they are included in the brief for any external data capture resources
Example of Business RulesExample of Business RulesDatabase attribute
Title From listForenameSurname MandatoryAddress line 1 MandatoryAddress line 2TownCountyPostcode MandatoryCountry From listTelephone numberE-mail address Must contain @Date of birth Set formatMarital status From listGender From listNumber of children <18Ref Number System GeneratedSource From list
Importance Accepted in the BusinessImportance Accepted in the Business• Lip service to business as usualLip service to business as usual• Incentives or penaltiesIncentives or penalties• Demonstration of the implications of poor data and/or how it Demonstration of the implications of poor data and/or how it
makes them more effective at their jobmakes them more effective at their job• Ensure they know how important it is that they complyEnsure they know how important it is that they comply• Make it easy for them to adhere to the rulesMake it easy for them to adhere to the rules• Listen to them and address their problems – their view of poor Listen to them and address their problems – their view of poor
quality data may be different to yoursquality data may be different to yours• Be prepared to changeBe prepared to change
— Software amendmentsSoftware amendments— Business processesBusiness processes— Forum or reporting channel for data issuesForum or reporting channel for data issues
• Encountering customers as part of regular business processes Encountering customers as part of regular business processes - the touch points- the touch points
• Opportunities forOpportunities for
— Acquiring new dataAcquiring new data
— Qualifying existing contactsQualifying existing contacts
— Verifying or updating existing dataVerifying or updating existing data
— Testing the relationship Testing the relationship (Jenkinson 1995)(Jenkinson 1995)
• Consider all of these opportunities within your business Consider all of these opportunities within your business processesprocesses
And finally….And finally….• Know your dataKnow your data• Document your data strategyDocument your data strategy• Score the data based on level of confidenceScore the data based on level of confidence• Determine internal & external solutions Determine internal & external solutions • Create rules to apply at all data collection points – don’t forget Create rules to apply at all data collection points – don’t forget
your external data capture bureaux, partners and sales your external data capture bureaux, partners and sales channelschannels
• Regular review in the light of on-going data qualityRegular review in the light of on-going data quality• Learn from your experience Learn from your experience • Be prepared for change to processes and softwareBe prepared for change to processes and software• Achieve quality and maintain quality – get it right, keep it right!Achieve quality and maintain quality – get it right, keep it right!
