1 DataIntegrationChallengesinOnboardingOperationalData
DataIntegrationChallengesinOnboardingOperationalData
WHITEPAPER
Latest Update: September 21, 2011
www.expressor-software.com
2 DataIntegrationChallengesinOnboardingOperationalData
Executive Summary
Organizationsinawidevarietyofindustriesmustimport,normalize,andintegrateoperationaldatafromvariousinternalandexternalbusinesspartnersonanon-goingbasis.Thisprocessofonboardingoperationaldataisoftenfraughtwithcomplexityduetovariationsinthewaypartnersandsystemsrepresentdata—andinconsistenciesfromoneonboard-ingprocesstothenext.Thesecomplexitiesoftenimpactanorganization’sabilitytoscaleitsprocessesandsupportitsbusiness.
Thiswhitepaperexaminesvariouschallengesinvolvedwithonboardingoperationaldata,andpresentsaneffectivesolutionbasedonapowerfulenablingtechnologyfromexpressorthatsimplifiesandstandardizesonboardingprocessestoenableorganizationstoovercomethesebusinesschallenges.
Introduction
Onboarding operational datareferstotheprocessofintegratingdatafromvariousinternalandexternalpartnersintoanorganization’sinformationsysteminfrastructuretosupportanorganization’sbusinessprocesses.Theabilitytoquicklyandaccuratelyonboardoperationaldataisoftenacore,mission-criticalrequirement.
Manycompanies,inawiderangeofindustries,havebusinessrequirementsforonboardingoperationaldatafromexternalsources—includingfromotherdepartmentswithinthecompany,aswellasfromthirdpartybusinesspartners—intotheirstructureddatamanagementenvironment.Whilesomeorganizationsonboarddatathatisdeliveredtotheminapredictableformat,manyorganizationsmustrespondinreal-timetonewdataformatsprovidedbynewpartners—creatingcomplexities,delays,anderrors.
Examplesoforganizationsthatmustquicklyandaccuratelyonboardoperationaldataincludethefollowing:
• B2BmanufacturersthatmustsynchronizeERPandotherdataacrossanetworkofsuppliers,subcontractors, anddistributors.• Pharmaceuticalvendorsthatneedtoanalyzethedistributionofdrugsbyprocessingthedailytransactionsofthe pharmaciesthatdispenseitsproducts.• Marketingservicesprovidersthatacceptdatafromcustomerstocreatenewcampaigns,performdataenrichment anddatacleansing,andotherservices.• Discountretailersthatdependonbeingabletorapidlyintegratewithnewsupplierstoquicklyofferandselltheir distressedinventory.• Servicesproviderssuchashealthinsurancevendorsor401-Kadministratorsthatmustquicklysupportnew enterprisecustomers.
Whileonboardingdataisoftenamission-criticalbusinessprocess,manyorganizationsdonottreatitasamission-criti-calcomponentoftheirenterprisedatastrategy.Asamission-criticalbusinessprocess,organizationscangaindramaticbusinessbenefitsbyimprovingaccuracyandefficiencyandreducingcostsandcomplexitiesassociatedwithdevelop-ing,executing,reusing,andmaintainingtheironboardingprocesses.
2www.expressor-software.com
3 DataIntegrationChallengesinOnboardingOperationalData
• Multiple Data Sources• Unique Artifact Types• Similar Data, Defined Differently
• Requires writing numerous programs & queries.• Can require modifying existing programs, creating a many custom programs.• Requires point-to-point data mapping.• Allows for limited development reuse across partners.• Increases in complexity with new data sources.• Is manual, costly, time consuming, and error prone.
• Data is successfully onboarded into an organization’s structured data management environment.
Custom Coding and “One-Off” Applications
Properly Structured and Transformed Data
Sourcing From Data Artifacts Typical Approach - Human Intervention Data Management Environment
XML
XLS
Challenges
Onboardingoperationaldataoftenrequireshandlingdatafromavarietyofsourcesandpartners,representedindissimilarformats.Coupledwithaneedtohandlemanydifferentdatasourcesandpartnersinarapidandefficientmanner,theprocessofonboardingoperationaldatatosupportbusinesspartnerscanbechallengingfororganizations.
Thedata artifactsthatareexchangedbetweenbusinesspartnersmaycontaindifferencesinthewaythatthedataisrepresented,andtheremaybeinconsistenciesfromoneonboardingtothenext,whichmakeitdifficulttodevelopstandardizeddataintegrationapplicationstohandletheprocess.Forexample,onepartnermaycreateanExcelspreadsheetwithadataelementcalled“Acct_Num,”definedasa16characterlengthalphanumericfield,whichisthedatabaserepresentationofthe“AccountNumber.”AnotherpartnermaycreateaCSVfilewith“Account_Nbr”definedasaninteger.Whilebothdataelementsarefunctionallythesame,theyarerepresentedindifferentformatsthatrequiredifferentprocessingatthedataintegrationlevel.Thesearethesortofinconsistenciesthatmakeitchallengingtocreateefficientandreusabledataprocessingapplicationsthatsupportmultipledatasourcesandbusinesspartners.
www.expressor-software.com
“Onboarding operational data is a source of pain for many organizations. We hear this again and again. Organizations need to normalize data from different clients or partners, and they need a solution that lets them adapt easily to each new onboarding requirement, which is where expressor can really help. expressor enables organizations to onboard new data much faster and more efficiently through a reusable data integration framework. With expressor, organizations will no longer need to write or modify one-off, custom programs and queries, which will simplify the onboarding process and reduce the maintenance associated with the traditional approach. Organizations will need fewer resources, while still reducing the time needed to add new customers or partners, enabling them to scale without adding significant costs to their business process.”
Wiqar Chaudry, expressor Product Manager
Figure 1: Workflow and Challenges Onboarding Operational Data
4 DataIntegrationChallengesinOnboardingOperationalData
• Creates a reusable onboarding infrastructure.• Simplifies data mapping with reusable Semantic Types.• Requires fewer technical resources.• Creates a more manageable data integration environment.• Supports a scalable process, which supports more files, more customers, and more data.
Map Data usingSemantic Data Types
Data is Now Onboarded Into Your Structured Data
Management Environment
Benefits of Semantic Data Integration
XML
XLS
Properly Structured and Transformed Data
Inaddition,apartfromthechallengesofintegratingnewdatafromnewpartnersandsources,themetadatathatdefinestheunderlyingdataisoftendifferentaswell.
Asaresult,itisdifficulttostandardizeandreuseexistingonboardingapplications,whetherdevelopedusingdataintegrationtoolsorcustomcodingtechniques,toaccommodatenewpartnersanddata.Organizationsareforcedtorepeatedlydevelopnewapplicationstohandle“one-off”dataartifactsincustomizedwaysforeachnewpartnerordatasource.Thisgrowinglibraryofone-offroutinesiscumbersome,expensive,andinefficientfororganizationstomanageandmaintain.Inaddition,thesubtledifferencesbetweenthedataelementsrequiretediousattentiontodetail,oftencausingdelaysanderrors.Traditionalapproachestoonboardingoperationaldataaretypicallymanuallyintensive,expensive,timeconsuming,anderrorprone.
Finally,asthevolumeofdata,dataelements,andfrequencyofonboardingeventsincrease,thechallengesandcostsassociatedwithprocessingthesedataartifactsaremagnified—creatingissuesaroundanorganization’sabilitytoscaleitsprocess,andultimatelyscaleitssupportofthebusiness.
expressor software: An Ideal Approach To Onboarding Operational Data
DataintegrationandETLtoolsareuniversallyrecognizedtobemoreefficientthancustomcodingandmanualdatamanipulationtechniquesfortheinitialdevelopmentandre-useofapplications.However,theuniquechallengespre-sentedbymanyonboardingtasksmakeapplicationre-usedifficultorimpossible,evenformostdataintegrationtools.
expressorsoftwareprovidesaradicallysimplifiedapproachtodataintegrationthatfarsurpassestheapplicationre-usecapabilitiesofeveryotherdataintegrationsoftware(aswellasthoseofcustomcoding),makingitidealforonboardingoperationaldata.expressorhaspioneeredauniquemetadata abstraction frameworkthatenablesunprecedentedre-useofdata,metadata,anddataintegrationartifacts,tofinallybringre-useandsimplicitytotheprocessofonboard-ingoperationaldatafrombusinesspartners.Perfectlysuitedtothisusecase,expressorisafast,reliable,easy-to-useplatformforaccessing,transforming,anddeliveringdatafromvirtuallyanybusinesssysteminanyformat.
www.expressor-software.com
Figure 2: Semantic Data Integration simplifies Onboarding Operational Data
5 DataIntegrationChallengesinOnboardingOperationalData
ThefoundationoftheexpressorplatformisapowerfulmetadatalayercalledtheSemantic Framework.TheSemanticFrameworksimplifiesandstandardizesthecomplexandoftenmessyprocessofdataintegrationapplicationdesign.ItemploysreusabledataobjectsreferredtoasSemantic Typesthatenableorganizationstodefine,inbusinessterms,howthedatashouldberepresented.SemanticTypeartifactsholdinformationsuchasfieldnamesanddatatypes,whichareautomaticallycreatedandmappedtoaschemaartifact,whichisareusablepieceofmetadatathatdescribestheexternaldatalayoutwhenanexternaldatasourceisread.Theauto-generatedSemanticTypeinheritsthecharac-teristicsoftheschemabydefault,simplifyingthetaskofworkingwithdisparateanddissimilardatasources.SemanticTypesarereusable,abstracted,internaldatastructuresthatcanbemappedtomultipledatasourcesortargets.TheyenableexternaldatatypestoautomaticallyadapttotheinternaldatatypesdefinedintheSemanticType,whichenablestheSemanticFrameworktobeeasilymodifiedtohandlenewdatasources,andautomaticallyinsulatesapplicationsfromschemachangesthatmightoccurinadatasourceortarget.
Semantic Types Deconstructed
SemanticTypesrepresentareusableabstractinternaldatastructurethatcanbeuserdefinedorautomaticallygeneratedfromthedata’ssourceschema.UserscanfurtherextendthecapabilitiesofSemanticTypesbyaddingconstraintstoeachattributeofaSemanticType.AnapplicationcanutilizevariousSemanticTypes:
Local Composite Type–Alocaltypeisanartifactthatisauto-generatedwheneveranewdatasourceisloaded.Alocaltyperepresentsaninternaldatastructurethatismappedtoaschemaartifact,whichdefinestheexternaldatastructureofthedatasourceortarget.
Shared Composite Type–Asharedtypecanbealocaltypethathasbeenpromoted,oritcanbeagenericstructurethatcanbedefinedwithintheStudioapplication.ThebenefitofthistypeofinternaldatastructureisthatitcanbereusedacrossmanyexternaldatasourcesandtargetswithoutaffectinganyoftheinternalETLlogicwrittenagainstit.
www.expressor-software.com
expressor Artifacts
Semantic Types define the structure of internal data.
Read Operator Transform Operator Read Operator
Semantic Types define the data type and constraints for data processed within a Dataflow.
Composite Types map to Schema records. Atomic Types are used in Composite Type attributes.
Attributes are mapped with rules in transform operators.
Dataflows Connections Semantic TypesSchemas
Schema Mappings Type Type Mappings SchemaAttributes Rules Attributes
Figure 3: Semantic Types in expressor
6 DataIntegrationChallengesinOnboardingOperationalData
Shared Atomic Type–AnatomictypeisthefundamentalbuildingblockofSemanticTypesandrepresentsasingleattributethatcanbeutilizedinoneormorecompositetypes.Constraints–Constraintsaredefinedonindividualattributesofacompositetype.Eachattributecarriesthefollowinginformationaspartofitsdefinition:
• Thenameoftheattribute,whichshouldbedescriptiveforhumanconsumption.• Aprimitivedatatypewhichisassignedtotheattribute(supporteddatatypes:string,integer,decimal,double, date-time,byte).• Constraints,whicharesetsofrulesthatthedatarepresentedbytheattributemustconformtobasedondatatype (exampleconstraints:minimumvalue,maximumvalue,minimum,regularexpressions,allowedvalues).
Onceaconstraintisdefined,theapplicationcanspecifythecorrectiveactionthatshouldbetakeniftheruleisviolated.Bydefault,allcorrectiveactionsaresettoescalateerrorstothedataflow,andtheapplicationcanreject,skip,orredirectdataandtakeappropriateactions.Anapplicationcanbesettocorrectdatawithintheconstraintdefinitionbyprovidingdefaultvaluesorotheravailableactionsbasedontheattributedatatype.
Constraintsareappliedtoattributesofcompositetypesoratomictypesandenabletheconstraintinformationtobecarriedforwardwhereveragivenattributeisused.Thisstreamlinesthedevelopmentprocessbydecouplingdatacleansinglogicfromanyactualtransformationthatneedstooccurtothedata.Allconstraint-relatedlogiciscentralizedandmanagedwithintheSemanticTypeartifactenablingeasiermaintenanceandartifactreusability,makingitidealforhandlingthewidevariationindataartifactsduringonboardingprocessing.
Simplified Data Mappings
TheSemanticFrameworksimplifiesthetypicallyhighlycomplexdatamappingprocessbyreplacingthepoint-to-pointmappingdoneintraditionalETLtoolswithaunique,streamlinedmappingstructureachievedthroughtheuseofSemanticTypes.Bymappingsourceandtargetdatasetstobusinessdataobjects,expressorautomaticallyreconcilesdatatypediscrepancies.
www.expressor-software.com
Figure 4: Constraints and Error Corrections Interface in expressor
7 DataIntegrationChallengesinOnboardingOperationalData
expressorispurpose-builttoalloworganizationstoreuseSemanticTypeartifactsasmanytimesasnecessary,enablingorganizationstobuildasingleapplicationthatcanbeappliedtothewiderangeofbusinesspartnersanddataformatsdictatedbyitsbusinessrequirements.SemanticTypedefinitionsincludemappings,datatypeconversions,constraintdefinitionsanderrorhandlingactions.expressor’suniqueapproachtodataintegrationensuresthatallchangesareautomaticallyandaccuratelypropagatedthroughalldataintegrationapplications.
Onboarding Operational Data Case Study: Managed Service Provider
Amarketingservicesprovider(MSP)servesalargenumberoffinancialinstitutions,providingvariousformsofemailcommunicationsandcampaignstoeachfinancialinstitution’scustomers.
Aspartofeachclientengagement,thefinancialinstitutionssubmittheirowncustomerfilestotheMSP,eachinauniqueformat.TheMSPmustvalidate,normalize,andimportthedataineachcustomerfileintoitsdatabasebeforecampaignscanberun.Thecompanyoperatesontighttimeframes,executingmultiplecampaignsforarangeofclientsacrossmultipletimezones.
Thecompany’sinitial,custom-codedsolutionforonboardingdatafromitsclientswasveryrigid.Sincenewmappingsandtransformationswererequiredforeachnewcustomerandcampaign,“developmentwasveryslow,andmainte-nancewasanightmare,”accordingtothecompany.Moreover,thecompanyhassevenemployeesthatcanfunctionasanalysts,butonlyonedeveloper.Thecompanyneededaflexibleonboardingsolutionthatwouldallowthesevenanalyststoquicklycustomizetheapplicationfornewcustomersandcampaigns.
Thecompanyimplementedexpressorsoftwaretodevelopaflexible,genericapplicationtoreadcustomerfiles,performvalidationanderrorhandling,mapandtransformthedataintheincomingfilestomatchtheschemaintheproductiontables,loadnewrecordstotheproductiontable,andwritethepriorrecordstoahistorytable.
www.expressor-software.com
Figure 5: expressor’s Simplified Mapping Process
Traditional Mapping Simplified Mapping
SemanticType
8 DataIntegrationChallengesinOnboardingOperationalData www.expressor-software.com
WhereastheMSPsprioronboardingapplicationrequired80hourstodevelop,theexpressorsoftwareapplication,builtusingexpressor’sgraphical,drag-and-drop,color-codedinterfacewasbuiltfourtimesfaster.
Withthepreviousapplication,eachnewcustomerorcampaignrequiredanewmapping,whichtookeighthoursofthedeveloper’stime.WithexpressorandtheuseofSemanticTypes,thenewfilesareincorporatedbyre-usingtheexistingapplicationthroughexpressor’sgraphicaldevelopmentenvironment.Thenewcustomerandcampaignmappingsarecreatedeighttimesfasterandarecreatedbythecompany’sanalysts,freeingthedevelopertoworkonmorecriticaltasks.
Thecompany’ssevenanalystsarenowabletoaccommodatetheirnewcustomersandcampaignsusingexpressor’sgraphicalenvironment,whichrequiresnocodingorcompiling,allowingthemtoeasilymapnewfileformatstothecom-monSemanticType.Andalloflogic,includingtransformations,fieldvalidations,anderrorhandlingareautomaticallymaintained—dramaticallyspeedingandsimplifyingnewclientengagementsandenforcingstandardization.
Benefits
expressorsoftwareprovidesanidealplatformforcompaniesthatmustperformonboardingofoperationaldatatosupportmultiplepartnersandcriticalbusinessprocesses.Organizationsusingexpressorsoftwareforonboardingoperationaldataareexperiencingmeasureablebenefits:
Speed of development. Withexpressor,customersdownload,install,build,andruntheirfirstapplicationinminutes.expressoriseasytodownload,install,learn,anduse.Itprovidesafamiliar,purpose-builtUIwithanMSOffice-likelookandfeel.Itprovidesdrag&dropconfigurationandusessimplified,graphicaltoolsfordefiningreusabledatamappingsandbusinessrulestospeedandsimplifythedevelopmentofdataintegrationapplications.
Ease of re-use.Withexpressor,customerscanreuseeverything,makingitidealforhandlingthechallengesandcom-plexitiesofonboardingoperationaldatafrombusinesspartners.Organizationscandefineandreuseanyprojectartifactfromsimplefileconnectionstocomplexdataflowsandbusinessrules.Userscandesignandreusetransformationoperators,dataflows,datasourceconnections,andtransformationlogic.Moreover,allprojectartifactscanbesharedandreusedwithinasingleworkspaceacrossmultipleprojects.
Accessible to a wider range of skill sets. Simplifiedre-usemeansthatlesstechnicaluserscanadapttheapplicationstohandlenewdatasourcesandbusinesspartners,freeingdeveloperstoworkonmorecriticaltasks.
Increased accuracy and standardization.Definingstandardapplicationsthatenforcedatamappings,constraints,anderrorhandlingactionsensuresconsistencywithnewdataandpartners,especiallycomparedwithcustomcodinganddevelopingone-offapplications.
Summary
Organizationsacrossawiderangeofindustriesthatmusteffectivelyonboardoperationaldataareexperiencingdra-maticbusinessbenefitsbyimplementingexpressorsoftware’sdataintegrationplatform,includingimprovedaccuracy,greaterefficiencies,andreducedcostsandcomplexityassociatedwithdeveloping,executing,reusing,andmaintainingitsonboardingprocesses.
9 DataIntegrationChallengesinOnboardingOperationalData©2010expressorsoftwarecorporation.expressorandredefiningdataintegrationaretrademarksofexpressorsoftwarecorporation.Allothertrademarksortradenamesarepropertiesoftheirrespectiveowners.Allrightsreserved.
©2011expressorsoftwarecorporation,allrightsreserved.Thefollowingaretrademarksofexpressorsoftwarecorporation:expressor,SmartSemantics,IntelligentLoadandGo.
expressorsoftwarecorporation1NewEnglandExecutiveParkBurlington,MA01803USAwww.expressor-software.com
About expressor
Atexpressorsoftware,we’resimplifyingdataintegrationbyprovidingsoftwarethatiseasytouse,powerful,andafford-able—enablingorganizationstoaccomplishmorewithfewerresources.Wedramaticallysimplifythevastmajorityofbasicdataintegrationtasks—makingitfareasiertomapandmovedatafromonelocationtoanother.And,whendataintegrationbecomesmorecomplicated—requiringvarioustypesofdatatransformations—wehelpmakedataintegrationandETLprojectsmoreflexibleandefficientbyenablingthecreationandsubsequentreuseofpowerfulartifacts,con-figurations,andbusinessrules.Thismeansthatwhenacomplexbusinessdatachallengeissolvedonce,thesolutioncanbereused.
expressorhasarapidlygrowingcommunityofover12,500uniqueexpressorStudiodownloaduserstoday.LeadingorganizationssuchasSKECHERSUSA,AmericanTower,Viverae,WesternWorld,LABVANTAGE,andClickRSVPhaveadoptedexpressor’saffordabledataintegrationplatformforserverdeploymenttohelpthemintegrateon-premisesandclouddatasourceswiththeirbusinesscriticalapplications.
expressorStudioisthedesigncomponentoftheexpressorDataIntegrationPlatform,andexpressormakesafreedownloadofStudioavailableatwww.expressorStudio.com.
expressorsoftwareisheadquarteredinBurlington,MA.Formoreinformationabouthowexpressorissimplifyingdataintegration,pleasevisitourwebsiteatwww.expressor-software.comorcontactusdirectlyat+1(781)505-4190x222.