Database/Data Warehouse Integration Architecture

ContentsIntroduction.................................................................................................................................................2

Overview......................................................................................................................................................2

BusinessIntelligenceIntegration.............................................................................................................2

DatabaseIntegration................................................................................................................................2

BusinessIntelligenceIntegration.................................................................................................................2

ExtractionProcess................................................................................................................................2

ApplicationConfiguration....................................................................................................................4

Entities..................................................................................................................................................4

DatabaseIntegration....................................................................................................................................6

SynchronizationProcess.......................................................................................................................6

ApplicationConfiguration....................................................................................................................9

Entities..................................................................................................................................................9

LinkstoRelatedMaterial...........................................................................................................................12

IntroductionThisdocumentprovidesdetailedinformationregardingimplementationof:

• ExtractionarchitecturebetweenMarketoandanexternalBusinessIntelligencesystem(BI)• SynchronizationarchitecturebetweenMarketoandanexternalDatabase/DataWarehouse

system(DB)Entitiesaredescribed,andthespecificsofmaintainingsynchronizationofnewandupdatedrecords.

Overview

BusinessIntelligenceIntegrationThisusecaseanswersthequestion,"HowdoIgetMarketodataintomyenterpriseBIsolutionforcustomreporting?"BIsolutionspermitflexiblereportingonMarketomarketingdatacombinedwithotherdatasourcessuchasfromcustomerservice,operationsorfinance.Combineddataisoftenanalyzedandpresentedintheformofdashboardsthatareusedtotrackbusinessmetrics.

MarketorecommendsimplementationofBIintegrationapplicationsusingaone-waypollingmodel,retrievingnewrecordsfromMarketo,andpushingthoserecordsasmappedintheapplicationconfigurationtotheBIsystem.TheapplicationshouldqueryfornewrecordsperiodicallyandthenperformanynecessarytranslationandfiltrationbeforepushingtheupdatestoBIdatastore.

DatabaseIntegrationThisusecaseanswersthequestion,"HowdoIsynchronizeMarketowithanexternalDatabase/DataWarehouse?"Acommondatamanagementpatternistomaintaina"systemofrecord"(SOR),whichservesastheauthoritativedatasourceforagivenelementorpieceofinformation.TheSORistypicallyarepositorywheredataobjectsaremaintained.Keepinglead/contactdatainsynchbetweenmarketingcampaignsandbackendsystemprocessesensuresconsistency.ModelingMarketocustomobjectsusingdataobjectssuchassalesordersororderfulfillmentcanhelpimproveyourmarketingcampaignsbyincludingenricheddataincustomercommunications.

MarketorecommendsimplementationofDBsynchronizationapplicationsusingatwo-waypollingmodel,retrievingchangesandnewrecordsfromeithersystem,andpushingthosechangesasmappedintheapplicationconfigurationtotheothersystem.TheapplicationshouldqueryforchangesofselectedrecordtypesperiodicallyandthenperformanynecessarytranslationandfiltrationbeforepushingtheupdatestoDBorMarketorespectively.

BusinessIntelligenceIntegration

ExtractionProcessThesimplestandmostefficientwaytoextractdataistoimplementapollingprocessthatretrievesnewactivityrecordsfromMarketoandpushesdatatotheBIdatastore.Thecycleisthenrepeatedafterapredeterminedperiodeachtime.Thiscouldbeimplementedasadailyscheduledjobforexample.

RecommendedExtractionOrder

1. ExtractActivitiesfromMarketo2. ExtractLeaddetailforeachActivityfromMarketo(optional)3. UpdateBIdatastore

Activitiesareread-onlyaftercreation.TheyarereadfromMarketoviatheGetLeadActivitiesendpoint,ingroupsofupto10activitytypes.TheendpointacceptsanearliestcreationdateviaapagingtokenwhichisretrievedviatheGetPagingTokenendpoint.ThesetofresultsshouldbepagedthroughuntilthemoreResultparameterintheresponseisreturnedasfalse,andthecorrespondingactivitieswrittentoBIdatastore.BeginthenextextractioncyclebyadjustingtheearliestcreationdateandcreatinganewpagingToken.Movethedateforwardintimebythedesiredamounttoestablishanewearliestcreationdate.Theamountoftimeselectedbecomesthepollinginterval.

SinceMarketomaintainsonlyper-secondresolutionfordatetimes,itispossiblethatthesameactivityrecordcouldbereturnedintwodifferentextractcycles.Toavoiddataduplication,theapplicationshouldusetheactivityidwhichisauniqueidentifier.Duplicateactivityrecordscanbesafelyignored.Iftheactivityrecorddoesnotprovideallofthelead-relateddatarequiredtomeetyourbusinessneed,youmayperformanadditionallookupoftherelatedleadrecord.Tolookuptherelatedleadrecord,youpasstheleadIdfieldfromtheactivityrecordtoGetMultipleLeadsbyFilterType.ThisendpointallowsyoutosendabatchofleadIdsinasinglerequest.ThishelpsminimizetheoverallnumberofAPIcallsrequiredforthelookup.Youwillthencombineactivityrecorddatawithleadfielddata.Thelogicusedtocombinethetwodatasetsshouldreflectaninnerjoinoperation,usingleadIdasprimarykeyandasforeignkey.

ApplicationConfigurationTheapplicationshouldsupportasetofconfigurationoptionstocontrolprogrambehavior.Theseoptionscouldbestoredinapropertiesfileforexample.

FieldsNotallfieldsarenecessarytobemappedbetweenMarketoandanassociatedBIsystem.AnoptiontoenableordisableextractionofspecificfieldsfromLeadandActivityentitytypesisrecommended.OnlytheleadIdfieldismandatory,whileallothersshouldbeoptional.Reducingthenumberofextractedfieldswillimproveperformanceinallcases.

ActivityTypesAdminusersshouldbeabletoselectwhichactivitytypesareandarenotextractedtolimitAPIcallusageandimproveperformance.

PollingIntervalThetimebetweenextractionbatchjobsneedstostrikeabalancebetweenlowdatalatency,andhighutilizationofAPIcalls.

YourpollingintervalshouldbebasedonhowmanyAPIcallsatypicalextractioncyclewilltake.ThenumberofnewactivityrecordsthataclientexpectstobecreatedwillbethegreatestinfluenceonthecumulativenumberofAPIcallswhichwillbeusedinagivenday.ForaccountsprovisionedorrenewedafterMarch2016,thedefaultnumberofAPIcallsperdayis50,000.AdditionalAPIcallsmaybepurchasedingroupsof10,000/day.

TraditionalBIpresentshistoricaldataformanualanalysis,solatencyisnotaconcern.Inthiscasea24hourintervalisappropriate.Real-timeBIreliesoneventdrivenprocessing,solatencyisaconcern.Inthiscasea5minuteintervalisappropriate.

EntitiesThefollowingaretheMarketoentitiesthatapplyforBIintegrationusecase:Leads,Activities.

LeadsPrimaryKey:id

TheintegeridofaMarketoleadrecordandtheprimarykey.ThisissystemmanagedbyMarketo,andmayonlybeassignedbyMarketo.

InMarketo,leadsrepresentanyperson-recordwhichrepresentsasalesormarketingtarget.AllSmartCampaigns(commonlyreferredtoasa“workflow”innon-Marketosystems),filter,trigger,andoperateonleadrecords,basedontheircharacteristicsandactions.

Model

LeadsarehighlyextensibleinMarketoandmayincludealargenumberofcustomfields.Whenextractingdatafromanyparticularsubscription,theDescribeLeadendpointoftheRESTAPIshouldbeusedastheexclusivesourceoftruthtodeterminefieldavailabilityinaparticularsubscription.

Notethatthemodelforaleadisalsopotentiallydynamic,asfieldsmaybeaddedorhiddenbyend-usersatanytime.Theapplicationmustberesilienttosuchchanges,andnotbreakwhentheyoccur.

Relationships

LeadsarerelatedtonumerousaccessibleobjecttypesinMarketo.Forthisusecase,weareonlyconcernedwithActivities.

Object RelationshipType LinkType LinkFieldActivities 1:N ActivityField leadIdAccessibility

Inordertoreadleads,anAPIusermusthaveRead-OnlyLeadpermission.Theycanbereadthroughthefollowingendpoints:

• GetMultipleLeadsbyFilterType• GetMultipleLeadsbyListId• GetMultipleLeadsbyProgramId

LeadMetadataisreadthroughtheDescribeLeadendpoint.

ActivitiesPrimaryKey:id

ActivitieshaveauniqueintegeridthatissetuponcreationbyMarketo.

ActivitiesarerecordsofactivitiesassociatedtoleadrecordsinMarketo.Theymayrecordactivityofmanydifferenttypes,asindicatedbytheiractivityTypeId.Activitiesareread-onlyinMarketo.Certainactivitytypesareprunedafter90daysintheMarketosystem.

PrunedActivities• DataValueChange• AddtoList• RemovefromList• VisitWebPage• ClickLink• ChangeScore

Thetypesofactivitiesavailableinagivensubscriptionvarydependingonmanyfactors,includingthetypeofsubscription.TheavailabletypesandtheirmetadatashouldalwaysbedeterminedbycallingGetActivityTypesendpointfromthetargetsubscription.

Model

Activitieshaveasemi-strictschema.Thefollowingfieldsaredefined,butnotnecessarilyusedforallactivitytypes:

Name Datatype Descriptionid Integer Uniqueidforactivity.leadId Integer Idofthelinkedlead.Mapstoidonleadrecords.activityTypeId Integer Idofthetypeofactivity.CorrespondstoaresultofGet

ActivityTypes.activityDate Datetime Datethattheactivityoccurred.primaryAttributeValue String Valueoftheprimaryattribute.primaryAttributeValueId Integer Idoftheprimaryattribute.attributes Array Arrayofname/valuepairsrepresentingthesecondary

attributesoftheactivity.

Eachactivitytypehasaprimaryattributethatcorrespondstoavalueofsomekind.TheprimaryattributemayberelatedtoanytypeofassetorobjectinMarketo.Forexample,theprimaryAttributeValueoftheVisitsWebPagetypecorrespondstothenameorURLofthewebpagethatwasvisited.TheMarketonamewillbepresentedifthepagewasalandingpageandthepageURLifitisnotaMarketopage.Secondaryattributesconsistofanarrayofname/valuepairs,namingeachofthefieldsforanactivitytypeandthecorrespondingvalue.Continuingwiththeexample,someofthesecondaryattributeswouldbeClientIPAddress,QueryParameters,ReferrerURL,andUserAgent.

Relationships

ActivitiesinMarketoarealwaysrelatedtoleadrecordsthroughtheleadIdfield.SomeactivitytypesmayhavearelationshiptootherMarketoassetsthroughtheirprimaryAttributeValue.

Accessibility

Inordertoreadactivities,anAPIusermusthavetheRead-OnlyActivitypermission.

ActivitiescanbereadthroughtheGetLeadActivities,GetLeadChanges,andGetDeletedLeadsendpoints.

DatabaseIntegration

SynchronizationProcessThesimplestandmostefficientwaytomaintaincontinuoussynchronizationistoimplementapollingprocessthatretrieveschangestoleadrecordsinMarketoandpushesthemtoDBandthenfromLead/Contact,CustomObject,orCompanyrecordsinDBandpushesthemtoMarketo.Thecycleisthenrepeatedafterapredeterminedperiodeachtime.Thiscouldbeimplementedasascheduledjobforexample.

ToretrievechangesfromMarketo,ahighwatermarkmustbemaintainedforchangestolead/companyfieldswhichhaveoccurredsincethemostrecentlyretrievedchange.Thesearedatetimevalues.Whenretrievingchangesinasubsequentsynchronizationcycle,theexactdatetimeofthemostrecentlycreatedrecordshouldbeused.SinceMarketomaintainsonlyper-secondresolutionfordatetimes,itispossiblethatthesameleadchangerecordcouldbereturnedintwodifferentextractcycles.Toavoiddataduplication,theapplicationshouldusetheactivityidwhichuniquelyidentifiestheactivityrecord.Duplicateactivityrecordscanbesafelyignored.

RecommendedSynchronizationOrder

1. RetrieveLeadschangesfromMarketo*2. RetrieveLead/ContactchangesfromDB*3. RetrieveCompanychangesfromDB(optional)*4. RetrieveCustomObjectchangesfromDB(optional)*5. Resolveconflicts(optional)6. UpdateDBwithMarketoData7. UpdateMarketowithDBData

*Includesnew,changed,anddeletedrecords

LeadsandCompanies

SynchronizationforLeadsandCompaniesismaintainedprimarilyusingtheGetLeadChangesendpointwhichretrievesdatavaluechangerecordswhichoccurafteratimestampgivenbyapagingtokenwhichisretrievedviatheGetPagingTokenendpoint.ThisendpointwillreturnbothNewLeadactivities,whichindicatethecreationofanewknownleadinMarketo,anddatavaluechangeactivitiesforasetoffieldsgivenintheparametersofthecall.

ThechangeactivitiesshouldbeappliedintheorderofthecreatedDate,fromearliesttolatest,givenintheactivitytotherecordsinDBwhichcorrespondtotherecordgivenbyleadIdintheactivity.NewLeadactivitiesshouldbeaddedtoaqueueofnewleadrecordswhichneedtoberetrievedbyid,“leadId”intheactivity,usingtheGetLeadsbyFilterTypeendpoint,withidasthefilterType.Thesemayberetrievedupto300atatime.Itisrecommendedtowaituntilthereare300recordstoretrieve,andthentomakethecalltoretrievetheserecords,insteadofcallingwheneverthesebecomeavailable.If,uponreachingtheendofthesetofchanges,therearelessthan300records,thanthesetshouldberetrieved.

Inadditiontochangesandnewleads,inordertomaintainsynchronizationtheMergeLeadandDeleteLeadactivitymustberetrievedinorderaccountforleadrecordswhicharemergedtogether.Themergeactivityindicatesthattworecordshavebeenmergedintoasinglerecord.TheDBmayormaynotchoosetohonorthemerge/delete,deletethelosingrecordandretrievethechangesfromthewinninglead,oritmaybeignoredandhavea“DeletedinMarketo”flagsettoindicatethattheMarketoLeadIDforthatrecordisnolongervalid.

Aspartofastandardsynchronizationcycle,changesfromtheDBshouldalsoberetrieved.Ideallyonlyfieldswhichhavebeenupdatedsincethemostrecentsynchcycleshouldberetrieved,butthismaynotbepossiblegiventheconstraintsofthesystem.Ifthisisavailable,allthechangesforagivenrecordshouldbeaggregatedintoaleadrecordtobesubmittedtoMarketo.Ifachanges-onlyoptionisnotavailable,thenitisviabletoretrievethewholerecordwithalloftheMarketo-mappedfieldsforsubmissiontoMarketo.

Topushchangesforleadrecords,usetheCreate/UpdateLeadsendpoint.Create/UpdateLeadsallowsfortheinputofupto300leadrecordsasJSON.

ForincrementalsynchingofupdatesfromDB,thelookupFieldshouldbespecifiedastheprimarykeyselectedfromtheDBsystem,andthecreateOrUpdatemodeshouldbeused.ThisallowssharingofthesamequeuebynetnewleadsandleadupdateswhichneedtobepushedintoMarketo.

InthecasethataMarketoLeadislinkedtoacompanyrecordviaexternalCompanyId,commonlyconceptualizedasacontact,thecompanytypefieldsthatwerepartoftheleadrecordarenolongerwriteablethroughtheleadrecordandaredeferredtothelinkedcompanyrecord

Companies

ItisimportanttodeterminewhichfieldsareCompany-typefields,andwhichfieldsareLead-typefields.ThiscanbedonewiththeDescribeCompanyendpoint.AllfieldslistedthereareCompany-typefields,ofwhichmostaremirroredasleadfieldsforunlinkedleads.IfaCompany-typefieldisreflectedinaChangeDataValueoperation,thenthechangeshouldbereflectedagainstthecompanyrecordinDBifthechangewasmadeagainstaleadwhichislinkedtoacompanyrecordviaexternalCompanyId.Ifnot,thechangeshouldjustbereflectedagainsttheleadrecord.

CustomObjects

SincetheDBisthesystemofrecordforCustomObjects,synchronizationisone-wayfromDBtoMarketo.ThedefinitionofaCustomObjectshouldbederivedfromthetypeinDB.CustomObjectrecordsshouldbecreated,updated,ordeletedinMarketowheneveracorrespondingeventoccursintheDBsystem.Thischeckshouldbeperformedbytheintegrationsoftwareuponeverysynchronizationcycletoseeifchangesarerequired.

ApplicationConfigurationTheapplicationshouldsupportasetofconfigurationoptionstocontrolprogrambehavior.Theseoptionscouldbestoredinapropertiesfileforexample.

FieldsNotallfieldsarenecessarytobemappedandsynchronizedbetweenMarketoandanassociatedDB.AnoptiontoenableordisablesynchronizationofspecificfieldsfromtheLeadorCustomObjectentitiesisrecommended.OnlytheleadIdandDBforeignkeyfieldsshouldbemandatoryforsynchronization,whileallothersshouldbeoptional.Reducingthenumberofsynchronizedfieldswillimproveperformanceinallcases.

CustomObjectsAnoptiontoenableordisablecustomobjectsynchronizationisrecommended.

PollingIntervalMarketo’snativesynchronizationconnectorsqueueanewbatchofpushesandpulls5minutesafterthecompletionofthepreviousbatch.Thiscoversagreatdealofcasesandstrikesanacceptablecompromisebetweenlowsynchronizationlatency,andexcessiveutilizationofAPIcalls.Foryoursolution,youshouldbasethesynchronizationintervalbasedonhowmanyAPIcallsatypicalsynchronizationcyclewilltake.ForaccountsprovisionedorrenewedafterMarch2016,thedefaultnumberofAPIcallsperdayis50,000.AdditionalAPIcallspurchasedingroupsof10,000/day.

ThenumberofchangesthataclientexpectstooccurfortheirleadrecordswillbethegreatestinfluenceonthecumulativenumberofAPIcallswhichwillbeusedinagivenday,andthisshouldinfluenceyourdesign.

EntitiesThefollowingaretheMarketoentitiesthatapplyforDBintegrationusecase:Leads,Companies,CustomObjects.

LeadsPrimaryKey:id

TheintegeridofaMarketoleadrecordandtheprimarykey.ThisissystemmanagedbyMarketo,andmayonlybeassignedbyMarketo.Anyinsertoperationsattemptedbyaforeignsystemwhichincludeidwillberejected.

InMarketoleadsrepresentanyperson-recordwhichrepresentsasalesormarketingtarget.AllSmartCampaigns(commonlyreferredtoasaworkflowinnon-Marketosystems),filter,trigger,andoperateonleadrecords,basedontheircharacteristicsandactions.

Model

LeadsarehighlyextensibleinMarketoandmayincludealargenumberofCustomFields.Whensynchronizinganyparticularsubscription,asetofstandardfieldsshouldnotbereliedupon,andtheDescribeLeadfunctionoftheRESTAPIshouldbeusedastheexclusivesourceoftruthtodeterminefieldavailabilityandupdateabilityinaparticularsubscription.

Notethatthemodelforaleadisalsopotentiallydynamic,asfieldsmaybeaddedorhiddenbyend-usersatanytime.Theapplicationmustberesilienttosuchchanges,andnotbreakwhentheyoccur.

Relationships

LeadsarerelatedtonumerousaccessibleobjecttypesinMarketo.Forthisusecase,weareonlyconcernedwithCompaniesandCustomObjects.

Object RelationshipType LinkType LinkFieldCompany N:1 LeadField externalCompanyIdCustomObjects 1:N,N:1,orN:N SeeCustomObject

SectionSeeCustomObjectSection

Accessibility

LeadsmaybereadandwrittenfreelyinMarketogiventheRead-WriteLeadPermissionisgrantedtotheAPIuserbeingused.Theycanbereadthroughthefollowingendpoints:

• GetMultipleLeadsbyFilterType• GetMultipleLeadsbyListId• GetMultipleLeadsbyProgramId

Theycanbewrittenthroughthefollowingendpoints:

• Create/UpdateLeads• ImportLead

LeadsmaybedeletedthroughtheDeleteLeadendpoint.

LeadMetadataisreadthroughtheDescribeLeadendpoint.

CompaniesPrimaryKey:externalCompanyId,id

externalCompanyIdisanarbitrarystringfieldsetuponcreationbytheexternalsystem.TheprimarykeyofcompanyrecordsinDBshouldbemappedtoexternalCompanyId,whichisnotupdatableandmustbeunique.Idisauniquesystem-generatedintegerid.

Companyobjectsrepresenttheorganizationtowhichleadrecordsbelong.LeadsareaddedtoaCompanybypopulatingtheircorrespondingexternalCompanyId.Leadslinkedtoacompanyrecordwilldirectlyinheritthevaluesfromacompanyrecordasthoughthevaluesexistedonthelead’sownrecord.Attributesavailableonthecompanyrecordareavailablefortriggeringandfilteringonleadrecordsfromwithintheapplication.

Companyrecordsmayonlybecreatedbyexternalsystems,andtheDBshouldbetreatedasthesourceoftruth.

Model

• Companiesarefullyextensibleandmayhaveanynumberandtypeofcustomfields• DescribeCompanyshouldbeusedtoobtaintheschemaofcompany

Relationships

Object RelationshipType LinkType LinkFieldLead 1:N LeadField externalCompanyIdOpportunity 1:N OpportunityField externalCompanyId

Accessibility

Inordertoreadandwritetocompanyrecords,anAPIusermusthavebeengrantedtheRead-WriteCompanypermission.

Companiesmaybereadthroughasingleendpoint,GetCompaniesbyFilterType.Companiesmayonlybefilteredonalimitednumberoffields,whichareprovidedinthesearchableFieldsattributeoftheDescribeCompanyresult.

CompaniescanbewrittentousingtheCreate/UpdateCompaniesendpoint.

CompaniesmaybedeletedthroughtheDeleteCompaniesendpoint.

CustomObjectsPrimaryKey:marketoGUID,AdditionalUserDefinedkeys

CustomObjectsalwayshaveauniquesystem-generatedmarketoGUIDwhichissetuponcreation.Therewillbeatleastoneadditionalkey,andpossiblymore,whichareuser-definedintheCustomObjectdefinition.KeysmaybesinglefieldsfortypeswhicharelinkeddirectlytoLeadsorCompanies,andmayhavecompoundkeysfortypeswhicharelinkedtoLeadsandanotherCustomObjecttype.KeyscanbedeterminedbyusingDescribeCustomObjecttoretrievethelistofdedupeFields.

MarketoallowsthedefinitionofCustomObjecttypesbyuserstoextendtheMarketoschema.MarketoCustomObjectsmayberelatedtoLeadsorCompaniesineithera1:NorN:Nconfigurationthroughtheusageofintermediatecustomobjects.

Model

NotethattheschemaforcustomobjectswithinaDBispotentiallydynamic,asfieldsmaybeaddedorremovedatanytime.Theapplicationmustberesilienttosuchchanges,andnotbreakwhentheyoccur.

Relationships

MarketoCustomObjectsmayberelatedtoleadsorcompaniesineithera1:NorN:Nconfigurationthroughtheusageofintermediatecustomobjects.RelationshipscanbederivedfromtherelationshipsparameteroftheresultofDescribeCustomObject.

Accessibility

ToreadandwritetoCustomObjects,anAPIusermusthavetheRead-WriteCustomObjectpermission.CustomObjectsaremanipulatedusingthefollowingendpoints:GetCustomObjects,Create/Update/UpsertCustomObjects,andDeleteCustomObjects.AListCustomObjectsendpointisalsoprovidedtoagiveameansofdeterminingwhatCustomObjecttypesareavailableinagivensubscription.

LinkstoRelatedMaterial• RESTAPIreference:MarketoDevelopersSite• RESTAPIerrorcodes:MarketoDevelopersSiteErrorCodes• Bestpracticesandcodesamples:MarketoDevelopersBlog