©2016XavientInformationSystems
InfrastructureasaService isaformofcloudcomputingthatprovidesvirtualizedcomputingresources.
PublicCloud
• AWS
• Azure
• GCE
PrivateCloud
• VMWare
• OpenStack
• ApacheCloudStack
IaaS
©2016XavientInformationSystems
• Platformasaservice (PaaS)isacategoryofcloudcomputingservicesthatprovidesaplatformallowingcustomerstodevelop,run,andmanageapplicationswithoutthecomplexityofbuildingandmaintainingtheinfrastructuretypicallyassociatedwithdevelopingandlaunchinganapp
• DCOS isanewkindofoperatingsystemthatorganizesallofyourmachines,VMs,andcloudinstancesintoasinglepoolofintelligentlyanddynamicallysharedresources.ItrunsontopofandenhancesanymodernversionofLinux
Paas andDCOS
©2016XavientInformationSystems
NotoStaticPartitioning
• Underutilize• Doesnotscale• Hardtodealwithfailures
©2016XavientInformationSystems
SoftwareonMesos
Aurora isaserviceschedulerthatrunsontopofMesos,enablingyoutorunlong-
runningservicesthattakeadvantageofMesos'scalability,fault-tolerance,andresource
isolation.
Marathon isaprivatePaaSbuiltonMesos.Itautomaticallyhandleshardwareor
softwarefailuresandensuresthatanappis“alwayson”.
Hadoop RunningHadooponMesosdistributesMapReducejobsefficientlyacrossan
entirecluster.
Spark isafastandgeneral-purposeclustercomputingsystemwhichmakesparalleljobs
easytowrite.
Chronos isadistributedjobschedulerthatsupportscomplexjobtopologies.Itcanbe
usedasamorefault-tolerantreplacementforCron.Jenkins isacontinuousintegrationserver.Themesos-jenkinspluginallowsitto
dynamicallylaunchworkersonaMesosclusterdependingontheworkload.
Cassandra isaperformantandhighlyavailabledistributeddatabase.Linearscalability
andprovenfault-toleranceoncommodityhardwareorcloudinfrastructuremakeitthe
perfectplatformformission-criticaldata.
ElasticSearch isadistributedsearchengine.Mesosmakesiteasytorunandscale.
©2016XavientInformationSystems
MesosArchitectureinHAmode
Behindthescenes,ApacheMesosusesApacheZooKeeper,adistributedconfigurationstoretopersistitsstate.The"state"ofMesosincludeseverythingfromconfigurationtothestatusandloadofindividualMesosnodes,aswellasframeworkslikeCassandra-Mesos.
©2016XavientInformationSystems
Marathonuse
• Constraintscontrolwhereappsruntoallowoptimizingforeitherfaulttolerance(byspreadingatask
outonmultiplenodes)orlocality(byrunningallofanapplicationstasksonthesamenode).Thereare
theredifferentpartsa)fieldb)operatorc)optional
"constraints":[["rack_id","CLUSTER","rack-1"]],"constraints":[["hostname","CLUSTER","a.specific.node.com"]]
"constraints":[["rack_id","GROUP_BY"]]
Example:https://github.com/mesosphere/marathon/tree/master/examples
curl-i -H'Content-Type:application/json'-d@<filename.json>localhost:8080/v2/apps
©2016XavientInformationSystems
• Reservationproblem
• DataPersistenceproblem
• Whereisthesecurity?AuthenticationandAuthorization
• Monitoring
• FrameworkRateLimiting
• Vendorsupport
• Managedservices
Let’sgetpractical
©2016XavientInformationSystems
• Resourcesarereservedforrolesandrolesusedtoreserveresources• StaticReservationmesos-slave\ --master=<ip>:<port>\ --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096”“8CPUsand4096MBofRAMreservedforadsonthisslave.”
• DynamicreservationinMesos>=0.23.0• CreatethroughFrameworkandOperatorAPI
Framework Operator
Reservation-mechanisms to reserve resourcesinspecificslaves
©2016XavientInformationSystems
• Amechanismtocreateapersistentvolumefromdiskresources• PersistentvolumesenablestatefulservicessuchasHDFS andCassandra tostoretheirdata
withinMesos• Authorizationismusttocreatepersistentvolumes• Persistentvolumescanonlybecreatedfrom reserveddiskresources,whetheritbe
staticallyreservedordynamicallyreserved- Important• CreatethroughFrameworkandOperatorAPI(SameasDynamicReservation)
https://mesosphere.github.io/marathon/docs/persistent-volumes.html
https://mesosphere.github.io/marathon/docs/external-volumes.htmlAnexternalstorageserviceenablesyourappstobemorefault-tolerant.Ifahostfails,Marathonreschedulesyourapponanotherhost,alongwithitsassociateddata,withoutuserintervention.https://github.com/emccode/rexray
PersistentVolumes
©2016XavientInformationSystems
• Authenticationisdisabledbydefault.Whenauthenticationisenabled,operatorscanconfigureMesostoeitherusethedefaultauthenticationmoduleortousea customauthenticationmodule.
• AuthenticationpermitsonlytrustedentitiestointeractwithaMesoscluster• Authenticationcanbeusedatframework,agentandoperatorlevel• Configureauthenticationduringthemesosmasterandslavestart• Authorization,localandcustom• Configuretheactionsthatcertainprincipalsareallowedtoperform
http://mesos.apache.org/documentation/latest/authentication/http://mesos.apache.org/documentation/latest/authorization/
*ThedefaultMesosauthenticationmoduleusesthe CyrusSASLlibrary.SASLisaflexibleframeworkthatallowstwoendpointstoauthenticatewitheachotherusingavarietyofmethods.Bydefault,MesosusesCRAM-MD5authentication.
Security
©2016XavientInformationSystems
MesosandContainers
"If a Docker application is a Lego brick, Kubernetes would be like a kit for building the Millennium Falcon and the Mesos cluster would be like a whole Star Wars universe made of Legos." ~ Solomon
©2016XavientInformationSystems
Mesos+Docker+Kubernetes
MesosclusterscanbedeployedonnearlyeveryIaaScloudproviderinfrastructureorinyourown
physicaldatacenter.KubernetesonMesosrunson-topofthatandthereforeallowsyoutoeasilymoveKubernetesworkloadsfromoneoftheseenvironmentstotheother.http://kubernetes.io/docs/getting-started-guides/mesos/
©2016XavientInformationSystems
CassandraandMesos/Marathon
https://mesosphere.github.io/cassandra-mesos/
https://github.com/mesosphere/dcos-cassandra-service
https://docs.mesosphere.com/1.7/usage/services/cassandra/install-and-customize/
©2016XavientInformationSystems
• Auroravs.Marathonhttp://code.hootsuite.com/marathon-vs-aurora-a-high-level-comparison/
• MesosandKuberneteshttp://stackoverflow.com/questions/26705201/whats-the-difference-between-apaches-mesos-and-googles-kubernetes
• JenkinsandMesoshttp://www.slideshare.net/carlossg/ci-and-cd-at-scale-scaling-jenkins-with-docker-and-apache-mesos
• BloombergandMonitoringhttp://schd.ws/hosted_files/mesosconna2016/98/SeamlessMonitoringForMesosClusters.pdf
• UberandCassandrahttp://schd.ws/hosted_files/mesosconna2016/60/mesoscon-uber.pdf• Mesoscon 2016
https://www.youtube.com/playlist?list=PLGeM09tlguZQVL7ZsfNMffX9h1rGNVqnC• Jenkinsalternatesolutionhttps://github.com/HubSpot/Blazar• Mesosoversubscriptionhttp://mesos.apache.org/documentation/latest/oversubscription/• https://support.mesosphere.com/hc/en-us/categories/200383389-Technical
Read
©2016XavientInformationSystems
CloudFoundry
CloudFoundryisanopen-sourceplatformasaservice(PaaS)thatprovidesyouwitha
choiceofclouds,developerframeworks,andapplicationservices
©2016XavientInformationSystems
Agenda
ü XavientIntroduction
• CompanySnapshot• GlobalPresence• OurDifferentiators
ü XavientBigData&BICapabilities
ü CaseStudies
©2016XavientInformationSystems
CompanySnapshot
WhoWeAre
• Founded in US in 2002 | Global Scale |Exponential Growth
• Fortune 200 clientele | Partner of choicefor4out of Top5CSPs inUS
• Comprehensive Technology Coverage &PersonalizedService
WhatWeDo
• Enable Enterprise Class Solutions onWeb, Mobility, Open Source, Cloud,
BigData
• Focus on Telecom, Retail, Healthcare&Banking
• OptimizeBAUSDLC Practices
HowWeDeliver
• Customer First Approach - Long termongoing engagements are testimony toour success
• Talent Farming – Thought leadership |Bootcamps | Embedded Factory
• Maturity & Agility – TechnologyInnovation | Reusable Best Practices |Early Adopters
©2016XavientInformationSystems
GlobalPresence
Tampa, FL
Atlanta,GA
Herndon,VA
Sterling, VA
Bethpage, NY
Frisco,TX
Englewood, CO
Denver, CO
St.Louis,MO
Bellevue,WA
SimiValley,CA
(Headquarters)
Noida
Bengaluru
OnshoreDeliveryCenter/
ClientLocation
OffshoreDeliveryCenter
Mississauga, Canada
©2016XavientInformationSystems
Xavient – EnablingDigitalTransformation
Mobility
OpenSource
BigData
IoT
UI/UX
• Dataingestionandanalysis fromsensors,edgesdevicesandsmartphoneapps.
• Transitionheavyweight,costlyandcompositesystemstoOpenSourceStackbasedEnterprise classsolutions
• CustomUI/UXbasedonthebusinessneeds.
• Deliveredinnovativesolutionshandlinglargevolumeofdata
• Mobilitysolutionaccelerators&frameworks toreducecosts&go-to-marketfaster
©2016XavientInformationSystems
OurDifferentiators
People
Pricing
Processes
Partnerships
Productivity
AmongstthebestglobalITtalent
Mature&Agileprocesses
In-houseR&Dcenter,TSG® forinnovationandautomation
Leveragingpartnershipsandstrategicalliances
CustomercentricGlobal Delivery
Highly-skilledconsultants &SMEs Lowattrition
ExcelinDevOpsmethodologies Extrabandwidthof10-15%
Bigenoughtoscale,smallenough toalign
LeadwithSolutionsandValue
Turningclientchallengesintosuccessstories
Flexiblepricingmodelsforcostefficiency
©2016XavientInformationSystems
Xavient’sBigDataandBIPracticeDifferentiators
Enterprise readysolutionwithcomplete
securityinplace.
100+ExpertsworkingonBigData/BI
projects.
Comprehensiveexperienceacrossthe
datastack
Readytouseaccelerators
StrategicallianceswithleadingvendorsDiversifiedIndustryExposure
©2016XavientInformationSystems
Xavientendtoendapproach
Application
Security
Compute
ResourceManagement
Storage
©2016XavientInformationSystems
ServicesthroughEVERYphaseofadoption
XavientisfocusedtoprovidelongtermpartnershipandsupportinallphasesofyourBigDataprojects
Services
Architecture
Development
Implementation
Operation
Managed
Services
©2016XavientInformationSystems
Xavient’sBigDataTeam
DeveloperandDataScientist
10+yearsindustryExperience3yearsBigDataexperience
DataArchitect/Engineer
10+yearsindustryExperience3yearsBigDataexperience
Sr.EngagementManager
8+yearsindustryExperience2+yearsBigDataexperience
Sr.Consultant
6+yearsindustryExperience2yearsBigDataexperience
• Develop dataingestionflowinbatchandrealtime
• Designanddevelopstatisticalproceduresandalgorithmsarounddatasources
• Recommend andbuildmodelsforvariousdatastudies,datadiscoveryandhypothesis
• Work withdeveloperstointegrateandpre-process dataforinputsintomodels
• Datacleanupandtransformation aspreparationforanalysis
• Focus areasinclude:statistics,appliedmath,machinelearning,modeldesignanddevelopment
• Configure andsetupvariousdevelopmentanddatasciencetools.
• Platformarchitecture analysisanddesign,including:datacenterlayout,hardware andnetworkconsiderations
• Securitymodeldesignandimplementationforauthentication,authorization,administrationandaudit
• Definerecommendedprocesses fordataingestion,management,transformation andegress.
• Perform installationforenterprise-scaleclusterenvironments.
• Resource managementatclusterandinfrastructurelevel.
• Facilitatetheselection ofappropriateBigDatatechnologyandtools
• Conduct technicalworkshops• Definetheoverallarchitecture forthesolutions/business applicationsandpreparetheenvironment
• Initialday-to-dayclusteroperations
• Overallresponsibilityforprojectsuccess
• Primarymanagement contact forthecustomer.
• CoordinatealltheProfessionalServicesresources
• Ensuretheprojectisdeliveredaccordingtoscheduleandbudget
• Reportstatus andmanageescalations.
• Leadtheimplementationofbusinessusecase(s).
• Provideexpertiseandguidanceduringtheprojectlifecycle.
• Providedesignanddevelopmentexpertiseforlarge-scale,clustereddataprocessing systems
• Developingest,transformation andegresscapabilitiesleveraginglatesttools suchasKafka, Storm, Pig,Hive,etc.
• Resolvetechnicalissuesintheenvironment
• Assist withthepreparationoftechnicaldeliverables,andreviewanddemonstrate thesystem andapplications
• Providetechnical supporteitheronsiteorremote.
©2016XavientInformationSystems
BigDataCase1:EnterpriseSocialNetworkingand
CollaborationPlatform
• Real timedata streaming• Search capability• Front endUI• Build recommendationmodel based onuser search
Xavient’sSolutionXavient’sSolution
• End to end design and implementation of enterpriseready solution/app to provide social networkingplatform
• Mobile app solution• Chat, Message exchange capabilities by leveraging
NoSQL databases• Resourcemanagement at the enterpriselevelTechnologies: Kafka, Storm, HDFS, Hive, HBase, ElasticSearch, Java, ApacheMesos/Mesosphere, ApacheMyraid
Xavient’sSolution
• Real time reports& analyticswith granular data• Health checkandmonitoringofkeyareas• Informed andquickdecisions• Both online& offlinemode access• Multiple Currency selection (USD& local currency)
KeyBenefits
BusinessRequirement
AnEnterpriseSocialNetworkingandCollaborationPlatform
©2016XavientInformationSystems
EnterpriseBasedDistributedArchitecture
(MultipleWeb&MultipleApp)Load
Balancer
WEB-n
ENCACHE
WEB-1
ENCACHE
APP
ENCACHE
ELASTICSEARCH-1
ELASTICSEARCH-n
………………
…………
…………
PostgreSQL-1
UTILITYSERVER
DOCSERVER
CHAT REDISSERVER…………
PostgreSQL-n
…………
MongoDB-1
MongoDB-n
…………
©2016XavientInformationSystems
BigDataCase2:Implementing aTrueEnterpriseDataLake
BusinessRequirement
• Store all the data at oneplace• Eliminate datasilos• Build COE
Xavient’sSolutionXavient’sSolution
• Developedoverallroadmapanddesignforenterprise‘datalake.’
• POCandHadoopknowledgesharingsessions• Securitysetupforstrongauthenticationandauthorization
• DataingestionfromRDBMStoHDFS/Hive.H• DefinedplanandhelpedimplementaCoE
Technologies used: HDP, Hive, Sqoop, AWS,Oozie
Xavient’sSolution
• Nomore silosof data• Fast response time delivered for long runningqueries
• Cost per TB dropped• InternalHadoop asa Service cloud• Store data for longer period of time• New technology stack increases employeeretention
KeyBenefits
AboutClientClient is a leading Satellite TVService Provider inUS
©2016XavientInformationSystems
• Operationaldashboards• Customerscorecards• CDRanalysis• ProactiveMaintenance• Infrastructureinvestment• Bandwidthallocation• Productdevelopment
EDW
MPP
CRM
ERP
DATASOURCES
DATAREPOS ANALYSIS
BillingData
SubscriberData
ProductCatalog
NetworkData
Clickstream
OnlineChat
SensorData
ServerLogs
CallDetailRecords
MerchantListing
DMP
SocialMedia
Ingest
SqoopFlume
HDFSPutWEBHDFS
Hadoop
Cluster
Batch
MapReduce
Script
PigSQL
HiveOnline
HbaseAccumulo
Real-Time
StormIn-Memory
SparkOthers
MetadataManagementHCatalog
MultitenantProcessing:YARN
HadoopOperatingSystem
LinearScale
Compute&HDFSStorage
Governance
tag,filter&process
Operations(Ambari)
Security(Knox,Kerberos,ApacheRanger)
1
N
. . . . . . ..
. .
.
. .
. . . . . . . . . .
. . .. . . .. . .
BigDataCase2:EnterpriseDataLake(Contd..)
©2016XavientInformationSystems
BigDataCase3:Dropcallandnetworkchangemonitoring in
RealTime
.
BusinessRequirement
• Xavient’sSolutionXavient’sSolution
Xavient’sSolutionKeyBenefits
• StorealltheCDRdataatoneplace“DataLake”• Realtimemonitoringofdroppedcalls• Realtimemonitoringofnetworkchanges
• Dataingestionflowwasbuiltusingflume,kafka andstormtostoredatainHDFS,HiveandHBaseincombinationwithApachePhoenix
• Solr andBananadeliveredthedashboardcapabilities• ApachePhoenixwastousedtorunqueriesagainstHBasetofinddropcallsandnetworkchange
• Longerdataretention• Fasterresponsetimefromthelargerdataset• CostreducedperTBbecauseofHadoopstack• Effectiveandfastalertingduringtheoutages• Abilitytorunpredictiveanalysisbasedonrealtimealerting
.AboutClientClient is a leading wirelessnetworkoperator in the US
©2016XavientInformationSystems
DataFlowandArchitecture
HDFS
InputFeed
Hive
Flume,Kafka,Storm
SearchUI
QueryUI
OutputFeed
SearchIndex
©2016XavientInformationSystems
BICase1:Executive DashboardForStrategicInsights
BusinessChallengesBusinessRequirement
Xavient’sSolution Xavient’sSolutionXavient’sSolution KeyBenefits
• Multi-channeldashboard• RealtimeanalyticsonCross-functionaldatawithvisualrepresentation• QuickerdecisionmakingforSeniorExecutives
• Crossplatformdashboardreportingapplication• Configurablebackgroundsyncforofflinedataaccess• CustomizedKPIsprovidingbird’seyeview• Dynamicreportingfrommultipledigitalmarketingchannelsviz.revenue,billings,costsandnumberofcampaigns
Technologies used: Informatica, Oracle, HTML5, Jquery,Java, Objective C
• Realtimereports&analyticswithgranulardata• Healthcheckandmonitoringofkeyareas• Informedandquickdecisions• Bothonline&offlinemodeaccess• MultipleCurrencyselection(USD&localcurrency)
AboutClientThe client is a Digital Marketing& Advertising ServiceProvider
©2016XavientInformationSystems
BICase2:InventoryandMerchandisingApp
BusinessChallengesBusinessRequirement
Xavient’sSolutionXavient’sSolution KeyBenefits
• Cross-Platformmobileapplicationthatgivesat-a-glanceviewofallSales,InventoryandGrossmarginforallproductsinastorefromanywhere
• Performancedashboardforassociatestomeasure,monitorandactonthesellablearticles
• Extendplatformtocatertocorporatevendorsforvariousproducts
• InterfacethemobileappwithSAPformetricsandreporting
• developedamobileapplicationto:• Browseproductcatalogue• Checkinventoryreportsandstatus• Tracksales(actualandplanned)• Readcustomerreviews• Findastore• Viewcurrentpromotionsofretailer
Technologiesused:PHP/JavaScript/MySQLObjectiveC(foriOSapp)Java(forAndroidapp)
• ConvenientandBettercustomerservice• Simpleuserfriendlydesignlayout• Customizedplatforms• Competitivepricing• BetterSalesandStockmanagement• SpeedscriptingtoincreaseTimetoMarket• Morevalueaddedservices
AboutClientOne of the leading retail chainsthe in US.