Post on 15-Apr-2017
transcript
BetterBigDataAnalyticswithCAWorkloadAutomationforHadoop
SimonPearsonCATechnologiesSeniorEngineeringServicesArchitectSession#:MFX31E
@peasi02#CAWORLD
WorkloadAutomation– OffTheFloorSessions
2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.
Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.
ForInformationalPurposesOnlyTermsofthisPresentation
3 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Abstract
BigData.BigDeal.Theneedtominemassivesetsofinformationforuniqueinsightsintocustomerbehaviours,competitiveplaysandmarketfluctuationshastransformedbigdatainitiativesintoimperative,business-criticalpriorities.
Theopen-sourceApache™Hadoop®platformhasrapidlyemergedasthedominantmeansbywhichbusinessesprocess,analyseandextractinsightsfromtheirgrowingsetsofdata.
WhileHadoopdoesincludeabasicschedulerthatdeliverssomeautomation,itisfocusedprimarilyonjobsthatrunonHadoopclusters,anddoesn’tintegratewellwithotherworkloadautomationengines.
CAWorkloadAutomationAdvancedIntegrationforHadoopmakesitpossibletointegrateHadoopwithtraditionaljobs.
SimonPearsonCATechnologiesSeniorEngineeringServicesArchitect
4 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Agenda
INSTALLATION
CONFIGURATION
SUPPORTEDHADOOPJOBTYPES
EXAMPLES– DEFININGHADOOPJOBS
ANDFINALLY……
1
2
3
4
5
5 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationDownloadCAWAAIforHadoopfromsupport.ca.com
6 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationCheckthatyouhaveasupportedHadoopdistribution
7 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationSupportingtheseCAWorkloadEngines
FormoreinformationaboutusingAdvancedIntegrationforHadoop,refertothedocumentationforyourschedulingengine:
•CAWorkloadAutomationAEandWorkloadControlCenter•CAWorkloadAutomationDE•CAWorkloadAutomationESP
8 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationHavethecorrectversionofthesystemagentinstalled
9 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationOrelsethiswillhappen
10 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationMounttheISOandruntheinstaller
11 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationTheHadoopAIinstallerruns………
12 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationCreateagentdefinitionfortheHadoopplugin
13 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationCreateconnectionprofile
ThisidentifiestheDBIwanttoextractdatafrom
14 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationSecurityProfile– IDandPWforlogintosourceDBforSqoop jobs
15 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationSecurityProfile– AccessrighttoHadoopHDFSfilesystem
16 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationSecurityProfile– IDwithrightstoexecuteHivejobs
17 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
SupportedHadoopJobTypes• HDFS- FileSystemactions
• Hive- Scriptexecution
• Pig- Scriptexecution
• Sqoop – DatabaseImport/Exportoperations
• Oozie – ExecuteOozie Workflows
18 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- HDFSOperationssupported– CreateDir,CreateFile,DeleteFile,DeleteDir
19 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- HDFSExample– Deletefile
HadoopconnectionandHDFSsecurityprofileselectedfromthosedefinedinpreviousstep
20 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- HDFSExample– MakeDirectory
ThefourHDSFoperationsSupportedcanbeselectedfromthedropdownlist
21 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- HDFSViewtheresultsviathejoblog
22 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- SqoopOperationssupported:
23 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- SqoopSimpleexample
24 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- SqoopSpoolFile
25 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- PigJobDefinition
26 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- PigScripttoexecuteandsampledata
27 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- PigSpoolFile
28 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
IntegratingHadoopwithExternalWorkflows
29 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
AndFinally…..WhatifyouarerunningCAWAAE?
30 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
AndFinally…..ThesameHadoopjobtypesareavailableintheQuickEditor
31 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
AndFinally…..HadoopjobtypesalsoavailableinApplicationEditor
32 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Questions?
Sr.EngineeringServicesArchitectSimonPearson@ca.com
SimonPearson
@cainc
Slideshare.net/CAinc
Linkedin.com/company/ca-technologies
CA.com
34 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Thankyou
Stayconnectedatcommunities.ca.com