1©Cloudera,Inc.Allrightsreserved.
FredKoopmansSr.DirectorofProductManagement
EnablingSecureHadoopEnvironments
1©Cloudera,Inc.Allrightsreserved.
2©Cloudera,Inc.Allrightsreserved.©Cloudera,Inc.Allrightsreserved.
Thefutureofgovernmentisdatamanagement
What’syourstrategy?
3©Cloudera,Inc.Allrightsreserved.
Cloudera’sEnterpriseDataHubmakesitpossible
Bringallyourdatatogether
Bringallyourknowledgeworkerstogether
Bringallyourdataapplicationstogether
Runanywhere
4©Cloudera,Inc.Allrightsreserved.
But,anEDHcanalsomakeajuicytarget
Alldatainoneplace?
Provideeveryoneaccesstooneplatform?
HowdoIensuresecurity?
HowdoImaintainsecurityastheplatformgrows?
5©Cloudera,Inc.Allrightsreserved.
4focusareasforsecuringyourHadoopenvironment
AccessDefiningwhatusersandapplicationscan
dowithdata
TechnicalConcepts:PermissionsAuthorization
DataProtectingdatainthe
clusterfromunauthorizedvisibility
TechnicalConcepts:Encryption,Keymanagement,Datamasking
VisibilityReportingonwheredatacamefromandhowit’sbeingused
TechnicalConcepts:AuditingLineage
PerimeterGuardingaccesstothe
clusteritself
TechnicalConcepts:Authentication
Networkisolation
6©Cloudera,Inc.Allrightsreserved.
PerimeterSecurityRequirements
PreserveuserchoiceofHadoopservice
Conformtocentrallymanagedauthenticationpolicies
Implementwithexistingstandardsystems
ClouderaManager
PerimeterGuardingaccesstothe
clusteritself
TechnicalConcepts:Authentication
Networkisolation
7©Cloudera,Inc.Allrightsreserved.
Authentication
• Kerberosauthentication• AutomationprovidedbyClouderaManagertoleverageActiveDirectory
CDHcomponents
• LDAPandSAMLauthentication
WebUIs
• LDAPandKerberosauthentication
SQLAccess
Userauthenticates
toAD
Authenticatedusergets
KerberosTicket
TicketgrantsaccesstoServices
e.g.ImpalaUser[ssmith]Password[*****]
8©Cloudera,Inc.Allrightsreserved.
NetworkIsolation
EdgeNodes
WebServers
Onlyadmins permittedaccesstofullcluster
Mostusersonlypermittedaccesstogatewayservicesrunningonclusterperiphery
9©Cloudera,Inc.Allrightsreserved.
AccessSecurityRequirements
Keeponlyonelogicalcopyofdata
Createonlyonepermissionsruleforallapplicationsandallcomputeframeworks
Enforcepermissionsatcolumnandrowlevelgranularity
AccessDefiningwhatusersandapplicationscan
dowithdata
InfoSecConcept:Authorization
ApacheSentry,RecordService
10©Cloudera,Inc.Allrightsreserved.
FilesystemHDFS
STORA
GECO
MPU
TE
HIVE,IMPALA
SPARK,MR
EarlydaysofHadoop:Storagepermissionsonly
• Simple“AllorNothing”permissionsforeachfile/table
But...• Tablesoftencontain10s– 100sofcolumns• Notallusersareallowedtoseeallcolumnsandrows
APPS DATAMEER SASPLATFORA TABLEAU ETC...
11©Cloudera,Inc.Allrightsreserved.
Usecasesforfine-grainedaccesscontrol
Columns• Differentusergroupsneedaccesstodifferentcolumns(ex:socialsecuritynumbers)
Rows• Differentusergroupsneedaccesstodifferentrecords(ex:bysecurityclearancelevel)
12©Cloudera,Inc.Allrightsreserved.
FilesystemHDFS
STORA
GECO
MPU
TE
HIVE,IMPALA
SPARK,MR
Fewyearsago:Storagepermissions+SQLAuth.
• Addscolumnandrow-levelpermissions
But...• Createsduplicatedata,duplicatepermissionsrulestosupportSparkandMR
APPS DATAMEER SASPLATFORA TABLEAU ETC...
ApacheSentry
X
13©Cloudera,Inc.Allrightsreserved.
FilesystemHDFS
STORA
GE
APACHESENTRY,RECORDSERVICE
COMPU
TE
HIVE,IMPALA
SPARK,MR
UpNext:ApacheSentry+RecordService*workingtogether
• ColumnandRow-levelPermissions• Onecopyofdata• Onesetofpermissions
APPS DATAMEER SASPLATFORA TABLEAU ETC...
*inbeta
14©Cloudera,Inc.Allrightsreserved.
Fine-grainedaccesscontrolwithout Sentry&RecordService*
Date/time Accnt # SSN Asset Trade Country
09:33:11 16-Feb-2015
0234837823 238-23-9876
AZP Sell US
11:33:0116-Feb-2015
3947848494 329-44-9847
TBT Buy EU
14:12:3416-Feb-2015
4848367383 123-56-2345
IDI Sell UK
09:22:03 16-Feb-2015
3485739384 585-11-2345
ICBD Buy US
11:55:3316-Feb-2015
3847598390 234-11-8765
FWQ Buy US
10:22:55 16-Feb-2015
8765432176 344-22-9876
UAD Buy UK
13:45:24 16-Feb-2015
3456789012 412-22-8765
NZMA Sell EU
09:03:4416-Feb-2015
4857389329 123-44-5678
TMV Buy US
15:55:5516-Feb-2015
4756983234 234-76-9274
DRW Buy UK
Date/time Accnt # SSN Asset Trade Country
14:12:3416-Feb-2015
4848367383 123-56-2345
IDI Sell UK
10:22:55 16-Feb-2015
8765432176 344-22-9876
UAD Buy UK
15:55:5516-Feb-2015
4756983234 234-76-9274
DRW Buy UK
Date/time Accnt # SSN Asset Trade Country
11:33:0116-Feb-2015
3947848494 329-44-9847
TBT Buy EU
13:45:24 16-Feb-2015
3456789012 412-22-8765
NZMA Sell EU
Date/time Accnt # SSN Asset Trade Country
09:33:11 16-Feb-2015
0234837823 238-23-9876
AZP Sell US
09:22:03 16-Feb-2015
3485739384 585-11-2345
ICBD Buy US
11:55:3316-Feb-2015
3847598390 234-11-8765
FWQ Buy US
09:03:4416-Feb-2015
4857389329 123-44-5678
TMV Buy US
• SplittheHDFSpermissionsoriginalfile• Usetolimitaccess
15©Cloudera,Inc.Allrightsreserved.
Fine-grainedaccesscontrolwith Sentry&RecordService*
• Sentry:Definepermissionsatthetable,columnandrowlevels• Sentry+RecordService:Enforcetheseacrossallaccesspaths
Date/time Accnt # SSN Asset Trade Country
09:33:11 16-Feb-2015
0234837823 238-23-9876
AZP Sell US
11:33:0116-Feb-2015
3947848494 329-44-9847
TBT Buy EU
14:12:3416-Feb-2015
4848367383 123-56-2345
IDI Sell EU
09:22:03 16-Feb-2015
3485739384 585-11-2345
ICBD Buy US
11:55:3316-Feb-2015
3847598390 234-11-8765
FWQ Buy US
10:22:55 16-Feb-2015
8765432176 344-22-9876
UAD Buy EU
13:45:24 16-Feb-2015
3456789012 412-22-8765
NZMA Sell EU
Column-LevelControls
Row-LevelCon
trols
Date/time Accnt # SSN Asset Trade Country
09:33:11 16-Feb-2015
0234837823 238-23-9876
AZP Sell US
11:33:0116-Feb-2015
3947848494 329-44-9847
TBT Buy group2
14:12:3416-Feb-2015
4848367383 123-56-2345
IBM Sell group3
09:22:03 16-Feb-2015
3485739384 585-11-2345
ICBD Buy US
11:55:3316-Feb-2015
3847598390 234-11-8765
FWQ Buy US
10:22:55 16-Feb-2015
8765432176 344-22-9876
UA Buy group3
13:45:24 16-Feb-2015
3456789012 412-22-8765
AMZN Sell group2
Column-LevelControls
Row-LevelCon
trols
XXX-XX
XXX-XX
XXX-XX
WhatU.S.BrokersSee
Hive,Impala,MR,Spark,Pig
SingleHDFSfile:
16©Cloudera,Inc.Allrightsreserved.
VisibilitySecurityRequirements
Complywithpoliciesforaudit,dataclassification,andlineage
Centralizetheauditrepository
VisibilityReportingonwheredatacamefromandhowit’sbeingused
InfoSecConcept:AuditingLineage
ClouderaNavigator
17©Cloudera,Inc.Allrightsreserved.
Audit&LineageTrustedforproduction• 100sofcustomerdeploymentsofClouderaNavigatoroverlast3+years
Compliance-ready• OnlyHadoopdistributiontopassPCIaudit
Detailed• Columnandrowlevelaccesstrail
Playsnicelywithothers• Integratedwiththeleadingpartnersolutions
18©Cloudera,Inc.Allrightsreserved.
DataSecurityRequirements
Performanalyticsonregulateddata
Encryptdata,conformtokeymanagementpolicies,protectfromroot
IntegratewithexistingHSMaspartofkeymanagementinfrastructure
DataProtectingdatainthe
clusterfromunauthorizedvisibility
InfoSecConcept:Encryption,Keymanagement,Datamasking
NavigatorEncrypt&KeyTrustee
19©Cloudera,Inc.Allrightsreserved.
ComprehensiveDataSecurity
Manager Navigator
Impala Hive
HDFS HBase
Sentry
NavigatorKeyTrustee
LogFiles
MetadataStore
EncryptedData
EncryptionKey
Legend
IngestPaths,Temp/Spillfiles
HSM(optional)
ALLdataonthewire• ALLdataatrest:HDFS,HBase,metadatadatabases,tempfiles,ingestpaths
• Automatedkeyreplication&backup• HSMbackedkeyprotection
• Sensitivedatainlogs• Passwordsinconfig files
Encryption
KeyManagement
DataMasking
20©Cloudera,Inc.Allrightsreserved.
Cloudera’scomprehensive,compliance-readysecuritysolution
AccessDefiningwhatusersandapplicationscan
dowithdata
TechnicalConcepts:PermissionsAuthorization
DataProtectingdatainthe
clusterfromunauthorizedvisibility
TechnicalConcepts:Encryption,Keymanagement,Datamasking
VisibilityReportingonwheredatacamefromandhowit’sbeingused
TechnicalConcepts:AuditingLineage
ClouderaManager ApacheSentry&RecordService
ClouderaNavigator NavigatorEncrypt&KeyTrustee
PerimeterGuardingaccesstothe
clusteritself
TechnicalConcepts:Authentication
Networkisolation
21©Cloudera,Inc.Allrightsreserved.
Beyondtraditionalsecuritycontrols
Automateddiscoveryandtaggingofsensitivedata• Automaticallyscanforprotectedattributetypes• Automaticallyapplyauthorizationandencryptionpolicy
“Followthedata”authorizationandprotectionpolicies• Leveragelineagedatataggingenforceauthorizationandencryptionpolicy• Eliminatemanualconfigurationofsecurityforeachnewtableandcolumn
Adminsfocusedonexceptionhandlingduetoinsufficientaccess
©Cloudera,Inc.Allrightsreserved. 22
ThankyouThankYouFredKoopmans