ECS with Nginx load Balancer and CloudArray VasilyPantyukhin
http://oldhenhut.com
ECSpreparation
CreateReplicationGroup
• ReplicationGroupcombinestwoECSsU300.
AddnewBaseURL
• Domainsuffixecspoc.localshouldbeusedastheBaseURL
• UseBaseURLwithoutNamespaceoptionchecked
CreateS3user
• CreateObjectuser
• GenerateS3password
4
CreateBucketandmodifyDNS
• CreateBucketwiththenamecabucket
• AddwildcardDNSrecord*.ecspoc.local
• Pingthebucket’sFQDN
5
ConfigureCloudArraypart1
UpgradeCloudArraytothelatestversion
• NavigatetoAdministration>CheckForUpgrade
• Upgradetoanewversion.
• Ittakesseveralminutes.
6
CheckCloudProviderwithoutaloadbalancer
• NavigatetoCloudArray>CloudProviders
• CloudProvidertype=ViPR
• EnableEncryption
• FortestpurposesCompressionshouldbedisabled(enabledonthescreenshot)
• Key=S3userconfiguredonECS
• Secret=S3passwordconfiguredonECSforthespecificuser
• Node=DNDsuffixecspoc.local
• BucketName=cabucket
• DisableSSL(canbeenabledinproduction)
• Port=9020(noSSL)
7
• Configureperformanceoptimizer
• CloudProviderisconfigured
8
DeleteCloudProvider
WecheckedthatCloudProvidercanbeconfiguredwithoutloadbalancer.Nowwecandeletethis
CloudProvider,configureNginxandconfigureanewCloudProvidervialoadbalancer.
• DeleteCloudProvider
9
ConfigureNginx
CheckNginxinstallation
• CheckNginxversionsandinstalledmodules
$/usr/local/nginx/sbin/nginx-Vnginxversion:nginx/1.9.2builtbygcc4.8.2(Ubuntu4.8.2-19ubuntu1)configurearguments:--with-http_ssl_module--with-http_spdy_module--with-http_realip_module--with-http_stub_status_module
• Usingcurl,youshouldseesomeHTMLoutputifyoutrytoaccessport80onnginxLinux
host.
$curl127.0.0.1<!DOCTYPEhtml><html><head><title>Welcometonginx!</title>
• Modifynginxconfig
$vi/etc/nginx/nginx.confhttp{server{listen*:9020;server_name*.ecs-poc-lb1.ecspoc.local;}}
• Reloadnginxconfig
$servicenginxrestartor
$kill-HUP`cat/var/run/nginx.pid`or
$nginx–sreload
• Inawebbrowser,gotoyourserver'sdomainnameorIPaddress:
http://ecs-poc-lb1.ecspoc.local:9020
10
Configureloadbalancing
InthattestSSLterminationwasnotused.IncaseifSSLisneeded,checktheAppendix.
• Modifyconfig
• ECS1nodesconfiguredastheupstreamservers
• DefaultloadbalancingmethodisRoundRobin
• Anylisteningportcanbespecified.Inthetest9020wasusedasforupstreamserversaccess.
• Bucketnamewillbeaddedtothebeginningofdomainprefixecspoc.localasservername.
ThatoptionisspecifiedasanobjectbaseURLinECS.
• Theoptionproxy_set_headerHost$hostisveryimportant.Bydefaultthe"Host"headeris
re-writtentotheIPaddressornameandportnumberoftheupstream(definedby
proxy_pass).When$hostvariableissetasaHostheader,hostnamefromtherequestwillbe
returnedback.
$vi/etc/nginx/nginx.confworker_processesauto;events{worker_connections512;}http{upstreamECS{serverECS-POC-NODE1.ecspoc.local:9020;serverECS-POC-NODE2.ecspoc.local:9020;serverECS-POC-NODE3.ecspoc.local:9020;serverECS-POC-NODE4.ecspoc.local:9020;}server{listen*:9020;server_name*.ecspoc.local;location/{proxy_passhttp://ECS;proxy_set_headerHost$host;}}}
• Checkifconfigurationfileisvalid.$/usr/sbin/nginx-tnginx:theconfigurationfile/etc/nginx/nginx.confsyntaxisoknginx:configurationfile/etc/nginx/nginx.conftestissuccessful
• Reloadnginxconfig
$servicenginxrestart
MoreaboutnginxDirectives:http://wiki.nginx.org/Modules
11
CheckaccesstoECSvialoadbalancer
• Inabrowsertypeloadbalancernamewith9020port.
• Checkthe“NoNamespaceForAnanymousRequest”errormessage.Thatiscorrectbehavior.
• AccessviaS3browsercanbecheckedaswell.
ConfigureCloudArraypart2
CreateCloudProvider
• Configurenewbucket.cabucket4.ecspoc.localshouldhaveIPaddressogtheloadbalancer.
• ConfigureCloudProviderthesamewayasitwasontheprevioustest.cabucket4shouldbe
specifiedastheBucket.
• FortestpurposesCompressionshouldbedisabled(enabledonscreenshots).
12
• ConfigurePerformanceOptimizer
• Checktheresult
• ConfigureaccesstoECSwithS3browser.
• NavigatetoAccount>EditAccount
• StorageType=S3CompatibleStorage.
• RESTEndpoint=cabucket4.ecspoc.local
• AccessKeyID=S3userconfiguredonECS
• SecretAccessKey=S3userpasswordgeneratedforS3user
13
• NavigatetoTools>Options>Connection.
• Uncheckusesecuretransfer(HTTPS).SSLwasnotusedinourtests.
• CheckifsomeobjectswerecreatedontheBucketusedbytheCloudArray.
ConfigureCache
• AddasecondHardDisktotheCloudArrayvirtualmachine
• 50Gbdiskinourcase
14
• NavigatetoCacheManagement>CacheSources
• Scanforadditionalcachesources
• 49GBPool_1wasdiscovered
• Addcache
15
• SelectPool_1asasource
• ChangeCachePageSizeto256KB.
Important,1MBdefaultPageSize(presentedonthescreenshot)canleadtoproblemswith
cachedraininginVMwareenvironment.AccordingtotheClouArraybestpracticedoc:“Cache
pagesgreaterthan1MBrequireaninternalsectorsizeof4KBratherthanthetypical512bytes.
Thelargersectorsizeisnotrecommendedfortransactionalworkloadssuchasdatabaseor
typicalfilesystemapplications.”But4Ksectorsizesaren’tcurrentlysupportedonESXi5.
• CheckSelectOneBigCacheUsingAllAvailableCapacity
• Bytheway,cacheisexpandableifothercachesourcesareavailable(notinourcase).
16
ConfigurePolicy
• NavigatetoProvisioningPolicies
• SelectCacheandCloudProvider
17
CloudArraylogs
• Incaseofanytroubles,pleasenavigatetoAdministration>SupportCaptureUtility>
GenerateSupportCapture
• Checkca_support.tgz>log>CloudArray.log
ConfigureNFS
• NavigatetoShares>Settings
• EnableNFS
18
• CreatenewShare
• SelectPolicy
• SpecifyCapacity
19
• AddClients
20
• Makeavailable
• LoginwithsshtoaLinuxmachine.
• CheckthatNFSexportisaccessible.
$showmount-e10.10.201.63Exportlistfor10.10.201.63:/export*/export/1TBNFS
21
• MountandcheckNFSexport.
$mkdir/NFSmnt$mount-tnfs10.10.201.63/export/backup/NFSmnt$df-hFilesystemSizeUsedAvailUse%Mountedon/dev/sda20G948M19G5%/udev119M4.0K119M1%/devtmpfs49M208K49M1%/runnone5.0M05.0M0%/run/locknone122M0122M0%/run/shm172.20.8.169/export/1TBNFS1024G70M492G0.01%/NFSmnt
• WritedatatoNFSexport.$cptest.txt/NFSmnt$ls/NFSmnttest.txt
ConfigureCIFS
• NavigatetoShares>Settings
• EnableCIFS
• SpecifyNameandDescription
• CheckAllowGuestAccessForCIFSShare
22
• CreatenewCIFSShare
• SelectPolicy
• SpecifyCapacity
• MakeAvailable
• ReadOnly=No
• Browseable=Yes
• CheckRead-WriteGuest
23
• LogintoWindowshost
• CheckCIFSShareavailability
• Checkreadandwrite
TestCloudArraybehavior
• Startwritingbigfiles
• NavigatetoDashboard
• CheckUsedandDirtycachecharacteristics(smoothlittlebitwavyincreasewithlater
decreaseareexpected).
24
• CheckCloudTransferRate(saw-toothcurveisexpected)
TestreplicationbetweenECSs
• LoginwithS3browsertoECS1
• Checkthenumberofobject
• Checkthesizeofthebucket
• LoginwithS3browsertoECS2
• Checkthenumberofobjectinthesamebucket(shouldbethesamenumberasforECS1)
• Checkthesizeofthesamebucket(shouldbethesamesizeasforECS1)
25
• ComparecapacityonECS1andECS2
• CurveonECS2shouldbemoresmoothcauseofECS1toECS2replicationnature
26
CheckRoundrobinloadbalancing
• ChecktrafficdistributionbetweenECSnodes
• Trafficshouldberoughlysimilar(otherloadbalancingtechnicsshouldgivebetterbalancing)
