Pivotal Container Service (PKS) Documentation › pdfs › pks-1-0.pdfNSX-T 2.1 Advanced Edition...

PivotalContainerService

(PKS)

Version1.0

Published:16Oct2018

©2018PivotalSoftware,Inc.AllRightsReserved.

2471718212224252729303234404456575864667881848591102103105107109110111112114115116117120121122124125126127128129130

TableofContents

TableofContentsPivotalContainerService(PKS)PKSReleaseNotesPKSConceptsPKSClusterManagementPKSAPIAuthenticationLoadBalancersinPKSPKSPrerequisitesInstallingthePKSCLIInstallingtheKubernetesCLIPreparingtoInstallPKSonvSpherevSpherePrerequisitesandResourceRequirementsFirewallPortsandProtocolsRequirementsforvSpherewithNSX-TPreparingtoDeployPKSonvSphereDeployingOpsManagertovSphereConfiguringOpsManageronvSpherePreparingtoInstallPKSonGCPGCPPrerequisitesandResourceRequirementsPreparingtoDeployPKSonGCPDeployingOpsManagertoGCPConfiguringOpsManageronGCPConfiguringaGCPLoadBalancerforthePKSAPIConfiguringaGCPLoadBalancerforPKSClustersInstallingPKSInstallingandConfiguringPKSInstallingandConfiguringPKSwithNSX-TIntegrationUpgradingPKSWhatHappensDuringPKSUpgradesUpgradePKSMaintainWorkloadUptimeConfiguretheUpgradePipelineManagingPKSConfigurePKSAPIAccessManageUsersinUAAManagePKSDeploymentswithBOSHAddCustomWorkloadsDownloadClusterLogsServiceInterruptionsDeletePKSUsingPKSCreateaClusterRetrieveClusterCredentialsandConfigurationViewClusterListViewClusterDetailsViewClusterPlansUsingDynamicPersistentVolumesScaleExistingClustersAccessDashboard

©CopyrightPivotalSoftwareInc,2013-2019 2 1.0

131133134135136137138139140143148

DeployandAccessBasicWorkloadsDeleteaClusterLogOutofthePKSEnvironmentUsingHelmwithPKSConfigureTillerInstallConcourseUsingHelmDiagnosingandTroubleshootingPKSDiagnosticToolsTroubleshootingPKSCLIPKSSecurityDisclosureandReleaseProcess


PivotalContainerService(PKS)Pagelastupdated:

PivotalContainerService(PKS)enablesoperatorstoprovision,operate,andmanageenterprise-gradeKubernetesclustersusingBOSHandPivotalOpsManager.

OverviewPKSusestheOn-DemandBroker todeployCloudFoundryContainerRuntime ,aBOSHreleasethatoffersauniformwaytoinstantiate,deploy,andmanagehighlyavailableKubernetesclustersonacloudplatformusingBOSH.

AfteroperatorsinstallthePKStileontheOpsManagerInstallationDashboard,developerscanprovisionKubernetesclustersusingthePKSCommandLineInterface(PKSCLI),andruncontainer-basedworkloadsontheclusterswiththeKubernetesCLI,kubectl.

PKSisavailableaspartofPivotalCloudFoundry orasastand-aloneproduct.

WhatPKSAddstoKubernetesThefollowingtabledetailsthefeaturesthatPKSaddstotheKubernetesplatform.

Feature IncludedinK8s IncludedinPKS

Singletenantingress ✓ ✓

Securemulti-tenantingress ✓

Statefulsetsofpods ✓ ✓

Multi-containerpods ✓ ✓

Rollingupgradestopods ✓ ✓

Rollingupgradestoclusterinfrastructure ✓

Podscalingandhighavailability ✓ ✓

Clusterprovisioningandscaling ✓

MonitoringandrecoveryofclusterVMsandprocesses ✓

Persistentdisks ✓ ✓

Securecontainerregistry ✓

Embedded,hardenedoperatingsystem ✓

FeaturesPKShasthefollowingfeatures:

KubernetesCompatibility:ConstantcompatibilitywithcurrentstablereleaseofKubernetes

Production-ready:Highlyavailablefromapplicationstoinfrastructure,withnosinglepointsoffailure

BOSHadvantages:Built-inhealthchecks,scaling,auto-healingandrollingupgrades

Fullyautomatedoperations:Fullyautomateddeploy,scale,patch,andupgradeexperience

Multi-cloud:Consistentoperationalexperienceacrossmultipleclouds

GCPAPIsaccess:TheGoogleCloudPlatform(GCP)ServiceBrokergivesapplicationsaccesstotheGoogleCloudAPIs,andGoogleContainerEngine(GKE)consistencyenablesthetransferofworkloadsfromortoGCP

OnvSphere,PKSsupportsdeployingandrunningKubernetesclustersinair-gappedenvironments.

PKSComponents


https://docs.pivotal.io/svc-sdk/odb/index.htmlhttps://docs-kubo.cfapps.io/https://docs.pivotal.io

ThePKScontrolplanecontainsthefollowingcomponents:

AnOn-DemandBroker thatdeploysCloudFoundryContainerRuntime (CFCR),anopen-sourceprojectthatprovidesasolutionfordeployingandmanagingKubernetes clustersusingBOSH .

AServiceAdapter

ThePKSAPI

FormoreinformationaboutthePKScontrolplane,seePKSClusterManagement.

ForadetailedlistofcomponentsandsupportedversionsbyaparticularPKSrelease,seethePKSReleaseNotes.

PKSConceptsForconceptualinformationaboutPKS,seePKSConcepts.

PKSPrerequisitesForinformationabouttherequirementsforinstallingPKS,seePKSPrerequisites.

PreparingtoInstallPKSToinstallPKS,youmustdeployOpsManagerv2.0orv2.1.YouuseOpsManagertoinstallandconfigurePKS.

IfyouareinstallingPKStovSphere,youcanalsoconfigureintegrationwithNSX-TandHarbor.

Consultthefollowingtableforcompatibilityinformation:

IaaS OpsManagerv2.0 NSX-T Harbor

vSphere Required Available Available

GCP Required NotAvailable NotAvailable

ForinformationaboutpreparingyourenvironmentbeforeinstallingPKS,seethetopicthatcorrespondstoyourcloudprovider:

PreparingtoInstallPKSonvSphere

PreparingtoInstallPKSonGCP

InstallingPKSForinformationaboutinstallingPKS,seeInstallingandConfiguringPKS.

UpgradingPKSForinformationaboutupgradingthePKStileandPKS-deployedKubernetesclusters,seeUpgradingPKS.

ManagingPKSForinformationaboutconfiguringauthentication,creatingusers,andmanagingyourPKSdeployment,seeManagingPKS.

UsingPKSForinformationaboutusingthePKSCLItocreateandmanageKubernetesclusters,seeUsingPKS.


https://docs.pivotal.io/svc-sdk/odb/https://docs-kubo.cfapps.iohttps://kubernetes.io/docs/home/https://bosh.io/docs

DiagnosingandTroubleshootingPKSForinformationaboutdiagnosingandtroubleshootingissuesinstallingorusingPKS,seeDiagnosingandTroubleshootingPKS.

[email protected].


mailto:[email protected]

PKSReleaseNotesPKS(PivotalContainerService)isusedtocreateandmanageon-demandKubernetesclustersviathePKSCLI.

v1.0.4ReleaseDate:May21,2018

UpgradeProcedure

ToupgradetoPKSv1.0.4,followtheproceduresinUpgradePKS.

FeaturesUpdatesKubernetestov1.9.7.

ComponentVersionsPKSv1.0.4includesorsupportsthefollowingcomponentversions:

ProductComponent VersionSupported Notes

PivotalCloudFoundryOperationsManager(OpsManager)

2.0.Xand2.1.X SeparatedownloadavailablefromPivotalNetwork

vSphere

6.5,6.5U1,and6.5U2-EditionsvSphereEnterprisePlusEdition

vSpherewithOperationsManagementEnterprisePlus

vSphereversionssupportedforPivotalContainerService(PKS)

VMwareHarborRegistry 1.4.2 SeparatedownloadavailablefromPivotalNetwork

NSX-T 2.1AdvancedEdition AvailablefromVMware

Stemcell 3468.XFloatingstemcelllineavailabletodownloadfromPivotalNetwork

Kubernetes 1.9.7* PackagedinthePKSTile(CFCR)

CFCR(Kubo) 0.13 PackagedinthePKSTile

Golang 1.9.5 PackagedinthePKSTile

NCP 2.1.3 PackagedinthePKSTile

KubernetesCLI 1.9.7*SeparatedownloadavailablefromthePKSsectionofPivotalNetwork

PKSCLI 1.0.3-build.15SeparatedownloadavailablefromthePKSsectionofPivotalNetwork

UAA 55

*Componentsmarkedwithanasteriskhavebeenpatchedtoresolvesecurityvulnerabilitiesorfixcomponentbehavior.

KnownIssuesThissectionincludesknownissueswithPKSv1.0.4andcorrespondingworkarounds.

Note:UpgradetoPKSv1.0.4fromeitherPKSv1.0.2orPKSv1.0.3.DonotupgradePKSv1.0.0directlytov1.0.4.Instead,upgradetov1.0.2,thenv1.0.4.Alternatively,doauniqueinstallofPKSv1.0.4.


AccesstotheKubernetesAPIisUnavailableDuringUpgrades

PKSupgradesincludeupgradestothemasternode.Whilethemasternodeisundergoinganupgrade,theKubernetesAPIisunavailable.

IfyouattempttoaccesstheAPIduringanupgrade,youwillnotbeabletoconnect.

StemcellUpdatesCauseAutomaticVMUpgrading

EnablingtheUpgradeallclusterserrandallowsautomaticupgradingforVMsinyourdeployment.PivotalrecommendsenablingthiserrandtoensurethatalldeployedclusterVMsarepatched.

WhenyouenabletheUpgradeallclusterserrand,thefollowingactionscancausedowntime:

UpdatingthePKStilewithanewstemcelltriggersupdatingeachVMineachcluster.

UpdatingothertilesinyourdeploymentwithnewstemcellscausestheupgradingofthePKStile.

UpgradeErrandFailswithFailedDeployments

TheUpgradeallclusterserrandfailsifanydeploymentsareinafailedstate.

Toworkaroundthisissue,deletethefailedclusterusingthePKSCLIorredeploythefailedclusterwiththeBOSHCLItoensuretheclusterisinasuccessfulstate.

PodsLoseNetworkConnectivityAfterVMColdMigration

WhenaKubernetesclusterworkerVMgoesthroughcoldmigrationinvSphere,newlyprovisionedpodslosenetworkconnectivity.

Thisissuecanoccurunderthefollowingconditions:

WhentheVMispoweredoffandissubjecttocoldmigration,andtheVMmovestoadifferentESXihost

WhentheVMispoweringonandissubjecttoDistributedResourceScheduler(DRS)beforethepowerupcompletes

WhenthevNICoftheVMisdetachedandreattached

Toworkaroundthisissue,deletetheworkerVM.BOSHrecreatestheworkerVMandrestoresnetworkconnectivity.  

KubernetesClusterCreationFailsifNSX-TManagerPasswordBeginswithCertainSpecialCharacters

IfyouselectNSX-TasaContainerNetworkTypeinPKSandyourNSX-TManagerpasswordbeginswithan @ , $ , ^ , ' ,orspacecharacter,Kubernetesclustercreationfails.Toresolvethisissue,resetyourNSX-TManagerpasswordsothatitdoesnotbeginwithanyofthesecharacters.AfterresettingyourNSX-TManagerpassword,reconfigureyourNSX-TManagercredentialsinthePKStilewiththeupdatedpassword.

v1.0.3ReleaseDate:May4,2018

UpgradeProcedure

ToupgradetoPKSv1.0.3,performthefollowingsteps:

1. Downloadthelatest3468.xstemcellfromPivotalNetwork andconfigurethePKStilewiththestemcell.

2. Createanewworkernodeserviceaccount.

TocreatetheserviceaccountonGCP,seeCreatetheWorkerNodeServiceAccount.

Note:TheonlysupportedupgradepathforPKSv1.0.3isfromPKSv1.0.2.DonotupgradePKSv1.0.0directlytov1.0.3.Instead,upgradetov1.0.2,thenv1.0.3.Alternatively,doauniqueinstallofPKSv1.0.3.


https://network.pivotal.io/products/stemcells

TocreatetheserviceaccountonvSphere,seeCreatetheWorkerNodeServiceAccount.

3. FollowtheproceduresinUpgradePKS.WhenconfiguringtheKubernetesCloudProviderconfigurationscreeninthePKStile,configurethenewworkernodecredentialsorserviceaccountkeyasappropriateforyourIaaS.

FeaturesSeparatesthemasterandworkernodecredentials.

UpdatesKubernetestov1.9.6.

UpdatesGolangtov1.9.5.





vSphere

6.5and6.5U1-EditionsvSphereEnterprisePlusEdition





Stemcell 3468.XFloatingstemcelllineavailabletodownloadfromPivotalNetwork



Golang 1.9.5* PackagedinthePKSTile

NCP 2.1.3* PackagedinthePKSTile


PKSCLI 1.0.3-build.15*SeparatedownloadavailablefromthePKSsectionofPivotalNetwork

UAA 55*






















StatefulSetsPodFailureAfterRecreatingaVM

WhenusingvSpherewithNSX-Tintegration,ifyourecreateanodethathostsaStatefulSetspod,thepodcangetstuckina ContainerCreating state.Thepodemitsawarningeventwitha FailedCreatePodSandBox reason.ThisissueaffectsStatefulSetspodscreatedbeforePKSv1.0.3.

AfixforthisbugisincludedinPKSv1.0.3,butthefixappliesonlytoStatefulSetscreatedusingPKSv1.0.2orlater.AfterupgradingPKStov1.0.3,manuallydeletingandrecreatingallpreexistingStatefulSetspodsisrecommended,eveniftheyareinarunningstate.

TogetallStatefulSetspods,runthefollowingcommandoneveryKubernetesclusterusingtheKubernetesadminuserpermissions:

$ kubectl get pods -l "statefulset.kubernetes.io/pod-name" \-o wide --all-namespaces

Foreachresult,deletethepodbyrunningthefollowingcommand:

$ kubectl delete pod POD-NAME -n POD-NAMESPACE

Youdonotneedtomanuallyrecreatethedeletedpods.KubernetesdetectsaStatefulSetwithmissingpodsandautomaticallyrecreatesthepods.

[KubernetesBug]UpgradingaClusterAffectsPersistentWorkloadUptime

Duringanupgradetov1.0.3onvSphere,persistentstoragevolumesdonotreattachtopodsuntilallworkernodeshavebeenupgraded,whichresultsinworkloaddowntimeuntiltheentireclusterisupgraded.

Thisissueoccurswhenyoudeployapodwithpersistentstorageattached,drainthenode,andthenimmediatelydeletethenodeVM.

TheexpectedbehaviorisforpersistentdiskstoreattachtotheupgradedVMsafterthepodisrestored.However,aKubernetesbugpreventsthediskfromreattaching.PKSv1.0.3worksaroundthisbugbyattachingthevolumesafterallworkersareupgraded.

Formoreinformation,seetheKubernetesissueonGitHub .

Inrarecases,podswithpersistentvolumescanstayin ContainerCreating state.Ifyouseetheerror FailedMountUnabletomountvolumesforpodPOD-NAME

,

performthefollowingsteps:

1. Findtheproblemnodebyrunning kubectl describe pod POD-NAME .


https://github.com/kubernetes/kubernetes/issues/61707

2. Preventschedulingonthenodethatrunsthepodbyrunning kubectl cordon NODE-NAME .

3. Deletepodbyrunning kubectl delete pod POD-NAME .

4. Waitforpodtoberescheduledandenter Running state.Thismaytakeseveralminutes.

5. Resumeschedulingonthenodethatrunsthepodbyrunning kubectl uncordon NODE-NAME .



v1.0.2ReleaseDate:April12,2018

UpgradeProcedureToupgradetoPKSv1.0.2,performthefollowingsteps:

1. Downloadthedocker_ctl script.

2. Downloadthedocker_ctl_update.sh script.

3. LogintotheBOSHDirectorbyrunning bosh -e MY-ENVIRONMENT log-in fromaVMthatcanaccessyourPKSdeployment.ReplaceMY-ENVIRONMENT withtheBOSHaliasforyourPKSenvironment.SeeManagePKSDeploymentswithBOSHformoreinformation.

IfyouchoosetologinfromtheOpsManagerVM,performthefollowingsteps:

a. Run sudo apt-get update .b. Run sudo apt-get install jq .

4. Run export BOSH_ENVIRONMENT=MY-ENVIRONMENT .Replace MY-ENVIRONMENT withtheBOSHaliasforyourPKSenvironment.

5. Runthe docker_ctl_update.sh script.ThisscriptcontainsthefixtocorrectlyunmountDockeroverlays.Seethecorrespondingknownissueformoreinformation.

6. Downloadthelatest3468.xstemcellfromPivotalNetwork andconfigurethePKStilewiththestemcell.

7. FollowtheproceduresinUpgradePKS.

FeaturesUpdatesKubernetestov1.9.5.

UpdatesGolangtov1.9.4.

FixedIssues

GeneralWorkernodesarenowdrainedbeforetheystopinordertominimizeworkloaddowntimeduringarollingupgrade.

UAAcredentialsandvCenterpasswordsnolongerappearinBOSHlogs.

BOSHDNSnolongercausesworkernodestofailafteramanualrestart.

TheKubernetesControllerManagercertificatenolongercontainsadditionalwhitespace.


http://localhost:9292/runtimes/pks/1-0/1.0.2/stemcell/docker_ctlhttp://localhost:9292/runtimes/pks/1-0/1.0.2/stemcell/docker_ctl_update.shhttps://network.pivotal.io/products/stemcells

Drainusernowhasadditionalpermissionstoremovereplicationcontroller-ownedpods.

UnmountingDockeroverlayvolumesnolongercausesBOSHunmountfailures.

Addressesupgradeissuesinconstrainedenvironments.

vSpherevSphereNSX-TintegrationnowworkswithBOSHstemcellv3468.25andlater.

ForvSpherewithNSX-T,thepodlogicalswitchport(LSP)isnowupdatedwhenyourecreatetheVMthathoststhepod.SeeStatefulSets intheKubernetesdocumentationandtheknownissuebelowformoreinformation.

Addedsupportforspecialcharacters # , & , ; , " , ' , ^ , \ ,space( ), % ,and ! invCenterpasswordsintheKubernetesCloudProvidertileconfigurationpage.

DrainscriptnowdeletesnodestofixavSphereissuewherenodenameschangedbetween1.9.2and1.9.5.





vSphere






Stemcell 3468.X*FloatingstemcelllineavailabletodownloadfromPivotalNetwork



Golang 1.9.4* PackagedinthePKSTile

NCP 2.1.2* PackagedinthePKSTile


PKSCLI 1.0.2-build.4*SeparatedownloadavailablefromthePKSsectionofPivotalNetwork






VolumeUnmountFailureAfterStemcellUpgrade

DuringanupgradetoPKSv1.0.2,BOSHcanfailtounmountthe /var/vcap/store volumeonworkernodes.ThisisduetoanissuewiththeDockerBOSH


https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

releaseinstalledbythePKSv1.0.0tile.

InthisversionoftheBOSHrelease,Dockeroccasionallyfailstounmountalloverlayswhenstoppinganode.WhenyouupgradethestemcellforthePKStile,BOSHrecreatesVMsandcanfailtocorrectlyunmountDockeroverlays.

Toavoidthisissue,followthestepsintheUpgradeProceduresectionwhenyouupgradethePKStile.The docker_ctl_update.sh scriptcorrectlyunmountsDockeroverlaysbyreplacingthe docker_ctl scriptonallworkernodesthathaveDockerdeployed.
















StatefulSetsPodFailureAfterRecreatingaVM

WhenusingvSpherewithNSX-Tintegration,ifyourecreateanodethathostsaStatefulSetspod,thepodcangetstuckina ContainerCreating state.Thepodemitsawarningeventwitha FailedCreatePodSandBox reason.ThisissueaffectsStatefulSetspodscreatedbeforePKSv1.0.2.

AfixforthisbugisincludedinPKSv1.0.2,butthefixappliesonlytoStatefulSetscreatedusingPKSv1.0.2orlater.AfterupgradingPKStov1.0.2,manuallydeletingandrecreatingallpreexistingStatefulSetspodsisrecommended,eveniftheyareinarunningstate.

TogetallStatefulSetspods,runthefollowingcommandoneveryKubernetesclusterusingtheKubernetesadminuserpermissions:

$ kubectl get pods -l "statefulset.kubernetes.io/pod-name" \-o wide --all-namespaces

Foreachresult,deletethepodbyrunningthefollowingcommand:

$ kubectl delete pod POD-NAME -n POD-NAMESPACE

Youdonotneedtomanuallyrecreatethedeletedpods.KubernetesdetectsaStatefulSetwithmissingpodsandautomaticallyrecreatesthepods.

[KubernetesBug]UpgradingaClusterAffectsPersistentWorkloadUptime

Duringanupgradetov1.0.2onvSphere,persistentstoragevolumesdonotreattachtopodsuntilallworkernodeshavebeenupgraded,whichresultsin


workloaddowntimeuntiltheentireclusterisupgraded.

Thisissueoccurswhenyoudeployapodwithpersistentstorageattached,drainthenode,andthenimmediatelydeletethenodeVM.

TheexpectedbehaviorisforpersistentdiskstoreattachtotheupgradedVMsafterthepodisrestored.However,aKubernetesbugpreventsthediskfromreattaching.PKSv1.0.2worksaroundthisbugbyattachingthevolumesafterallworkersareupgraded.

Formoreinformation,seetheKubernetesissueonGitHub .

Inrarecases,podswithpersistentvolumescanstayin ContainerCreating state.Ifyouseetheerror FailedMountUnabletomountvolumesforpodPOD-NAME

,

performthefollowingsteps:

1. Findtheproblemnodebyrunning kubectl describe pod POD-NAME .

2. Preventschedulingonthenodethatrunsthepodbyrunning kubectl cordon NODE-NAME .

3. Deletepodbyrunning kubectl delete pod POD-NAME .

4. Waitforpodtoberescheduledandenter Running state.Thismaytakeseveralminutes.

5. Resumeschedulingonthenodethatrunsthepodbyrunning kubectl uncordon NODE-NAME .



v1.0.0ReleaseDate:February8,2018

FeaturesCreate,resize,delete,list,andshowclustersthroughthePKSCLI

NativesupportforNSX-TandFlannel

Easilyobtainkubeconfigstouseeachcluster

UsekubectltoviewtheKubernetesdashboard

Defineplansthatpre-configureVMsize,authentication,defaultnumberofworkers,andaddonswhencreatingKubernetesclusters

User/AdminconfigurationsforaccesstoPKSAPI

Centralizedloggingthroughsyslog




2.0.0-2.0.5 SeparatedownloadavailablefromPivotalNetwork

vSphere







https://github.com/kubernetes/kubernetes/issues/61707

Stemcell 3468.21 SeparatedownloadavailablefromPivotalNetwork

Kubernetes 1.9.2 PackagedinthePKSTile(CFCR)


NCP 2.1.0.1 PackagedinthePKSTile

KubernetesCLI 1.9.2SeparatedownloadavailablefromthePKSsectionofPivotalNetwork

PKSCLI 1.0.0-build.3SeparatedownloadavailablefromthePKSsectionofPivotalNetwork





SpecialCharacters

InPKSv1.0.0,specialcharacters,suchas # , & , ; , " , ' , ^ , \ ,space( ), ! ,and % cannotbeusedinvCenterpasswords.Toresolvethisissue,resetyourpasswordsothatitdoesnotincludeanyofthespecialcharacterslistedabove.AfterresettingyourpasswordinvCenter,reconfigureyourcredentialsinthePKStilewiththeupdatedpassword.PKSv1.0.2addssupportforthespecialcharacterslistedabove.

StemcellIncompatibilitywithNSX-T

WhendeployingPKSv1.0.0usingNSX-Tasthenetworkinglayerwithastemcellotherthan3468.21,Kubernetesclusterdeploymentsfail.PKSv1.0.2addssupportforstemcellsv3468.25andlater.




UpdatingthePKStilewithanewstemcelltriggerstherollingofeachVMineachcluster.

UpdatingothertilesinyourdeploymentwithnewstemcellscausestherollingofthePKStile.




SyslogSecurityRecommendations

BOSHDirectorlogscontainsensitiveinformationthatshouldbeconsideredprivileged.Forexample,theselogsmaycontaincloudprovidercredentialsinPKSv1.0.0.Ifyouchoosetoforwardlogstoanexternalsyslogendpoint,usingTLSencryptionisstronglyrecommendedtopreventinformationfrombeinginterceptedbyathirdparty.


[email protected].



PKSConceptsPagelastupdated:

ThistopicdescribesPivotalContainerService(PKS)concepts.Seethefollowingsections:

PKSClusterManagement

PKSAPIAuthentication

LoadBalancersinPKS

[email protected].



PKSClusterManagementThistopicdescribeshowPivotalContainerService(PKS)managesthedeploymentofKubernetesclusters.

OverviewUsersinteractwithPKSandPKS-deployedKubernetesclustersintwoways:

DeployingKubernetesclusterswithBOSHandmanagingtheirlifecycle.ThesetasksareperformedusingthePKScommandlineinterface(CLI)andthePKScontrolplane.

Deployingandmanagingcontainer-basedworkloadsonKubernetesclusters.ThesetasksareperformedusingtheKubernetesCLI, kubectl .

ClusterLifecycleManagementThePKScontrolplaneenablesuserstodeployandmanageKubernetesclusters.

ForcommunicatingwiththePKScontrolplane,PKSprovidesacommandlineinterface,thePKSCLI.SeeInstallingthePKSCLIforinstallationinstructions.

PKSControlPlaneOverviewThePKScontrolplanemanagesthelifecycleofKubernetesclustersdeployedusingPKS.ThecontrolplaneallowsuserstodothefollowingthroughthePKSCLI:

Viewclusterplans

Createclusters

Viewinformationaboutclusters

Obtaincredentialstodeployworkloadstoclusters

Scaleclusters

Deleteclusters

Inaddition,thePKScontrolplanecanupgradeallexistingclustersusingtheUpgradeallclustersBOSHerrand.Formoreinformation,seeUpgradeKubernetesClustersinUpgradePKS.

PKSControlPlaneArchitectureThePKScontrolplaneisdeployedonasingleVMthatincludesthefollowingcomponents:

ThePKSAPIserver

ThePKSBroker

AUserAccountandAuthentication(UAA)server

Formoreinformationabouthowthesecomponentsinteract,seethefollowingdiagram:


UAA

WhenauserlogsintoorlogsoutofthePKSAPIthroughthePKSCLI,thePKSCLIcommunicateswithUAAtoauthenticatethem.ThePKSAPIpermitsonlyauthenticateduserstomanageKubernetesclusters.Formoreinformationaboutauthenticating,seePKSAPIAuthentication.

UAAmustbeconfiguredwiththeappropriateusersanduserpermissions.Formoreinformation,seeManageUsersinUAA.

PKSAPI

ThroughthePKSCLI,usersinstructthePKSAPIservertodeploy,scaleup,anddeleteKubernetesclustersaswellasshowclusterdetailsandplans.ThePKSAPIcanalsowriteKubernetesclustercredentialstoalocalkubeconfigfile,whichenablesuserstoconnecttoaclusterthrough kubectl .

ThePKSAPIsendsallclustermanagementrequests,exceptread-onlyrequests,tothePKSBroker.

PKSBroker

WhenthePKSAPIreceivesarequesttomodifyaKubernetescluster,itinstructsthePKSBrokertomaketherequestedchange.

ThePKSBrokerconsistsofanOn-DemandServiceBroker andaServiceAdapter.ThePKSBrokergeneratesaBOSHmanifestandinstructstheBOSHDirectortodeployordeletetheKubernetescluster.

ClusterWorkloadManagementPKSusersmanagetheircontainer-basedworkloadsonKubernetesclustersthrough kubectl .

Formoreinformationabout kubectl ,seeOverviewofkubectl intheKubernetesdocumentation.


https://docs.pivotal.io/svc-sdk/odb/index.htmlhttps://kubernetes.io/docs/reference/kubectl/overview/

[email protected].



PKSAPIAuthenticationPagelastupdated:

ThistopicdescribeshowthePivotalContainerService(PKS)APIworkswithUserAccountandAuthentication(UAA)tomanageauthenticationandauthorizationinyourPKSdeployment.

AuthenticatingPKSAPIRequestsBeforeuserscanloginandusethePKSCLI,youmustconfigurePKSAPIaccesswithUAA.YouusetheUAACommandLineInterface(UAAC)totargettheUAAserverandrequestanaccesstokenfortheUAAadminuser.Ifyourrequestissuccessful,theUAAserverreturnstheaccesstoken.TheUAAadminaccesstokenauthorizesyoutomakerequeststothePKSAPIusingthePKSCLIandgrantclusteraccesstoneworexistingusers.

WhenauserwithclusteraccesslogsintothePKSCLI,theCLIrequestsanaccesstokenfortheuserfromtheUAAserver.Iftherequestissuccessful,theUAAserverreturnsanaccesstokentothePKSCLI.WhentheuserrunsPKSCLIcommands,forexample, pksclusters ,theCLIsendstherequesttothePKSAPIserverandincludestheuser’sUAAtoken.

ThePKSAPIsendsarequesttotheUAAservertovalidatetheuser’stoken.IftheUAAserverconfirmsthatthetokenisvalid,thePKSAPIusestheclusterinformationfromthePKSbrokertorespondtotherequest.Forexample,iftheuserruns pksclusters ,theCLIreturnsalistoftheclustersthattheuserisauthorizedtomanage.

RoutingtothePKSAPIControlPlaneVMThePKSAPIserverandtheUAAserverusedifferentportnumbersonthecontrolplaneVM.Forexample,ifyourPKSAPIdomainis api.pks.example.com ,youcanreachyourPKSAPIandUAAserversatthefollowingURLs:

Server URL

PKSAPI api.pks.example.com:9021

UAA api.pks.example.com:8443

RefertoOpsManager>PivotalContainerService>UAA>UAAURLforyourPKSAPIdomain.

WhenyouinstallthePKStile,youconfigurealoadbalancerforthePKSAPI.ThisloadbalancerallowsyoutorunPKSCLIcommandsfromyourlocalworkstation.Formoreinformation,seetheConfigureExternalLoadBalancersectionofInstallingandConfiguringPKS.

[email protected].



LoadBalancersinPKSPagelastupdated:

ThistopicdescribesthetypesofloadbalancersthatareusedinPivotalContainerService(PKS).

Youcanconfigureloadbalancersforthefollowing:

PKSAPI:ConfiguringthisloadbalancerallowsyoutorunPKSCommandLineInterface(CLI)commandsfromyourlocalworkstation.

KubernetesClusters:ConfiguringaloadbalancerforeachnewclusterallowsyoutorunKubernetesCLI(kubectl)commandsonthecluster.

Workloads:Configuringaloadbalancerforyourapplicationworkloadsallowsexternalaccesstotheservicesthatrunonyourcluster.

ThefollowingdiagramshowswhereeachoftheaboveloadbalancerscanbeusedwithinyourPKSdeployment:

IfyouuseeithervSpherewithNSX-TorGCP,youcancreateloadbalancerswithinyourcloudproviderconsole.

Ifyourcloudproviderdoesnotofferloadbalancing,youcanuseanyexternalTCPorHTTPSloadbalancerofyourchoice.

AboutthePKSAPILoadBalancerTheloadbalancerforthePKSAPIallowsyoutoaccessthePKSAPIfromoutsidethenetwork.Forexample,configuringaloadbalancerforthePKSAPIallowsyoutorunPKSCLIcommandsfromyourlocalworkstation.

ForinformationaboutconfiguringthePKSAPIloadbalancer,seetheConfigureExternalLoadBalancersectionofInstallingandConfiguringPKS.


AboutKubernetesClusterLoadBalancersWhenyoucreateacluster,youmustconfigureexternalaccesstotheclusterbycreatinganexternalTCPorHTTPSloadbalancer.TheloadbalancerallowstheKubernetesCLItocommunicatewiththecluster.

Ifyoucreateaclusterinanon-productionenvironment,youcanchoosenottousealoadbalancer.Toallowkubectltoaccesstheclusterwithoutaloadbalancer,youcandooneofthefollowing:

CreateaDNSentrythatpointstothecluster’smasterVM.Forexample:

my-cluster.example.com A 10.0.0.5

Ontheworkstationwhereyourunkubectlcommands,addthemasterIPaddressofyourclusterand kubo.internal tothe /etc/hosts file.Forexample:

10.0.0.5 kubo.internal

Forinformationaboutconfiguringaclusterloadbalancer,seeCreateaCluster.

AboutWorkloadLoadBalancersToallowexternalaccesstoyourapp,youcaneithercreatealoadbalancerorexposeastaticportonyourworkload.

Forinformationaboutconfiguringaloadbalancerforyourappworkload,seeDeployandAccessBasicWorkloads.

[email protected].



PKSPrerequisitesPagelastupdated:

ThistopicdescribestheprerequisitesforinstallingPivotalContainerService(PKS)onvSphereorGoogleCloudPlatform(GCP).

GeneralPKSPrerequisitesPKSrequiresthePKSCommandLineInterface(PKSCLI)andtheKubernetesCLI(kubectl).SeethefollowingtopicsforinformationaboutinstallingeachCLI:

InstallingthePKSCLI

InstallingtheKubernetesCLI

ResourceRequirementsForinformationabouttheresourcerequirementsforinstallingPKS,seethetopicthatcorrespondstoyourcloudprovider:

vSpherePrerequisitesandResourceRequirements

GCPPrerequisitesandResourceRequirements

[email protected].



InstallingthePKSCLIPagelastupdated:

ThistopicdescribeshowtoinstallthePivotalContainerServiceCommandLineInterface(PKSCLI).

ToinstallthePKSCLI,followtheproceduresforyouroperatingsystemtodownloadthePKSCLIfromPivotalNetwork .Binariesareonlyprovidedfor64-bitarchitectures.

MacOSX1. NavigatetoPivotalNetwork andlogin.

2. ClickPivotalContainerService(PKS).

3. ClickPKSCLI.

4. ClickPKSCLI-MactodownloadtheMacOSXbinary.

5. Renamethedownloadedbinaryto pks .

6. Onthecommandline,runthefollowingcommandtomakethePKSbinaryexecutable:

$chmod+xpks

7. Movethebinaryintoyour PATH .

Forexample:

$mvpks/usr/local/bin/pks

Linux1. NavigatetoPivotalNetwork andlogin.


3. ClickPKSCLI.

4. ClickPKSCLI-LinuxtodownloadtheLinuxbinary.

5. Renamethedownloadedbinaryto pks .

6. Onthecommandline,runthefollowingcommandtomakethePKSbinaryexecutable:

$chmod+xpks


Forexample:

$mvpks/usr/local/bin/pks

Windows1. NavigatetoPivotalNetwork andlogin.



https://network.pivotal.io/products/pivotal-container-servicehttps://network.pivotal.io/https://network.pivotal.io/https://network.pivotal.io/

3. ClickPKSCLI.

4. ClickPKSCLI-WindowstodownloadtheWindowsexecutablefile.

5. Renamethedownloadedbinaryto pks.exe .


LogintoPKSCLIOnthecommandline,runthefollowingcommandtologintothePKSCLI:

pks login -a PKS_API -u USERNAME -p PASSWORD --ca-cert CERT-PATH

Replacetheplaceholdervaluesinthecommandasfollows:

PKS_API isthedomainnameyouenteredinOpsManager>PivotalContainerService>UAA>UAAURL.Forexample, api.pks.example.com .

USERNAME and PASSWORD belongtotheaccountyoucreatedintheGrantClusterAccesstoaUserstepinManageUsersinUAA.

CERT-PATH isthepathtoyourrootCAcertificate.ProvidethecertificatetovalidatethePKSAPIcertificatewithSSL.

Forexample:

$pkslogin-aapi.pks.example.com-ualana\--ca-cert/var/tempest/workspaces/default/root_ca_certificate

Ifyouareloggingintoatrustedenvironment,youcanuse -k toskipSSLverificationinsteadof --ca-certCERT-PATH .

Forexample:

$pkslogin-aapi.pks.example.com-ualana-k

Uponsuccessfullogin,thePKSCLIgeneratesa creds.yml filecontainingtheAPIendpoint,CAcertificate(ifapplicable),refreshtoken,andaccesstoken.

Bydefault, creds.yml issavedinthe ~/.pks directory.Youcanusethe PKS_HOME environmentvariabletooverridethislocationanduse creds.yml fromanydirectory.

[email protected].



InstallingtheKubernetesCLIPagelastupdated:

ThistopicdescribeshowtoinstalltheKubernetesCommandLineInterface(kubectl).

Toinstallkubectl,followtheproceduresforyouroperatingsystemtodownloadkubectlfromPivotalNetwork .Binariesareonlyprovidedfor64-bitarchitectures.

MacOSX1. NavigatetoPivotalNetwork andlogin.


3. ClickKubectlCLIs.

4. ClickkubectlCLI-Mactodownloadthekubectlbinary.

5. Renamethedownloadedbinaryto kubectl .

6. Onthecommandline,runthefollowingcommandtomakethekubectlbinaryexecutable:

$chmod+xkubectl

7. Movethebinaryintoyour PATH .Forexample:

$mvkubectl/usr/local/bin/kubectl

Linux1. NavigatetoPivotalNetwork andlogin.



4. ClickkubectlCLI-Linuxtodownloadthekubectlbinary.

5. Renamethedownloadedbinaryto kubectl .

6. Onthecommandline,runthefollowingcommandtomakethekubectlbinaryexecutable:

$chmod+xkubectl

7. Movethebinaryintoyour PATH .Forexample:

$mvkubectl/usr/local/bin/kubectl

Windows1. NavigatetoPivotalNetwork andlogin.



4. ClickkubectlCLI-Windowstodownloadthekubectlexecutablefile.


https://network.pivotal.io/products/pivotal-container-servicehttps://network.pivotal.io/https://network.pivotal.io/https://network.pivotal.io/

5. Renamethedownloadedbinaryto kubectl.exe .


[email protected].



PreparingtoInstallPKSonvSphereThistopicoutlinesthestepsforpreparingtoinstallPivotalContainerService(PKS)onvSphere.Seethefollowingsections:

vSpherePrerequisitesandResourceRequirements

FirewallPortsandProtocolsRequirementsforvSpherewithNSX-T

PreparingtoDeployPKStovSphere

DeployingOpsManagertovSphere

ConfiguringOpsManageronvSphere

InstallingandIntegratingVMwareHarborRegistrywithPKS

[email protected].


https://docs.pivotal.io/partners/vmware-harbor/index.htmlmailto:[email protected]

vSpherePrerequisitesandResourceRequirementsPagelastupdated:

ThistopicdescribestheprerequisitesandresourcerequirementsforinstallingPivotalContainerService(PKS)onvSpherewithorwithoutNSX-Tintegration.

PKSsupportsair-gappeddeploymentsonvSpherewithorwithoutNSX-Tintegration.

YoucanalsoconfigureintegrationwiththeHarbortile,anenterprise-classregistryserverforcontainerimages.Formoreinformation,seetheVMwareHarborRegistry documentation.

ComponentVersionRequirements

vSphereVersionRequirementsPKSonvSpheresupportsthefollowingvSpherecomponentversions:

Versions Editions

VMwarevSphere6.5GA

VMwarevSphere6.5U1

vSphereEnterprisePlus


NSX-TIntegrationVersionRequirementsDeployingNSX-Trequirestheadditionalfollowingcomponentversions:

Component Version

VMwareNSX-T 2.1

ResourceRequirementsInstallingPKSdeploysthefollowingtwovirtualmachines(VMs):

VM CPU RAM Storage

PivotalContainerService 1 4GB 20GB

PivotalOpsManager 1 8GB 160GB

EachKubernetesclusterprovisionedthroughPKSdeploystheVMslistedbelow.IfyoudeploymorethanoneKubernetescluster,youmustscaleyourallocatedresourcesappropriately.

VMName Number CPUCores RAM EphemeralDisk PersistentDisk

master 1 2 4GB 8GB 5GB

worker 1 2 4GB 8GB 10GB

NSX-TIntegrationResourceRequirementsDeployingNSX-TrequirestheadditionalfollowingresourcesfromyourvSphereenvironment:

NSX-TComponent InstanceCount MemoryperInstance vCPUperInstance DiskSpaceperInstance

NSXManagerAppliance 1 16GB 4 140GB

NSXControllers 3 16GB 4 120GB

NSX-TEdge 1upto8 16GB 8 120GB


https://docs.pivotal.io/partners/vmware-harbor/index.html

InstallingPKSonvSpherewithNSX-TForinformationaboutthefirewallportsandprotocolsrequirementsforusingPKSonvSpherewithNSX-T,seeFirewallPortsandProtocolsRequirementsforvSpherewithNSX-T.

ToinstallandconfigurePKSwithNSX-Tintegration,followtheproceduresbelow:

1. InstallingandConfiguringPKSwithNSX-TIntegration

2. (Optional)InstallingandIntegratingVMwareHarborRegistrywithPKS

InstallingPKSonvSpherewithoutNSX-TToinstallPKSonvSpherewithoutNSX-Tintegration,followtheproceduresbelow:

1. PreparingtoDeployPKStovSphere

2. DeployingOpsManagertovSphere

3. ConfiguringOpsManageronvSphere

4. InstallingandConfiguringPKS

5. (Optional)InstallingandIntegratingVMwareHarborRegistrywithPKS

AboutDeployingPASandPKSThePivotalApplicationService(PAS)andPKSruntimeplatformsarebothdeployedbyOpsManagerusingBOSH.YoucandeploybothPASandPKSusingthesameOpsManagerinstanceinadevelopmentortestenvironment,butwerecommendthatyoudeployproductioninstallationsofPASandPKStoseparateOpsManagerinstances.Forincreasedsecurity,werecommenddeployingeachOpsManagerinstanceusingauniquecloudprovideraccount.

SeparateinstallationsofOpsManagerallowyoutocustomizeandtroubleshootruntimetilesindependently.YoumaychoosetoconfigureOpsManagerwithdifferentsettingsforyourPASandPKSdeployments.Forexample,PKSandmanyPASfeaturesdependonBOSHDNS.

IfyoudeployPAStoaseparateOpsManagerinstance,youcandisableBOSHDNSfortroubleshootingpurposes.PAScanrunwithoutBOSHDNS,butkeyfeaturessuchassecureservicecredentialswithCredHub,servicediscoveryforcontainer-to-containernetworking,andNSX-TintegrationdonotworkwhenBOSHDNSisdisabled.

IfyoudeployPASandPKStothesameOpsManagerinstance,youcannotdisableBOSHDNSwithoutbreakingyourPKSinstallationalongwiththePASfeaturesthatdependonBOSHDNS.

[email protected].


https://docs.pivotal.io/partners/vmware-harbor/https://docs.pivotal.io/partners/vmware-harbor/mailto:[email protected]

FirewallPortsandProtocolsRequirementsforvSpherewithNSX-TPagelastupdated:

ThistopicdescribesthefirewallportsandprotocolsrequirementsforusingPivotalContainerService(PKS)onvSpherewithNSX-Tintegration.

Inenvironmentswithstrictinter-networkaccesscontrolpolicies,firewallsoftenrequireconduitstopasscommunicationbetweensystemcomponentsonadifferentnetworkorallowinterfacingwithexternalsystemssuchaswithenterpriseapplicationsorthepublicInternet.

ForPKS,therecommendationistodisablesecuritypoliciesthatfiltertrafficbetweenthenetworkssupportingthesystem.Whenthatisnotanoption,refertothefollowingtable,whichidentifiestheflowsbetweensystemcomponentsinatypicalPKSdeployment.

SourceComponent DestinationComponent DestinationProtocol DestinationPort Service

ApplicationUser K8sClusterWorkerNodes TCP 30000-32767 k8snodeport

ApplicationUser K8sLoad-Balancers TCP/UDP varies varies

ApplicationUser K8sIngress-Controllers TCP/UDP varies varies

CloudFoundryBOSHDirector DomainNameServer UDP 53 dns

CloudFoundryBOSHDirector vCenterServer TCP 443 https

CloudFoundryBOSHDirector vSphereESXIMgmt.vmknic TCP 443 https

CompilationJobVMs DomainNameServer UDP 53 dns

Developer HarborPrivateImageRegistry TCP 4443 notary

Developer HarborPrivateImageRegistry TCP 443 https

Developer HarborPrivateImageRegistry TCP 80 http

Developer K8sClusterMaster/EtcdNodes TCP 8443 uaaauth

Developer K8sClusterWorkerNodes TCP 30000-32767 k8snodeport

Developer K8sLoad-Balancers TCP/UDP varies varies

Developer K8sIngress-Controllers TCP/UDP varies varies

DomainNameServer vCenterServer UDP 1433 ms-sql-server

HarborPrivateImageRegistry DomainNameServer UDP 53 dns

HarborPrivateImageRegistry PublicCVESourceDatabase TCP 443 https

HarborPrivateImageRegistry PublicCVESourceDatabase TCP 80 http

K8sClusterMaster/EtcdNodes CloudFoundryBOSHDirector TCP 4222 boshnatsserver

K8sClusterMaster/EtcdNodes CloudFoundryBOSHDirector TCP 25250 boshblobstore

K8sClusterMaster/EtcdNodes DomainNameServer UDP 53 dns

K8sClusterMaster/EtcdNodes NSXManagerServer TCP 443 https

K8sClusterMaster/EtcdNodes vCenterServer TCP 443 https

K8sClusterWorkerNodes CloudFoundryBOSHDirector TCP 4222 boshnatsserver

K8sClusterWorkerNodes CloudFoundryBOSHDirector TCP 25250 boshblobstore

K8sClusterWorkerNodes DomainNameServer UDP 53 dns

K8sClusterWorkerNodes HarborPrivateImageRegistry TCP 8853 boshdnshealth

K8sClusterWorkerNodes HarborPrivateImageRegistry TCP 443 https

K8sClusterWorkerNodes NSXManagerServer TCP 443 https

K8sClusterWorkerNodes vCenterServer TCP 443 https

NSXControllers NetworkTimeServer UDP 123 ntp

NSXEdgeManagement NSXEdgeTEPvNIC UDP 3784 bfd

NSXManagerServer DomainNameServer UDP 53 dns

NSXManagerServer SFTPBackupServer TCP 22 ssh

Operator HarborPrivateImageRegistry TCP 443 https

Operator HarborPrivateImageRegistry TCP 80 http

Operator K8sLoad-Balancers TCP 80 http

Operator NSXManagerServer TCP 443 https


Operator NSXManagerServer TCP 443 https

Operator PCFOperationsManager TCP 22 ssh

Operator PCFOperationsManager TCP 443 https

Operator PCFOperationsManager TCP 80 http

Operator PKSController TCP 8443 uaaauth

Operator PKSController TCP 9021 pksapiserver

Operator vCenterServer TCP 443 https

Operator vCenterServer TCP 80 http

Operator vSphereESXIMgmt.vmknic TCP 22 ssh

PCFOperationsManager DomainNameServer UDP 53 dns

PCFOperationsManager K8sClusterWorkerNodes TCP 22 ssh

PCFOperationsManager NetworkTimeServer UDP 123 ntp

PCFOperationsManager vCenterServer TCP 443 https

PCFOperationsManager vSphereESXIMgmt.vmknic TCP 443 https

PKSController DomainNameServer UDP 53 dns

PKSController K8sClusterMaster/EtcdNodes TCP 8443 uaaauth

PKSController NSXManagerServer TCP 443 https

PKSController vCenterServer TCP 443 https

vCenterServer DomainNameServer UDP 53 dns

vCenterServer NetworkTimeServer UDP 123 ntp

vCenterServer vSphereESXIMgmt.vmknic TCP 8080 vsanvp

vCenterServer vSphereESXIMgmt.vmknic TCP 9080 iofilterstorage

vCenterServer vSphereESXIMgmt.vmknic TCP 443 https

vCenterServer vSphereESXIMgmt.vmknic TCP 902 ideafarm-door

SourceComponent DestinationComponent DestinationProtocol DestinationPort Service

[email protected].

Note:Youhavetheoptiontoexposecontainerizedapplications,runninginaKubernetescluster,forexternalconsumptionthroughvariousportsandmethods.YoucanenableexternalaccesstoapplicationsbywayofKubernetesNodePorts,load-balancers,andingress.EnablingaccesstoapplicationsviaKubernetesload-balancersandingresscontrollertypesallowforspecificportandprotocoldesignations,whileNodePortofferstheleastcontrolanddynamicallyallocatesportsfromapre-definedrangeofports.



PreparingtoDeployPKSonvSpherePagelastupdated:

BeforeyouinstallPivotalContainerService(PKS)onvSpherewithoutNSX-Tintegration,youmustprepareyourvSphereenvironment.InadditiontofulfillingtheprerequisitesspecifiedinvSpherePrerequisitesandResourceRequirements,youmustcreatethefollowingtwoserviceaccountsinvSphere:

MasterNodeServiceAccount:YoumustcreateaserviceaccountforKubernetesclustermasterVMs.

BOSH/OpsManagerServiceAccount:YoumustcreateaserviceaccountforBOSHandOpsManager.

Afteryoucreatetheserviceaccountslistedabove,youmustgrantthemprivilegesinvSphere.Pivotalrecommendsconfiguringeachserviceaccountwiththeleastpermissiveprivilegesanduniquecredentials.

Forthemasternodeserviceaccount,youcancreateacustomroleinvSpherebasedonyourstorageconfiguration.KubernetesmasternodeVMsrequirestoragepermissionstocreateloadbalancersandattachpersistentdiskstopods.CreatingacustomroleallowsvSpheretoapplythesameprivilegestoallKubernetesmasternodeVMsinyourPKSinstallation.

WhenyouconfiguretheKubernetesCloudProviderpaneofthePKStile,youenterthemasternodeserviceaccountcredentialsinthevSphereMasterCredentialsfields.Formoreinformation,seetheKubernetesCloudProvidersectionofInstallingandConfiguringPKS.

FortheBOSH/OpsManagerserviceaccount,youcanapplyprivilegesdirectlytotheserviceaccountwithoutcreatingarole.YoucanalsoapplythedefaultVMwareAdministratorSystemRole totheserviceaccounttoachievetheappropriatepermissionlevel.

Step1:CreatetheMasterNodeServiceAccount1. FromthevCenterconsole,createaserviceaccountforKubernetesclustermasterVMs.

2. GrantthefollowingVirtualMachineObjectpermissionstotheserviceaccount:

Privilege(UI) Privilege(API)

Advanced VirtualMachine.Configuration.Advanced

Settings VirtualMachine.Configuration.Settings

Step2:GrantAdditionalStoragePermissionsKubernetesmasternodeVMserviceaccountsrequirethefollowing:

Readaccesstothefolder,host,anddatacenteroftheclusternodeVMs

PermissiontocreateanddeleteVMswithintheresourcepoolwherePKSisdeployed

Grantthesepermissionstothemasternodeserviceaccountbasedonyourstorageconfigurationusingoneoftheproceduresbelow:

StaticOnlyPersistentVolumeProvisioning

DynamicPersistentVolumeProvisioning(withStoragePolicy-BasedVolumePlacement)

DynamicPersistentVolumeProvisioning(withoutStoragePolicy-BasedVolumePlacement)

SeevSphereStorageforKubernetes intheVMwaredocumentationformoreinformation.

StoragePermissionsforServiceAccountsThefollowingtablesdescribetheminimumpermissionsrequiredbythemasternodeserviceaccountbasedonyourstorageconfiguration.

StaticOnlyPersistentVolumeProvisioning

Roles Privileges Entities PropagatetoChildren

Note:IfyourKubernetesclustersspanmultiplevCenters,youmustsettheserviceaccountprivilegescorrectlyineachvCenter.


http://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.wssdk.pg.doc/PG_Authenticate_Authorize.8.6.html#1110514https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/index.html

manage-k8s-node-vms

VirtualMachine.Config.AddExistingDisk

VirtualMachine.Config.AddNewDisk

VirtualMachine.Config.AddRemoveDevice

VirtualMachine.Config.RemoveDisk

VMFolder Yes

manage-k8s-volumes Datastore.FileManagement(Lowlevelfileoperations) Datastore No

Read-only(pre-existingdefaultrole)

System.Anonymous

System.Read

System.View

vCenter

Datacenter

DatastoreCluster

DatastoreStorageFolder

No

DynamicPersistentVolumeProvisioning(withStoragePolicy-BasedVolumePlacement)


manage-k8s-node-vms

Resource.AssignVMToPool





VirtualMachine.Inventory.Create

VirtualMachine.Inventory.Delete

Cluster

Hosts

VMFolderYes

manage-k8s-volumes

Datastore.AllocateSpace

Datastore.FileManagement(Lowlevelfileoperations)

Datastore No

k8s-system-read-and-spbm-profile-view

StorageProfile.View(Profile-drivenstorageview) vCenter No

Read-only(pre-existingdefaultrole)

System.Anonymous

System.Read

System.View

Datacenter

DatastoreCluster


No

DynamicVolumeProvisioning(withoutStoragePolicy-BasedVolumePlacement)


manage-k8s-node-vms





VMFolder Yes

manage-k8s-volumes

Datastore.AllocateSpace

Datastore.FileManagement(Lowlevelfileoperations)

Datastore No

System.AnonymousvCenter

Datacenter

Note:Datastore.FileManagementisonlyrequiredfortherole manage-k8s-volumes ifaPersistentVolumeClaim(PVC)iscreatedtobindwithastaticallyprovisionedPersistentVolume(PV),andthereclaimpolicysettodelete.WhenthePVCisdeleted,thestaticallyprovisionedPVisalsodeleted.


Read-only(pre-existingdefaultrole) System.Read

System.View

DatastoreCluster


No

Step3:CreatetheBOSH/OpsManagerServiceAccount1. FromthevCenterconsole,createaserviceaccountforBOSHandOpsManager.

2. GrantthepermissionsbelowtotheBOSHandOpsManagerserviceaccount.

vCenterRootPrivilegesGrantthefollowingprivilegesontherootvCenterserverentitytotheserviceaccount:


Read-only System.Anonymous

System.Read

System.View

Managecustomattributes Global.ManageCustomFields

vCenterDatacenterPrivilegesGrantthefollowingprivilegesonanyentitiesinadatacenterwhereyoudeployPKS:

RoleObject


UsersinherittheRead-OnlyrolefromthevCenterrootlevel System.Anonymous

System.Read

System.View

DatastoreObject

Grantthefollowingprivilegesmustatthedatacenterleveltouploadanddeletevirtualmachinefiles:


Allocatespace Datastore.AllocateSpace

Browsedatastore Datastore.Browse

Lowlevelfileoperations Datastore.FileManagement

Removefile Datastore.DeleteFile

Updatevirtualmachinefiles Datastore.UpdateVirtualMachineFiles

FolderObject


Deletefolder Folder.Delete

Note:TheprivilegeslistedinthissectiondescribetheminimumrequiredpermissionstodeployBOSH.YoucanalsoapplythedefaultVMwareAdministratorSystemRole totheserviceaccounttoachievetheappropriatepermissionlevel,butthedefaultroleincludesmoreprivilegesthanthoselistedbelow.


http://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.wssdk.pg.doc/PG_Authenticate_Authorize.8.6.html#1110514

Createfolder Folder.CreateMovefolder Folder.Move

Renamefolder Folder.Rename

GlobalObject


Setcustomattribute Global.SetCustomField

HostObject


Modifycluster Host.Inventory.EditCluster

InventoryServiceObject


vSphereTagging>CreatevSphereTag InventoryService.Tagging.CreateTag

vSphereTagging>DeletevSphereTag InventoryService.Tagging.EditTag

vSphereTagging>EditvSphereTag InventoryService.Tagging.DeleteTag

NetworkObject


Assignnetwork Network.Assign

ResourceObject


Assignvirtualmachinetoresourcepool Resource.AssignVMToPool

Migratepoweredoffvirtualmachine Resource.ColdMigrate

Migratepoweredonvirtualmachine Resource.HotMigrate

vAppObject

Granttheseprivilegesattheresourcepoollevel.


Import VApp.Import

vAppapplicationconfiguration VApp.ApplicationConfig

VirtualMachineObject

Configuration


Addexistingdisk VirtualMachine.Config.AddExistingDisk

Addnewdisk VirtualMachine.Config.AddNewDisk

Addorremovedevice VirtualMachine.Config.AddRemoveDevice


Advanced VirtualMachine.Config.AdvancedConfigChangeCPUcount VirtualMachine.Config.CPUCount

Changeresource VirtualMachine.Config.Resource

ConfiguremanagedBy VirtualMachine.Config.ManagedBy

Diskchangetracking VirtualMachine.Config.ChangeTracking

Disklease VirtualMachine.Config.DiskLease

Displayconnectionsettings VirtualMachine.Config.MksControl

Extendvirtualdisk VirtualMachine.Config.DiskExtend

Memory VirtualMachine.Config.Memory

Modifydevicesettings VirtualMachine.Config.EditDevice

Rawdevice VirtualMachine.Config.RawDevice

Reloadfrompath VirtualMachine.Config.ReloadFromPath

Removedisk VirtualMachine.Config.RemoveDisk

Rename VirtualMachine.Config.Rename

Resetguestinformation VirtualMachine.Config.ResetGuestInfo

Setannotation VirtualMachine.Config.Annotation

Settings VirtualMachine.Config.Settings

Swapfileplacement VirtualMachine.Config.SwapPlacement

Unlockvirtualmachine VirtualMachine.Config.Unlock

GuestOperations


GuestOperationProgramExecution VirtualMachine.GuestOperations.Execute

GuestOperationModifications VirtualMachine.GuestOperations.Modify

GuestOperationQueries VirtualMachine.GuestOperations.Query

Interaction


Answerquestion VirtualMachine.Interact.AnswerQuestion

ConfigureCDmedia VirtualMachine.Interact.SetCDMedia

Consoleinteraction VirtualMachine.Interact.ConsoleInteract

Defragmentalldisks VirtualMachine.Interact.DefragmentAllDisks

Deviceconnection VirtualMachine.Interact.DeviceConnection

GuestoperatingsystemmanagementbyVIXAPI VirtualMachine.Interact.GuestControl

Poweroff VirtualMachine.Interact.PowerOff

Poweron VirtualMachine.Interact.PowerOn

Reset VirtualMachine.Interact.Reset

Suspend VirtualMachine.Interact.Suspend

VMwareToolsinstall VirtualMachine.Interact.ToolsInstall

Inventory


Createfromexisting VirtualMachine.Inventory.CreateFromExisting

Createnew VirtualMachine.Inventory.Create

Move VirtualMachine.Inventory.Move

Register VirtualMachine.Inventory.Register

Remove VirtualMachine.Inventory.Delete

Unregister VirtualMachine.Inventory.Unregister


Provisioning


Allowdiskaccess VirtualMachine.Provisioning.DiskRandomAccess

Allowread-onlydiskaccess VirtualMachine.Provisioning.DiskRandomRead

Allowvirtualmachinedownload VirtualMachine.Provisioning.GetVmFiles

Allowvirtualmachinefilesupload VirtualMachine.Provisioning.PutVmFiles

Clonetemplate VirtualMachine.Provisioning.CloneTemplate

Clonevirtualmachine VirtualMachine.Provisioning.Clone

Customize VirtualMachine.Provisioning.Customize

Deploytemplate VirtualMachine.Provisioning.DeployTemplate

Markastemplate VirtualMachine.Provisioning.MarkAsTemplate

Markasvirtualmachine VirtualMachine.Provisioning.MarkAsVM

Modifycustomizationspecification VirtualMachine.Provisioning.ModifyCustSpecs

Promotedisks VirtualMachine.Provisioning.PromoteDisks

Readcustomizationspecifications VirtualMachine.Provisioning.ReadCustSpecs

SnapshotManagement


Createsnapshot VirtualMachine.State.CreateSnapshot

Removesnapshot VirtualMachine.State.RemoveSnapshot

Renamesnapshot VirtualMachine.State.RenameSnapshot

Revertsnapshot VirtualMachine.State.RevertToSnapshot

NextStepsToinstallPKSonvSphere,followtheproceduresinDeployingOpsManagertovSphere.

[email protected].



DeployingOpsManagertovSpherePagelastupdated:

ThistopicprovidesinstructionsfordeployingOpsManagertoVMwarevSphere.

1. Beforestarting,refertotheknownissuesinthePCFOpsManagerReleasev2.0ReleaseNotes .

2. DownloadthePivotalCloudFoundry (PCF)OpsManager .ova fileatPivotalNetwork .ClickthePivotalCloudFoundryregiontoaccessthePCFproductpage.UsethedropdownmenutoselectanOpsManagerrelease.

3. LogintovCenter.

4. SelecttheVMandTemplatesview.

5. RightclickonyourdatacenterandselectNewFolder.

Note:WithvSphere6.5andNSX-T2.1,wheninitiallydeployingtheOperationsManagerOVF,youcannotconnectdirectlytoanNSX-Tlogicalswitch.YoumustfirstconnecttoavSphereStandard(vSS)orvSphereDistributedSwitch(vDS).AsuggestedapproachistoconnecttoaVSSorVDSwhendeployingtheOVF,butdonotpowertheVMon.AftertheOVFdeploymenthascompleted,youcanthenconnectthenetworkinterfacetotheappropriateNSX-TlogicalswitchandpowertheVMontoproceedwiththeinstall.ThisissueisresolvedinVMwarevCenterServer6.7.Formoreinformationaboutthisissue,seetheVMwareKnowledgeBase .


https://kb.vmware.com/kb/54142http://docs.pivotal.io/pivotalcf/2-0/pcf-release-notes/opsmanager-rn.htmlhttps://network.pivotal.io/products/pivotal-cfhttps://network.pivotal.io

6. Namethefolder pivotal_cf andselectit.

7. SelectFile>DeployOVFTemplate.

8. Selectthe.ovafileandclickNext.

9. ReviewtheproductdetailsandclickNext.

10. AcceptthelicenseagreementandclickNext.

11. NamethevirtualmachineandclickNext.

12. SelectavSphereclusterandclickNext.

13. Ifprompted,selectaresourcepoolandclickNext.

14. Ifprompted,selectahostandclickNext.

Note:Theselectedfolderistheoneyoucreated.

Note:IfyourvSpherehostdoesnotsupportVT-X/EPT,hardwarevirtualizationmustbe**off**.Formoreinformation,seePCFonvSphereRequirements .


https://docs.pivotal.io/pivotalcf/customizing/vsphere.html#vsphere-reqs

15. SelectastoragedestinationandclickNext.

16. SelectadiskformatandclickNext.Formoreinformationaboutdiskformats,seeProvisioningaVirtualDiskonvSphere .

17. SelectanetworkfromthedropdownlistandclickNext.

18. EnternetworkinformationandpasswordsfortheOpsManagerVMadminuser.

19. IntheAdminPasswordfield,enteradefaultpasswordfortheubuntuuser.Ifyoudonotenteradefaultpassword,yourOpsManagerwillnotbootup.

20. ClickNext.

21. CheckthePoweronafterdeploymentcheckboxandclickFinish.OncetheVMboots,theinterfaceisavailableattheIPaddressyouspecified.

Warning:OpsManagerv2.0requiresaDirectorVMwithatleast8GBmemory.

Note:Recordthisnetworkinformation.TheIPAddresswillbethelocationoftheOpsManagerinterface.

Note:ItisnormaltoexperienceabriefdelaybeforetheinterfaceisaccessiblewhilethewebserverandVMstartup.


https://docs.pivotal.io/pivotalcf/customizing/disk-format.html

22. CreateaDNSentryfortheIPaddressthatyouusedforOpsManager.YoumustusethisfullyqualifieddomainnamewhenyoulogintoOpsManagerinInstallingPivotalCloudFoundryonvSphere .

NextStepsAfteryoucompletethisprocedure,followtheinstructionsinConfiguringOpsManageronvSphere.

[email protected].

Note:OpsManagersecurityfeaturesrequireyoutocreateafullyqualifieddomainnametoaccessOpsManagerduringtheinitialconfiguration.


https://docs.pivotal.io/pivotalcf/customizing/vsphere.html#paashttps://docs.pivotal.io/pivotalcf/customizing/vsphere.html#paasmailto:[email protected]

ConfiguringOpsManageronvSpherePagelastupdated:

ThistopicdescribeshowtoconfigureOpsManagerforVMwarevSphere.

IfyouareinstallingPivotalContainerService(PKS)tovSpherewithoutNSX-Tintegration,beforeyoubeginthisprocedure,ensurethatyouhavesuccessfullycompletedallofthestepsinDeployingOpsManagertovSphere.

Step1:SetUpOpsManager1. NavigatetothefullyqualifieddomainofyourOpsManagerinawebbrowser.

2. ThefirsttimeyoustartOpsManager,youmustchooseoneofthefollowing:

UseanIdentityProvider:IfyouuseanIdentityProvider,anexternalidentityservermaintainsyouruserdatabase.InternalAuthentication:IfyouuseInternalAuthentication,PCFmaintainsyouruserdatabase.

UseanIdentityProvider(IdP)1. LogintoyourIdPconsoleanddownloadtheIdPmetadataXML.Optionally,ifyourIdPsupportsmetadataURL,youcancopythemetadataURLinsteadoftheXML.

Note:YoucanalsoperformtheproceduresinthistopicusingtheOpsManagerAPI.Formoreinformation,seeUsingtheOpsManagerAPI .


https://docs.pivotal.io/pivotalcf/customizing/ops-man-api.html

2. CopytheIdPmetadataXMLorURLtotheOpsManagerUseanIdentityProviderloginpage.

3. EnteryourDecryptionpassphrase.ReadtheEndUserLicenseAgreement,andselectthecheckboxtoaccepttheterms.

4. YourOpsManagerloginpageappears.Enteryourusernameandpassword.ClickLogin.

5. DownloadyourSAMLServiceProvidermetadata(SAMLRelyingPartymetadata)bynavigatingtothefollowingURLs:

5a.OpsManagerSAMLserviceprovidermetadata: https://OPS-MAN-FQDN:443/uaa/saml/metadata5b.BOSHDirectorSAMLserviceprovidermetadata: https://BOSH-IP-ADDRESS:8443/saml/metadata

6. ConfigureyourIdPwithyourSAMLServiceProvidermetadata.ImporttheOpsManagerSAMLprovidermetadatafromStep5aabovetoyourIdP.IfyourIdPdoesnotsupportimporting,providethevaluesbelow.

SinglesignonURL: https://OPS-MAN-FQDN:443/uaa/saml/SSO/alias/OPS-MAN-FQDNAudienceURI(SPEntityID): https://OP-MAN-FQDN:443/uaaNameID:EmailAddressSAMLauthenticationrequestsarealwayssigned

7. ImporttheBOSHDirectorSAMLprovidermetadatafromStep5btoyourIdP.IftheIdPdoesnotsupportanimport,providethevaluesbelow.

SinglesignonURL: https://BOSH-IP:8443/saml/SSO/alias/BOSH-IPAudienceURI(SPEntityID): https://BOSH-IP:8443NameID:EmailAddressSAMLauthenticationrequestsarealwayssigned

8. ReturntotheOpsManagerDirectortile,andcontinuewiththeconfigurationstepsbelow.

InternalAuthentication1. WhenredirectedtotheInternalAuthenticationpage,youmustcompletethefollowingsteps:

EnteraUsername,Password,andPasswordconfirmationtocreateanAdminuser.EnteraDecryptionpassphraseandtheDecryptionpassphraseconfirmation.ThispassphraseencryptstheOpsManagerdatastore,andisnotrecoverable.IfyouareusinganHTTPproxyorHTTPSproxy,followtheinstructionsinConfiguringProxySettingsfortheBOSHCPI .ReadtheEndUserLicenseAgreement,andselectthecheckboxtoaccepttheterms.

Note:ThesameIdPmetadataURLorXMLisappliedfortheBOSHDirector.IfyouuseaseparateIdPforBOSH,copythemetadataXMLorURLfromthatIdPandenteritintotheBOSHIdPMetadatatextboxintheOpsManagerloginpage.

Note:Toretrieveyour BOSH-IP-ADDRESS ,navigatetotheOpsManagerDirectortile>Statustab.RecordtheOpsManagerDirectorIPaddress.


https://docs.pivotal.io/pivotalcf/customizing/pcf-director-proxy-settings.html

Step2:vCenterConfigPage1. LogintoOpsManagerwiththeAdminusernameandpasswordyoucreatedinthepreviousstep.

2. ClicktheOpsManagerDirectortile.

3. SelectvCenterConfig.


4. Enterthefollowinginformation:

vCenterHost:ThehostnameofthevCenterthatmanagesESXi/vSphere.vCenterUsername:AvCenterusernamewithcreateanddeleteprivilegesforvirtualmachines(VMs)andfolders.vCenterPassword:ThepasswordforthevCenteruserspecifiedabove.DatacenterName:ThenameofthedatacenterasitappearsinvCenter.VirtualDiskType:TheVirtualDiskTypetoprovisionforallVMs.Forguidanceonselectingavirtualdisktype,seeProvisioningaVirtualDiskinvSphere .EphemeralDatastoreNames(commadelimited):ThenamesofthedatastoresthatstoreephemeralVMdisksdeployedbyOpsManager.PersistentDatastoreNames(commadelimited):ThenamesofthedatastoresthatstorepersistentVMdisksdeployedbyOpsManager.VMFolder:ThevSpheredatacenterfolder(default: pcf_vms )whereOpsManagerplacesVMs.TemplateFolder:ThevSpheredatacenterfolder(default: pcf_templates )whereOpsManagerplacesVMs.


https://docs.pivotal.io/pivotalcf/customizing/disk-format.html

DiskpathFolder:ThevSpheredatastorefolder(default: pcf_disk )whereOpsManagercreatesattacheddiskimages.Youmustnotnestthisfolder.

5. SelectStandardvCenterNetworking.ThisisthedefaultoptionwhenupgradingOpsManager.ThisconfigurationisutilizedforPASonly.YouconfigureNSX-TintegrationforPKSwithinthePKStile.

6. ClickSave.

Step3:DirectorConfigPage1. SelectDirectorConfig.

2. IntheNTPServers(commadelimited)field,enteryourNTPserveraddresses.

3. LeavetheJMXProviderIPAddressfieldblank.

4. LeavetheBoshHMForwarderIPAddressfieldblank.

5. SelecttheEnableVMResurrectorPlugintoenableOpsManagerResurrectorfunctionality.

6. SelectEnablePostDeployScriptstorunapost-deployscriptafterdeployment.Thisscriptallowsthejobtoexecuteadditionalcommandsagainstadeployment.

Note:Afteryourinitialdeployment,youwillnotbeabletoedittheVMFolder,TemplateFolder,andDiskpathFoldernames.

Note:StartingfromPCFv2.0,BOSH-reportedcomponentmetricsareavailableintheLoggregatorFirehosebydefault.Therefore,ifyoucontinuetousePCFJMXBridgeforconsumingthemoutsideoftheFirehose,youmayreceiveduplicatedata.Topreventthis,leavetheJMXProviderIPAddressfieldblank.

Note:StartingfromPCFv2.0,BOSH-reportedcomponentmetricsareavailableintheLoggregatorFirehosebydefault.Therefore,ifyoucontinuetousetheBOSHHMForwarderforconsumingthem,youmayreceiveduplicatedata.Topreventthis,leavetheBoshHMForwarderIPAddressfieldblank.


7. SelectRecreateallVMstoforceBOSHtorecreateallVMsonthenextdeploy.Thisprocessdoesnotdestroyanypersistentdiskdata.

8. SelectEnableboshdeployretriesifyouwantOpsManagertoretryfailedBOSHoperationsuptofivetimes.

GCSBlobstore:SelectthisoptiontouseanexternalGoogleCloudStorage(GCS)endpoint.TocreateaGCSbucket,youwillneedaGCSaccount.FollowtheproceduresinCreatingStorageBuckets intheGCPdocumentation.AfteryouhavecreatedaGCSbucket,completethefollowingsteps:

1. BucketName:EnterthenameofyourGCSbucket.2. StorageClass:SelectthestorageclassforyourGCSbucket.Formoreinformation,seeStorageClasses intheGCPdocumentation.3. ServiceAccountKey:FollowthestepsintheCreateServiceAccountssectiontodownloadaJSONfilewithaprivatekey,andthenenterthecontentsoftheJSONfileintothefield.

9. Bydefault,PCFdeploysandmanagesanInternaldatabaseforyou.IfyouchoosetouseanExternalMySQLDatabase,completetheassociatedfieldswithinformationobtainedfromyourexternalMySQLDatabaseprovider:Host,Port,Username,Password,andDatabase.

Note:Youmustenablepost-deployscriptstoinstallPKS.


https://cloud.google.com/storage/docs/creating-bucketshttps://cloud.google.com/storage/docs/storage-classes

10. (Optional)DirectorWorkerssetsthenumberofworkersavailabletoexecuteDirectortasks.Thisfielddefaultsto 5 .

11. (Optional)MaxThreadssetsthemaximumnumberofthreadsthattheOpsManagerDirectorcanrunsimultaneously.ForvSphere,thedefaultvalueis 32 .Leavethefieldblanktousethisdefaultvalue.PivotalrecommendsthatyouusethedefaultvalueunlessdoingsoresultsinratelimitingorerrorsonyourIaaS.

12. LeavetheDirectorHostnamefieldblank.

13. EnsuretheDisableBOSHDNSserverfortroubleshootingpurposescheckboxisnotselected.

14. Optional:TosetacustombannerthatusersseewhenloggingintotheDirectorusingSSH,entertextintheCustomSSHBannerfield.

15. ClickSave.

Step4:CreateAvailabilityZonePageOpsManagerAvailabilityZonescorrespondtoyourvCenterclustersandresourcepools.MultipleAvailabilityZonesallowyoutoprovidehigh-availabilityandloadbalancingtoyourapplications.Whenyourunmorethanoneinstanceofanapplication,OpsManagerbalancesthoseinstancesacrossalloftheAvailabilityZonesassignedtotheapplication.Atleastthreeavailabilityzonesarerecommendedforahighlyavailableinstallationofyourchosenruntime.

Note:BOSHDNSmustbeenabledinallPKSdeployments.IfPASandPKSarerunningonthesameinstanceofOpsManager,youcannotusetheopt-outfeatureofBOSHDNSforyourPASwithoutbreakingPKS.IfyouwanttooptoutofBOSHDNSinyourPASdeployment,installthetileonaseparateinstanceofOpsManager.FormoreinformationaboutoptingoutofBOSHDNS,seeDisablingorOptingOutofBOSHDNSinPCF(PivotalKnowledgeBasearticle) andBOSHDNSServiceDiscovery(Beta)andOpt-OutOption intheOpsManagerv2.0ReleaseNotes.

Note:Afteryourinitialdeployment,youwillnotbeabletoedittheBlobstoreandDatabaselocations.


https://discuss.pivotal.io/hc/en-us/articles/115015720428-Disabling-or-Opting-Out-of-BOSH-DNS-in-PCFhttps://docs.pivotal.io/pivotalcf/2-0/pcf-release-notes/opsmanager-rn.html#bosh-dns

1. SelectCreateAvailabilityZones.

2. UsethefollowingstepstocreateoneormoreAvailabilityZonesforyourapplicationstouse:

ClickAdd.EnterauniqueNamefortheAvailabilityZone.EnterthenameofanexistingvCenterClustertouseasanAvailabilityZone.(Optional)EnterthenameofaResourcePoolinthevCenterclusterthatyouspecifiedabove.ThejobsrunninginthisAvailabilityZonesharetheCPUandmemoryresourcesdefinedbythepool.(Optional)ClickAddClustertocreateanothersetofClusterandResourcePoolfields.Youcanaddmultipleclusters.Clickthetrashicontodeleteacluster.Thefirstclustercannotbedeleted.

3. ClickSave.

Step5:CreateNetworksPage1. SelectCreateNetworks.

2. SelectEnableICMPcheckstoenableICMPonyournetworks.OpsManagerusesICMPcheckstoconfirmthatcomponentswithinyournetworkarereachable.

3. ClickAddNetworkandcreatethefollowingnetworks:

pks-infrastructure :forOpsManager,theBOSHDirector,thePKSbroker,andthePKSAPI.Ifyouhavealargedeploymentwithmultipletiles,youcanchoosetodeploythePKSbrokerandPKSAPItoaseparatenetworknamed pks-main .Seethetablebelowformoreinformation.pks-services :forcreatingthemasterandworkerVMsforKubernetesclusters.

Usethevaluesfromthefollowingtableasaguidewhenyoucreateeachnetwork,replacingtheIPaddresseswithrangesthatareavailableinyourvSphereenvironment:

Field Configuration

Name

Note:FormoreinformationaboutusingavailabilityzonesinvSphere,seeUnderstandingAvailabilityZonesinVMwareInstallations .

Note:IfyouaredeployingPKSwithNSX-Tintegration,seethenetworkconfigurationtableintheConfigureOpsManagersectionofInstallingandConfiguringPKSwithNSX-TIntegration.


https://docs.pivotal.io/pivotalcf/customizing/understand-az.html

InfrastructureNetwork

pks-infrastructureServiceNetwork LeaveServiceNetworkunchecked.

vSphereNetworkName MY-PKS-virt-net/MY-PKS-subnet-infrastructure

CIDR 192.168.101.0/26

ReservedIPRanges 192.168.101.1-192.168.101.9

DNS 192.168.101.2

Gateway 192.168.101.1

MainNetwork(Optional)

Field Configuration

Name pks-main

ServiceNetwork LeaveServiceNetworkunchecked.

vSphereNetworkName MY-PKS-virt-net/MY-PKS-subnet-pks

CIDR 192.168.16.0/26


DNS 192.168.16.2

Gateway 192.168.16.1

ServiceNetwork

Field Configuration

Name pks-services

ServiceNetwork SelecttheServiceNetworkcheckbox.

vSphereNetworkName MY-PKS-virt-net/MY-PKS-subnet-services

CIDR 192.168.20.0/22


DNS 192.168.20.2

Gateway 192.168.20.1

4. SelectwhichAvailabilityZonestousewiththenetwork.

5. ClickSave.

Step6:AssignAZsandNetworksPage1. SelectAssignAZsandNetworks.

Note:MultiplenetworksallowyoutoplacevCenteronaprivatenetworkandtherestofyourdeploymentonapublicnetwork.IsolatingvCenterinthismannerdeniesaccesstoitfromoutsidesourcesandreducespossiblesecurityvulnerabilities.

Note:IfyouareusingtheCiscoNexus1000vSwitch,seemoreinformationinUsingtheCiscoNexus1000vSwitchwithOpsManager .


https://docs.pivotal.io/pivotalcf/customizing/nexus-switch.html

2. Usethedrop-downmenutoselectaSingletonAvailabilityZone.TheOpsManagerDirectorinstallsinthisAvailabilityZone.

3. Usethedrop-downmenutoselectaNetworkforyourOpsManagerDirector.

4. ClickSave.

Step7:SecurityPage1. SelectSecurity.

2. InTrustedCertificates,enteracustomcertificateauthority(CA)certificatetoinsertintoyourorganization’scertificatetrustchain.ThisfeatureenablesallBOSH-deployedcomponentsinyourdeploymenttotrustacustomrootcertificate.IfyouwanttouseDockerRegistriesforrunningappinstancesinDockercontainers,usethisfieldtoenteryourcertificateforyourprivateDockerRegistry.Formoreinformation,seeUsingDockerRegistries .


https://docs.pivotal.io/pivotalcf/opsguide/docker-registry.html

3. ChooseGeneratepasswordsorUsedefaultBOSHpassword.PivotalrecommendsthatyouusetheGeneratepasswordsoptionforincreasedsecurity.

4. ClickSave.ToviewyoursavedDirectorpassword,clicktheCredentialstab.

Step8:SyslogPage1. SelectSyslog.

2. (Optional)TosendBOSHDirectorsystemlogstoaremoteserver,selectYes.

3. IntheAddressfield,entertheIPaddressorDNSnamefortheremoteserver.

4. InthePortfield,entertheportnumberthattheremoteserverlistenson.

5. IntheTransportProtocoldropdownmenu,selectTCP,UDP,orRELP.Thisselectiondetermineswhichtransportprotocolisusedtosendthelogstotheremoteserver.

6. (Optional)MarktheEnableTLScheckboxtouseTLSencryptionwhensendinglogstotheremoteserver.

InthePermittedPeerfield,entereitherthenameorSHA1fingerprintoftheremotepeer.IntheSSLCertificatefield,entertheSSLcertificatefortheremoteserver.


7. ClickSave.

Step9:ResourceConfigPage1. SelectResourceConfig.

2. Adjustanyvaluesasnecessaryforyourdeployment.UndertheInstances,PersistentDiskType,andVMTypefields,chooseAutomaticfromthedrop-downmenutoallocatetherecommendedresourcesforthejob.IfthePersistentDiskTypefieldreadsNone,thejobdoesnotrequirepersistentdiskspace.

3. ClickSave.

Step10:CompletetheOpsManagerInstallation1. ClicktheInstallationDashboardlinktoreturntotheInstallationDashboard.

2. ClickApplyChangesontherightnavigation.

NextStepsToinstallPKSonvSpherewithNSX-Tintegration,performtheproceduresinInstallingandConfiguringPKSwithNSX-TIntegration.

ToinstallPKSonvSpherewithoutNSX-Tintegration,performtheproceduresinInstallingandConfiguringPKS.

TouseHarbortostoreandmanagecontainerimages,seeInstallingandIntegratingVMwareHarborRegistrywithPKS .

[email protected].

Note:OpsManagerrequiresaDirectorVMwithatleast8GBmemory.

Note:IfyousetafieldtoAutomaticandtherecommendedresourceallocationchangesinafutureversion,OpsManagerautomaticallyusestheupdatedrecommendedallocation.


https://docs.pivotal.io/partners/vmware-harbormailto:[email protected]

PreparingtoInstallPKSonGCPThistopicoutlinesthestepsforpreparingtoinstallPivotalContainerService(PKS)onGCP.Seethefollowingsections:

GCPPrerequisitesandResourceRequirements

PreparingtoDeployPKSonGCP

DeployingOpsManagertoGCP

ConfiguringOpsManageronGCP

ConfiguringaGCPLoadBalancerforthePKSAPI

ConfiguringaGCPLoadBalancerforPKSClusters

[email protected].



GCPPrerequisitesandResourceRequirementsPagelastupdated:

ThistopicdescribestheprerequisitesandresourcerequirementsforinstallingPivotalContainerService(PKS)onGoogleCloudPlatform(GCP).

ResourceRequirementsInstallingPKSdeploysthefollowingtwovirtualmachines(VMs):

VM CPU RAM Storage

PivotalContainerService 1 4GB 20GB

PivotalOpsManager 1 8GB 160GB

EachKubernetesclusterprovisionedthroughPKSdeploystheVMslistedbelow.IfyoudeploymorethanoneKubernetescluster,youmustscaleyourallocatedresourcesappropriately.

VMName Number CPUCores RAM EphemeralDisk PersistentDisk

master 1 2 4GB 8GB 5GB

worker 1 2 4GB 8GB 10GB

InstallingPKSonGCPToinstallPKSonGCP,followtheproceduresbelow:

1. PreparingtoDeployPKSonGCP

2. DeployingOpsManagertoGCP

3. ConfiguringOpsManageronGCP

4. InstallingandConfiguringPKS

AboutDeployingPASandPKSThePivotalApplicationService(PAS)andPKSruntimeplatformsarebothdeployedbyOpsManagerusingBOSH.YoucandeploybothPASandPKSusingthesameOpsManagerinstanceinadevelopmentortestenvironment,butwerecommendthatyoudeployproductioninstallationsofPASandPKStoseparateOpsManagerinstances.Forincreasedsecurity,werecommenddeployingeachOpsManagerinstanceusingauniquecloudprovideraccount.

SeparateinstallationsofOpsManagerallowyoutocustomizeandtroubleshootruntimetilesindependently.YoumaychoosetoconfigureOpsManagerwithdifferentsettingsforyourPASandPKSdeployments.Forexample,PKSandmanyPASfeaturesdependonBOSHDNS.

IfyoudeployPAStoaseparateOpsManagerinstance,youcandisableBOSHDNSfortroubleshootingpurposes.PAScanrunwithoutBOSHDNS,butkeyfeaturessuchassecureservicecredentialswithCredHub,servicediscoveryforcontainer-to-containernetworking,andNSX-TintegrationdonotworkwhenBOSHDNSisdisabled.

IfyoudeployPASandPKStothesameOpsManagerinstance,youcannotdisableBOSHDNSwithoutbreakingyourPKSinstallationalongwiththePASfeaturesthatdependonBOSHDNS.

[email protected].



PreparingtoDeployPKSonGCPPagelastupdated:

ThisguidedescribesthepreparationstepsrequiredtoinstallPivotalContainerService(PKS)onGoogleCloudPlatform(GCP).

InadditiontofulfillingtheprerequisiteslistedintheGCPPrerequisitesandResourceRequirementstopic,youmustcreateresourcesinGCPsuchasanewnetwork,firewallrules,loadbalancers,andaserviceaccountbeforedeployingPKS.FollowtheseprocedurestoprepareyourGCPenvironment.

Step1:EnableGoogleCloudAPIsOpsManagermanagesGCPresourcesusingtheGoogleComputeEngineandCloudResourceManagerAPIs.ToenabletheseAPIs,performthefollowingsteps:

1. LogintotheGoogleDevelopersconsoleathttps://console.developers.google.com .

2. Intheconsole,navigatetotheGCPprojectwhereyouwanttoinstallPKS.

3. SelectEnableAPIs&ServicestoaccesstheAPILibrary.

4. Inthesearchfield,enter Compute Engine API andpressEnter.

5. OntheGoogleComputeEngineAPIpage,clickEnable.

6. Inthesearchfield,enter Cloud Resource Manager API andpressEnter.

7. OntheGoogleCloudResourceManagerAPIpage,clickEnable.

8. ToverifythattheAPIshavebeenenabled,performthefollowingsteps:

a. LogintoGCP:

$gcloudauthlogin

b. Listyourprojects:

$gcloudprojectslistPROJECT_IDNAMEPROJECT_NUMBERmy-project-idmy-project-name##############

ThiscommandliststheprojectswhereyouenabledGoogleCloudAPIs.

Step2:CreateServiceAccountsInorderforKubernetestocreateloadbalancersandattachpersistentdiskstopods,youmustcreateserviceaccountswithsufficientpermissions.

YouneedseparateserviceaccountsforKubernetesclustermasterandworkernodeVMs,andathirdaccountforBOSHandOpsManager.Pivotalrecommendsconfiguringeachserviceaccountwiththeleastpermissiveprivilegesanduniquecredentials.

CreatetheMasterNodeServiceAccount1. FromtheGCPConsole,selectIAM&admin>Serviceaccounts.

2. ClickCreateServiceAccount.

3. Enteranamefortheserviceaccount,andaddthefollowingroles:

ComputeEngine

StorageAdminNetworkAdminSecurityAdmin


https://console.developers.google.com

InstanceAdmin(v1)ComputeViewer

IAM

ServiceAccountUser

4. SelectFurnishanewprivatekeyandselectJSON.

5. ClickCreate.YourbrowserautomaticallydownloadsaJSONfilewithaprivatekeyforthisaccount.Savethisfileinasecurelocation.

CreatetheWorkerNodeServiceAccount1. FromtheGCPConsole,selectIAM&admin>Serviceaccounts.


3. Enteranamefortheserviceaccount,andaddtheComputeEngine>ComputeViewerrole.



CreatetheBOSH/OpsManagerServiceAccount1. FromtheGCPConsole,selectIAM&admin>Serviceaccounts.


3. Enteranamefortheserviceaccount,andaddthefollowingroles:

ServiceAccounts

ServiceAccountUserServiceAccountTokenCreator

ComputeEngine

ComputeInstanceAdmin(v1)ComputeNetworkAdminComputeStorageAdmin

Storage

StorageAdmin



Step3:CreateaGCPNetworkwithSubnets1. LogintotheGCPConsole .

2. NavigatetotheGCPprojectwhereyouwanttoinstallPKS.

3. SelectVPCnetwork,thenCREATEVPCNETWORK.

4. IntheNamefield,enter your-pks-virt-net . your-pks isalower-caseprefixtohelpyouidentifyresourcesforthisPKSdeploymentintheGCPconsole.

Note:PivotalrecommendsconfirmingthepermissionsofyourMasterNodeServiceAccount,WorkerNodeServiceAccount,andBOSH/OpsManagerServiceAccountafteryoucreatethem.Toverifytheseaccountpermissions,runthe gcloudauth

listcommand.Formoreinformation,see

gcloudaut

Date post:	04-Jul-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

Pivotal Container Service (PKS) Documentation › pdfs › pks-1-0.pdfNSX-T 2.1 Advanced Edition...

Documents