Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
ZeroDowntimeforJavaApplications:UseCasesandSolutions
Nirmala Sundarappa,Kuassi Mensah,JeanDeLavareneJDBCandUCPTeam
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
SafeHarborStatement
Thefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,timing, andpricingofanyfeaturesorfunctionalitydescribedforOracle’sproductsmaychangeandremainsatthesolediscretionofOracleCorporation.
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
ProgramAgenda
BuildingBlocksofHighAvailability
PlannedMaintenance:UsecasesandSolutions
UnplannedOutage:UsecasesandSolutions
Questions
1
2
3
4
3
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
ProgramAgenda
HighAvailabilityBuildingBlocks
PlannedMaintenance:Usecases&Solutions
UnplannedOutage:Usecases&Solutions
Questions
1
2
3
4
4
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
HighAvailabilityBuildingBlocks• FastApplicationNotification(FAN)• StandardRequestboundary• RecommendedConnectionURL• TransparentApplicationContinuity(TAC)• OracleConnectionManagerinTrafficDirectorMode• JavaintheDatabase(OJVM)RollingUpgrade
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
FastApplicationNotification(FAN)
• TheFastApplicationNotification(FAN)allowsapplicationstobenotifiedinstantaneouslyofOracleDatabaseevents
• SupportedFANeventsarementionedbelow– NodeEvents:NodeDown,PublicNetworkDown– InstanceEvents:InstanceDown,InstanceUp– ServiceEvents:ServicememberDown,ServicememberUp,ServiceDown,Service
Up– DatabaseEvents:DatabaseDown,DatabaseUp
• TheSimplefan.jar furnishestheJavaAPIsformanagingFANevents• TheUniversalConnectionPool(UCP)andJDBCsupportFAN
6
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
UniversalConnectionPool(UCP)• ucp.jar availableonOTN,OracleMavenand$ORACLE_HOME/ucp/lib• Forusestand-aloneorwithJavaapplicationcontainers
– Tomcat,WebSphere,JBOSS,WLS(ActiveGridLink)
• TightintegrationwithRAC,ADG,andGDS• Built-insupportforFastApplicationNotification(FAN)events• Built-insupportforTransparentApplicationContinuity
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
StandardRequestBoundary
PoolDataSource pds = getPoolDataSource();Connection conn = pds.getConnection();conn.beginRequest()PreparedStatement pstmt = …
…SQLstmt,localcalls,RPC…
conn.commit();conn.endRequest();conn.close();
8
BeginRequest
RequestBody
EndRequest
JDK9
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
RecommendedConnectionURL
alias =(DESCRIPTION =
(CONNECT_TIMEOUT=90)(RETRY_COUNT=20)(RETRY_DELAY=3)
(TRANSPORT_CONNECT_TIMEOUT=3)
(ADDRESS_LIST =
(LOAD_BALANCE=on)
(ADDRESS = (PROTOCOL = TCP)(HOST=primary-scan)(PORT=1521)))
(ADDRESS_LIST =
(LOAD_BALANCE=on)
(ADDRESS = (PROTOCOL = TCP)(HOST=secondary-scan)(PORT=1521)))
(CONNECT_DATA=(SERVICE_NAME = gold-cloud)))
DB12.2
9
SingledescriptionAutomaticRetries
NorelianceonDNSconfig
NeverusetheDefaultService(DBorPDBName)
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
ApplicationContinuityDB18c
• Supportsapplicationswithsimplestate,orhavelogintriggerorcallbacktorestoretheircomplexstates,orthatcleantheircomplexstates(egDBMS_SESSION.RESET_PACKAGE,freetemplobs)
• Checksthatstateisrestorableatconnectioncheckout– Notallstateschecked–Won’treplayside-effects(stuffoutsidethesession,eg UTL_HTTP)
• Firstcallfailure:need18cclient• Applicationsw/oourpools:implicitboundaries(JDBC18c;OCI20c)
– Requiresoptionalclientsetting
Server-side
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
ConnectionManager(CMAN)- TrafficDirectorMode
11
C++
CMAN- TrafficDirectorMode
C
Databases
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.| 12
ZeroApplicationDowntime
• Automaticallyredirectandreestablishdatabaseconnections– Hidedatabaseplanned/unplannedoutagesandPDBrelocationsfromapplications
• SupportmultipleinstancesofConnectionManager– NoSinglePointofFailure
OracleDatabase
CMAN– TrafficDirectorMode
Eventnotification
Connection isre-directedtoanewinstanceorPDB
C++C
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
ProgramAgendawithHighlight
BuildingBlocksofHighAvailability
PlannedMaintenance:Usecases&Solutions
UnplannedOutage:Usecases&Solutions
Questions
1
2
3
4
13
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
PlannedMaintenance:Usecases&Solutions• NodeMaintenance• DatacenterMaintenance• PDBrelocation• PlannedMaintenancewithCMAN-TDM• ConditionalRAC-RollingofOJVMPatching
PlannedMaintenance
PatchesRepairs
UpgradesChanges
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
PlannedNodeMaintenance
15
InstanceA
InstanceB
Application
DrainSession
MAINTENANCE
FANDOWN
FANUP
BOTHINSTANCES ONEINSTANCE BOTHINSTANCES
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
PlannedNodeMaintenanceJavaRequirements (classpath)• DB12.2+ojdbc8.jar
• DB12.2+ucp.jar• DB12.2+ ons.jar
• Simplefan.jar ifnotusingUCP
ServerSideConfiguration• Auto-draining(12.2)srvctl stop service -node . -drain_timeout.. –stopoption[immediate|transactional]
• SetFAILOVER_RESTOREtolevel1srvctl modify service –db <db_name> -service <svc_name> -failover_restorelevel1
• EnableRuntimeLoadBalancing– Set –rlbgoal to SERVICE_TIME or THROUGHPUT
– Set –clbgoal to SHORT
16
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
PlannedNodeMaintenancewithJDBCDB12.2JDBCdriver(ojdbc8.jar)
• Driverclosesthedatabasesessionsata“SafePlace”(Seeconnectionvalidationoptions)• Applicationmustuseoneofthesafedrainingmethods
– ValidationSQLwithahint/*+CLIENT_CONNECTION_VALIDATION*/SELECT1FROMDUAL– isValid(),pingDatabase(),isUsable(),endRequest()methods
• Requiressimplefan.jar andons.jar intheclasspath
17
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
ConnectionValidationOptionsforContainersJavaConnectionPool ConnectionValidationOptions
OracleWebLogicGenericandMDSdatasources TestConnectionsOnReserve,TestConnectionsOnRelease,TestConnectionsOnCreate
IBMWebSphere PreTest Connection
WildFly /RedHat JBoss check-valid-connection-sql
ApacheTomCat TestonBorrow, TestonRelease
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
NECUseCase:DataCenterMaintenanceusingRAC+DataGuard
FAN
WebLogic
FAN
StartServices
TNSconnectionretry
Primary
Standby Primary
Standby
Swichover wait
RAC#1
RAC#2
Upgrade
Maintenance
21
DrainSession
:requestandresponse
120sec
Confirmnoimpacttoclients
NoImpact!!!
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
DataCenterMaintenanceJavaRequirements• DB12.2+ojdbc8.jar
• DB12.2+ucp.jar• DB12.2+ ons.jar
• SetRETRY_COUNT, RETRY_DELAY,longenoughtosustainswitchoverE.g.,RETRY_COUNT=30 andRETRY_DELAY=10
ServerSideConfiguration• UseRAC+DataGuard
• Auto-drainingsrvctl stop service -node . -drain_timeout.. –stopoption[immediate|transactional]
• EnableRLBontheserverside
• Killlongrunningsessions(enableAConJavaside)
20
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
REDOsync
NECUseCase:PDB OnlineRelocate
Other PDBs
Other PDBs
Migration PDB
requestandresponse
CDB#1
WebLogic
…
CDB#2
(1) Relocate
OnlineCopy
(2)Open &Start Service
Confirmednoerrorstoalltenant’sclients
Application Continuityenablesnoerrorreplay
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
PDBRelocateJavaRequirements• DB12.2+ojdbc8.jar
• DB12.2+ucp.jar• DB12.2+ ons.jar
• Drainsatsafeplaces
ServerSideConfiguration• Auto-drainingsrvctl stop service -node . -drain_timeout.. –stopoption[immediate|transactional]
22
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
CMANTDMPlannedOutages&PDBRelocation
• Non-pooledConnections- Anyclient(11.2.0.4+)– Norequestboundaryinformationfromtheclient– Servicestopped/startedappropriatelyonServer– 11.2.0.4+Server:StopServiceatTransactionBoundary– 18Server:StopServiceatRequestBoundary,basedonSQLcommands(Cont.AC)
• PooledConnections- JDBC(12.1+),– Servicerelocationforplannedoutage(11.2.0.4+Server)– SupportPDBrelocationevenwhenONSisnotconfigured(18.1+Serveronly)– ConnectionMultiplexingatrequestboundary– PDBRelocation:CMAN-TDMrespondstoin-bandnotificationswhenaPDBisrelocatedi.e.,evenwhenONSisnotconfigured(18.1+Serveronly)
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
DB18.4OJVMPatching(planned)• Nofunctionalchange• Non-Javaserviceavailablealltime (atleastoneinstance)• Javaserviceavailablealltime,except~10secbrown-out• Noerrorreportedduringbrown-out• CDB/PDBssupported(mayincurlongerbrown-out)• Lookingintoback-portingto11.2and12.1(plannotfinalized)
OJVM Rolling Upgrade- Planned MaintenanceOverview
~10secstallJavaService
(OJVMunpatched)
NonJavaService
Clusterwide
JavaService(OJVMpatched)
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
ProgramAgendawithHighlight
BuildingBlocksofHighAvailability
PlannedMaintenance:Usecases&Solutions
UnplannedOutage:Usecases&Solutions
Questions
1
3
2
4
25
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.| 26
S#@$%
UnplannedOutages
UnpredictableResponse&Throughput
SiteDisasters
DataCorruption
Whichoutageclassesdoesyourbusinessneedtohandle?
HumanErrors
Whatkindofoutages?
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
UnplannedOutages:Usecases&Solutions• NetworkOutage• DatabaseInstanceOutage• NodeDown• DatacenterDown• UnplannedOutagewithCMAN-TDM
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
SolutionsforhandlingUnplannedOutage
• UseUCPwithreplaydatasource(oracle.jdbc.replay.OracleDataSourceImpl)• EnableApplicationContinuity(AC)ontheserverside• UseAPIstochecktheACstatistics
– OracleDataSource.getReplayStatistics(StatisticsReportType) and OracleDataSource.getReplayStatistics()
• AlignApplicationandServerTimeouts– http_request_timeout shouldbehigherthanREAD_TIMEOUTontheconnURL
• DisableAUTOCOMMIT
28
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
NECUseCase:PublicNetworkDown
Applicationwait1s
FAN
:requestandresponse
…RACNode#1
WebLogic
RACNode#2
PDBservicesPDBs
PDBservicesPDBs
PublicNetworkdown
NoErrors!!!
Confirmednoerrorstoalltenant’sclients
Reconnect
ApplicationContinuity
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
NECUseCase:DBInstanceDown
ApplicationwaitsmaxFAST_START_MTTR_TARGET
Reconnect
FAN
:requestandresponse
…RACNode#1
WebLogic
RACNode#2
PDBservicesPDBs
PDBservicesPDBs
DB Reconfig & Recovery
instancedown
NoErrors!!!
Confirmednoerrorstoalltenant’sclients
ApplicationContinuity
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
NECUseCase:DBNodeDown
ApplicationwaitMisscount+FSMT (max)
FAN
:requestandresponse
…RACNode#1
WebLogic
RACNode#2
PDBservicesPDBs
PDBservicesPDBs
GRID + DB Reconfig & Recovery
Nodedown
NoErrors!!!
Confirmednoerrorstoalltenant’sclients
Reconnect
ApplicationContinuity
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
UnplannedOutage:TimerAlignment• Applicationtimeout(READ_TIMEOUT)shouldbehigherthanthedetectionandrecoverytimesfortheunderlyingsystems.
• AligntheapplicationtimeoutswithRACandDataGuardfailovertimers.– Read_timeout >>EXADATAspecialnodeeviction(2secondsin12.1.0.2)– Read_timeout >>Misscount (default30sec,modifiable)– Read_timeout >>DataGuardObserver,FastStartFailoverThreshold(default30sec,modifiable)AndFastStartFailoverThreshold>>>Misscount (atleasttwice)
– Read_timeout >>FAST_START_MTTR_TARGET– Read_timeout >>NETlevel(RETRY_COUNT+1)*RETRY_DELAY– Read_timeout << Replay_Initiation_Timeout (modifiableontheservice,default300second)
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
CMAN-TDMTransparentUnplannedOutages
• Non-PooledConnections- Anyclient(11.2.0.4+)– Simplesessionstateandcursorstatearere-established– SELECTSandfirstDMLsarereplayed– SupportsPlannedOutageformanyapplications(solongasonlysimplesessionstateandcursorstateneedtobepreservedacrosstherequest/transactionboundaries)
– SupportsUnplannedOutageforreadmostlyapplications
• PooledConnections- JDBC(12.1+)– LeverageTAF“FailoverRestore”featureonCMAN-TDMtoreconnectandrestoresimplestate
– LeverageApplicationContinuity(planned)
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
ProgramAgendawithHighlight
BuildingBlocksofHighAvailability
PlannedMaintenance:Usecases&Solutions
UnplannedOutage:Usecases&Solutions
Questions
1
4
2
3
34
Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|
Resources
35
JDBC&UCPLandingpage– oracle.com/jdbc
JDBC&UCPBlogs- https://blogs.oracle.com/dev2dev
JDBCwithDBCloud- https://bit.ly/2MP1NIE
JDBCDownloadPage- https://bit.ly/2wccOdB