2
Dad Oracle DBA Sailor Musician working for Exitas Oracle Platinum Partner in BELGIUM
http://vanpupi.stepi.net
@vanpupi
About Exitas
Quality - Passion - Personality
• Team of 25 dba’s• Consultancy, managed services,…• http://www.exitas.be
The journey of a bi-stack to the cloud
3MembershipTiers• OracleACEDirector• OracleACE• OracleACEAssociate
bit.ly/OracleACEProgram
500+TechnicalExpertsHelpingPeersGlobally
Connect:
Nominateyourselforsomeoneyouknow:acenomination.oracle.com
@oracleace
Facebook.com/oracleaces
Agenda
• The project• Cloud semantics• The design• Tests• Provisioning• Initial load• Lessons learned• Q/A
The journey of a bi-stack to the cloud
Little red ridinghood and the big bad GDPR: how does it impact you
The project dates from January 2017 (The new content is added, where applicable)
Safe Harbour statement
And the message is still valid ;-)
How it started …
• Request for assistance for a move to the cloud?• it’s about 55TB prod data
• Prddb1 : 34TB• Prddb2 : 16TB• Prddb3 : 5TB
• Agile: “iterative incremental process”• oh by the way … non-prod is ~ 360TB … we think …
The journey of a bi-stack to the cloud
Disks
The journey of a bi-stack to the cloud
StandardDisks (HDD) PremiumDisks(SSD)
ConsistencyofprovisionedperformancePerformanceisnotprovisioned
This is what was available during the project:
Jan’17
Networking
Howto: https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-create-vm-accelerated-networking
The journey of a bi-stack to the cloud
Jan’17
Accelerated networking (ACN)
Accelerated networking:• General available • Recommended
https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-create-vm-accelerated-networking
The journey of a bi-stack to the cloud
Availability sets
• https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-manage-availability
• VMsdeployedondifferentracks(=FaultDomain(FD))anddifferentphysicalservers(=UpdateDomains(UD))
• CfrVMwareAnti-AffinityRules
The journey of a bi-stack to the cloud
Images
The journey of a bi-stack to the cloud
The current vm’s: Linux: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes
Windows: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes
Choosing a VM
• MS defined categories• Own requirements • VM = predefined set of resources
• # vCPUs• # vRAM• standard vs premium storage os/data disks• Maximum disk throughput• # data disks• Max disk throughput• Network bandwidth
The journey of a bi-stack to the cloud
Choosing a VM
The journey of a bi-stack to the cloud
1. Workload- Cpurequirements- Memory requirements- Diskrequirements
3.VMSize- CPU- RAM- #datadisks- Diskthroughput
2. Storage- S:faststorage- VMscanusepremiumstorage- VMsusesolid statedrives(SSD)
Standard_DS15_v2
Choosing a VM
The journey of a bi-stack to the cloud
Requirements:- High workload- High #disks- ACN needed- High throughput
—> DS15_V2—>Hard coupling between memory and cpu
Topology - current
The journey of a bi-stack to the cloud
DC1 DC2
BACKUPSHARED
BIDBDEDICATED
SAPBIBODEDICATED
DC1
DC2DC1ESXStretchedCluster
ESXClusterPROD ESXClusterNON-PROD
RHS– 8nodesspread over2DCs
VM VM VM VM
VM VM VM VM
DataguardVM
VM VM
Topology - To be: Requirements
•high network throughput ~ 825 MB/s—> RMAN Backup – CIFS share (Mos-note: 444809.1)
•High # number of disks —> Oracle database requires 34 TB
•Accelerated networking only available on DS15_v2 —> Backup Server
•Currently DS15_v2 only available and West Europe —> with accelerated networking
The journey of a bi-stack to the cloud
Topology - To be
The journey of a bi-stack to the cloud
VM1
VM2
VM3AZUREWest-Europe
PRODUCTION/STANDBY
DEVDB1
UATDB1 TESTDB1
REFRESH(Non-Prod)
Snapshots
DEDICATEDBACKUP
AZUREWest-Europe
Dataguard
ACCDB1
NFSServerArchiveData
DBPRD1
DBVM DBVM
DBPRD2
DBPRD3
DBDG1
DBDG2
DBDG3
Folder perDB
Folder perDB
Folder perDB
DEVDB2
UATDB2 TESTDB2
ACCDB2
DEVDB3
UATDB3 TESTDB3
ACCDB3
Testing
Network tests• Simple ping + TCP tests
Local disk performance - 2 tests:• 1 storage account + 34 disks• 34 storage accounts + 1 disk
Layout:• 1 volume group • XFS striped lvm
The journey of a bi-stack to the cloud
Testing
Test case 1 : Windows 2 windows
The journey of a bi-stack to the cloud
0100200300400500600700
1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829303132333435363738
MB/sec
Counter SampleNumber
SMBServerWriteMB/sec
0
200
400
600
800
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
MB/sec
Counter SampleCounter
NetworkAdapterReceivedMB/sec
Testing
Test case 2a : Linux 2 windows - Non Accelerated Networking
The journey of a bi-stack to the cloud
Testing
Test case 2b : Linux 2 windows - Accelerated Networking
The journey of a bi-stack to the cloud
0
50
100
150
200
250
300
350
400
450
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
MB/sec
Counter SampleNumber
SMBServerWriteMB/sec
NoAcceleratedNetworkingforLinux WithAcceleratedNetworkingforLinux
Testing
The journey of a bi-stack to the cloud
VM1
VM2
VM3AZUREWest-Europe
PRODUCTION/STANDBY
DEVDB1
UATDB1 TESTDB1
REFRESH(Non-Prod)
Snapshots
DEDICATEDBACKUP
AZUREWest-Europe
Dataguard
ACCDB1
NFSServerArchiveData
DBPRD1
DBVM DBVM
DBPRD2
DBPRD3
DBDG1
DBDG2
DBDG3
Folder perDB
Folder perDB
Folder perDB
DEVDB2
UATDB2 TESTDB2
ACCDB2
DEVDB3
UATDB3 TESTDB3
ACCDB3
Testing
The journey of a bi-stack to the cloud
0
50
100
150
200
250
300
350
400
450
500
1 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 52 56 60 64 68 72 76 80 88 96 112 128
PhysicalI/O- ReadMB/s
upd0% upd10%
upd20%
upd30%
Testing
The journey of a bi-stack to the cloud
0
100
200
300
400
500
600
1 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 52 56 60 64 68 72 76 80 88 96 112 128
PhysicalI/O- ReadMB/s
upd0% upd10%
upd20%
upd30%
Testing
Backups• Prod db
• Protected by Dataguard• Bi-weekly full backup
• Week 1 -> Big PRDDB1 backup • Week 2 -> Medium PRDDB2 / Small PRDDB3 backup
• Retention• 1st week is on premium storage available via share• Week 2 -> 4: cfr azure offload
The journey of a bi-stack to the cloud
Testing
- 3 backup servers in AS + ACN, but in Separate AS than the db servers- 1 share per server- rman - 4 channels per share - tuning parameters needed: - sql 'alter system set "_backup_disk_bufcnt"=20 scope=memory'; - sql 'alter system set "_backup_disk_bufsz"=1048576 scope=memory'; - sql 'alter system set "_backup_file_bufcnt"=20 scope=memory'; - sql 'alter system set "_backup_file_bufsz"=1048576 scope=memory'; - CONFIGURE DEVICE TYPE DISK PARALLELISM 4; - CONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 20G MAXOPENFILES=40;
- Maximum backup speed achieved was the vm limit (450MB/s) , so throttling will be needed.- Restore speed is about 400MB/s per oracle server
The journey of a bi-stack to the cloud
Expect the unexpected
• Managed disks are now globally available. • Simple management• Snapshots possible
—> Recommended!
https://docs.microsoft.com/en-us/azure/virtual-machines/windows/managed-disks-overview
The journey of a bi-stack to the cloud
Provisioning
The journey of a bi-stack to the cloud
FullprovisioningScenario RefreshScenario
VirtualMachine Exists: sourceVM+db(standby)New: targetVMnon-production
Exists:sourceVM+db (standby)Exists:targetVM
Storage Exists:sourcestorageNew:targetstoragenewVM
Exists:sourcestorageExists:target storagetargetVM
Data Exists:source dbNew:targetdb
Exists:sourcedbNew:targetdb
1.STDBYDBCOMPONENTS2.ACCDBCOMPONENTS3.ACCDBPRE-REFRESHSCRIPTS4.SHUTDOWNACCDB5.UNMOUNTACCFILESYSTEMS6.SHUTDOWNSTDBYDB7.SNAPSHOTSTDBYDISKS8.DETACHACCAZUREDISKS9.SNAPSHOT->AZUREDISK10.ATTTACHACCAZUREDISKS11.MOUNTACCFILESYSTEMS12.STARTU/RECOVERACCDB13.ACCDBPOST-REFRESHSCRIPTS
Provisioning
The journey of a bi-stack to the cloud
VGACCDB1
D1
1
ORA
CLESTAN
DBY
1
D1
1
D1
2 2
D1
2
sda
1DGDB1
DB
D1
40
D1
40 40
2
sdb
sdal
......
40
ORACLEACCEPTAN
CE1
sda
1 ACCDB1
DB2
sdb
sdal
...
40
VGDGDB1
Initial data load
• AZ copy
• express route
• No snowball / truck /… (yet)
The journey of a bi-stack to the cloud
And we needed it …
Initial data load
• Lowest Lead Time for production cut-over Oracle Databases
• Phased approach to spread workload• Tested and validated through dry-runs to assure
production move• One-off approach : good enough solution/cost
The journey of a bi-stack to the cloud
Initial data load
• Full Production backup – Physical TransferWhen ? Migration Day – 6What ? 50 TB of three databasesHow ? Disk copy and Courier Transfer
• Incremental Production backup – Physical TransferWhen ? Migration Day-2What ? 15 TB of incremental changes of 5 days How ? Disk copy and Courier Transfer
• Incremental Production backup – Network TransferWhen ? Migration DayWhat ? 6 TB of incremental changes of 5 daysHow ? ExpressRoute Network Transfer
The journey of a bi-stack to the cloud
Initial data load
LIMITS:• Azure BLOBS have a throughput limit of 60 Mbytes / sec• Azure Storage Accounts : # : 250 per subscription• Max ingress per storage account (EU) : 5 Gbps (GRS/ZRS), 10 Gbps (LRS)• Max egress per storage account (EU) : 10 Gbps (GRS/ZRS), 15 Gbps (LRS)• Standard Storage Account : 20.000 IOPS, 800MB/sec• Premium Storage disk P20 : 2300 IOPS, 150 MB/sec• Premium Storage disk P30 : 5000 IOPS, 200 MB/sec• VM DS15_V2 : Disk : 64.000 IOPS, 640 MB/sec , Network : 15 Gbps• VM GS5 : 80.000 IOPS / 2000 MB/sec, , Network : 32 Gbps
The journey of a bi-stack to the cloud
Lessons learned
• Get people out of their comfortzones!—> old procedures, lots of manual work.
• People get uncomfortable by change.—> No decent service catalog so every exceptional situation is executed.
• Trust but verify the facts
The journey of a bi-stack to the cloud
Lessons learned
• Check! What?! Double check
• Fail early and correct.
• Involve the vendor
• Ask for previews, document and Test them!
The journey of a bi-stack to the cloud
Lessons learned
Expect the unexpected• project manager got fired• oracle changed the licensing rules
!!! But we delivered !!!
The journey of a bi-stack to the cloud
The journey of a bi-stack to the cloud
Mail: [email protected] Twitter: @vanpupi