Post on 21-Jan-2018
transcript
© 2015 Pythian - Presentation for Oracle OpenWorld
Who Am I
2
• Oracle ACE
• 15 years in Oracle field
• Joined Pythian 2003
• Part of Pythian Consulting Group
• Special projects
• Performance tuning
• Critical services
• Presenter at: IOUG, RMOUG, UKOUG, OpenWorld
“oracle pinup”
© 2015 Pythian - Presentation for Oracle OpenWorld3 © 2011 Pythian - Confidential
Why Companies Trust Pythian• Recognized Leader:
• Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and SQL Server
• Work with over 150 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments
• Expertise:
• One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 7 Oracle ACEs/ACE Directors.
• Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC.
• Global Reach & Scalability:
• 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response
© 2015 Pythian - Presentation for Oracle OpenWorld
Questions for Audience
5
• How many have RAC today?
• How many have RAC for HA?
• How many have RAC for scalability?
• How many plan to move to RAC?
• How many develop on RAC?
© 2015 Pythian - Presentation for Oracle OpenWorld
Scale UP vs Scale OUT
7
• Can have many meanings
• Scale UP
• Get a bigger machine
• Scale OUT
• Get more machines
© 2015 Pythian - Presentation for Oracle OpenWorld
Database Performance
8
• Has 2 primary components
• CPU – compute capacity
• IO – data transfer capacity
• Disk
• Network
• Cache (RAM)
• Implementation specifics
• Database scalability – latches, mutexes, threading etc.
© 2015 Pythian - Presentation for Oracle OpenWorld
Database Performance
9
Disk
Server
CPU
RAM RAM
APP
© 2015 Pythian - Presentation for Oracle OpenWorld
Database Performance – Scale UP
10
Disk
Server
CPU
RAM RAM
APP
CPU
RAM RAM
© 2015 Pythian - Presentation for Oracle OpenWorld
Database – Scale OUT
11
Disk
Server
CPU
RAM RAM
APP
Server
CPU
RAM RAM
Server
CPU
RAM RAM
© 2015 Pythian - Presentation for Oracle OpenWorld
Scale UP – challenges 2011
12
• Scale UP
• Limited “max size”
• Appears unified, but is not, NUMA
• Slower CPUs
• 12.8-GB/sec per QPI
• Up to 88 Gb/sec of network (NEM)
• 8 PCIe slots
Sources: http://www.oracle.com/technetwork/articles/systems-hardware-architecture/sun-server-x2-8-architecture-1689083.pdf
© 2015 Pythian - Presentation for Oracle OpenWorld
CPU Comparison 2011
13
X3-2• 8 cores per chip
• 2 chips max
• Server with 16 cores:
702 (SPECint®_rate2006)
• 35% capacity of x3-8
• 43.8 per core
• 1.8 x faster
X3-8• 10 cores per chip
• 8 chips max
• server with 80 cores:
1990 (SPECint®_rate2006)
• 2.8 X Capacity
• 24.8 per core
• 56% speed of x3-2
Sources:
http://www.spec.org/cpu2006/results/res2012q4/cpu2006-20121104-24909.html
http://www.spec.org/cpu2006/results/res2012q3/cpu2006-20120710-23574.html
© 2015 Pythian - Presentation for Oracle OpenWorld
CPU Comparison 2015
14
X5-2• 18 cores per chip
• 2 chips max
• Server with 36 cores:
1400 (SPECint®_rate2006)
• 30% Capacity of X4-8
• 38.8 per core
• 45 MB CPU cache
• 1U
X4-8• 15 cores per chip
• 8 chips max
• Server with 120 cores:
4610 (SPECint®_rate2006)
• 3.2 X capacity of x5-2
• 38.4 per core
• 40 MB CPU cache
• 4USources:
http://spec.org/cpu2006/results/res2014q4/cpu2006-20141007-32157.html
http://spec.org/cpu2006/results/res2014q4/cpu2006-20141120-33211.html
© 2015 Pythian - Presentation for Oracle OpenWorld
Scale OUT
16
• Requires Synchronisation Software – Oracle RAC
• Requires “interconnect” network
• Low latency critical
• Requires additional CPU resources for cross-node synchronisation
• Some operations (ex. locking) significantly increased latency (network vs. memory)
© 2015 Pythian - Presentation for Oracle OpenWorld
Adding CPU hardware
17
• Both Scale UP and Scale OUT increase total CPU capacity
• and IO network bandwidth
• Does not improve IO latency
• Does not improve IO requests capacity
• Does not improve application response time
• Does not make your batch job run faster
© 2015 Pythian - Presentation for Oracle OpenWorld
Adding CPU hardware
18
• Adds more workers, but you must keep them busy – at the same time
• Execute more tasks in parallel
• Parallelise existing tasks
© 2015 Pythian - Presentation for Oracle OpenWorld
Adding CPU hardware
19
• Unless more RAM is added
• Indirectly affects IO requests capacity
• Indirectly affects IO latency
© 2015 Pythian - Presentation for Oracle OpenWorld
Why Scale OUT Oracle
21
• Natural High Availability
• Infinite scalability – realistically
• Smaller “block” of failure
© 2015 Pythian - Presentation for Oracle OpenWorld
Challenges with Oracle RAC
22
• Concurrent modifications latency increased drastically
• Requires block transfer (CPU + network)
• Requires partial disk flush (CPU + latency)
• Concurrent reads do not share RAM cache
• Each node has local copy
• Oracle RAC Cache Fusion Reads slow
• timings similar to SSD read speeds – 10,000x slower than memory reads
© 2015 Pythian - Presentation for Oracle OpenWorld
Data Block
24
• Oracle stores data in blocks, in rows
• Each segment (table) has independent blocks*
• Lowest granularity in RAC data exchange is block*
• Modifying any data requires the block to be ‘acquired’ in local SGA (cache)
Oracle Table Block
8 KiB (8192)
1, Christo Kutrovsky, Canada, 99$
2, Alex Gorbachev, Canada, 98$
3, Marc Fielding, Canada, 101$
© 2015 Pythian - Presentation for Oracle OpenWorld
Inserting Data
25
• Inserting a row is modifying the block
Oracle Table Block
8 KiB (8192)
1, Christo Kutrovsky, Canada, 99$
2, Alex Gorbachev, Canada, 98$
3, Marc Fielding, Canada, 101$
4, Vanessa Simons, Canada, 199$
© 2015 Pythian - Presentation for Oracle OpenWorld
Block needs to move
26
• Block will bounce from node to node depending on where the insert runs
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
APP
4, Vanessa Simons, Canada, 199$5, John Smith, Canada, 777$ 6, Scott Tiger, Canada, 333$
© 2015 Pythian - Presentation for Oracle OpenWorld
Inserting Data
27
• Well that’s not efficient, can’t each node insert into it’s own block?
© 2015 Pythian - Presentation for Oracle OpenWorld
Inserting Data
28
• Yes we can! But…
• ASSM
• Instances can “own” L1 space metadata, which maps to N blocks
• Larger segments
• To have multiple L1 space metadata, larger segments are required
© 2015 Pythian - Presentation for Oracle OpenWorld
ASSM – bitmapped space mgmt
29
ASSM
L3
L2
L2
L1
1
L1
2
L1
3
L1
4
L1
5
L1
6
L1
7
L1
8
BLOCKS
1 2 3 4
5 6 7 8
BLOCKS9 10 11 12
13 14 15 16
© 2015 Pythian - Presentation for Oracle OpenWorld
ASSM
30
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
APP
4, Vanessa Simons, Canada, 199$ 5, John Smith, Canada, 777$ 6, Scott Tiger, Canada, 333$
L1
1
BLOCKS
L1
2
BLOCKS
L1
3
BLOCKS
© 2015 Pythian - Presentation for Oracle OpenWorld
Primary Key
31
• Primary Key – an index
• Each row has a very specific place
• Must insert into last block, always
• Sequentially generated
• … or is it?
Oracle Table Block
8 KiB (8192)
1, Christo Kutrovsky, Canada, 99$
2, Alex Gorbachev, Canada, 98$
3, Marc Fielding, Canada, 101$
4, Vanessa Simons, Canada, 199$
5, John Smith, Canada, 777$
6, Scott Tiger, Canada, 333$
7, ……………………..
© 2015 Pythian - Presentation for Oracle OpenWorld
Sequences
32
• Sequences can be
• Cached or NOcached
• Ordered or NOordered
• Cached NOordered with large cache are your best friend
• A sequence’s cache “sticks” with a RAC node
• cache 10,000 – 10,000 sequential values assigned to node 1
© 2015 Pythian - Presentation for Oracle OpenWorld
NO ORDER
33
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
APP
?, Vanessa Simons, Canada, 199$ ?, John Smith, Canada, 777$ ?, Scott Tiger, Canada, 333$
10,001
10,002
20,001
20,002
30,001
30,002
© 2015 Pythian - Presentation for Oracle OpenWorld
Sequences
34
• Cached noordered 10,000 cache – almost perfect
• Oracle numbers “compress” zeroes
• 10,000 and 20,000 are next to each other
• Sequence dictionary updates moves blocks
• Order not maintained
• But what if it’s required?
• Is it required?
• When do you consider the row “inserted”?
• What if delayed due to lock/concurrency?
© 2015 Pythian - Presentation for Oracle OpenWorld
Index on date inserted
35
• Concurrently inserted records have same date or very close
• Specific ordering in blocks
• No flexibility on values
• Same for index on Country or any other “same value” column
Oracle Table Block
8 KiB (8192)
1, Christo Kutrovsky, Canada, 99$
2, Alex Gorbachev, Canada, 98$
3, Marc Fielding, Canada, 101$
4, Vanessa Simons, Canada, 199$
5, John Smith, Canada, 777$
6, Scott Tiger, Canada, 333$
7, ……………………..
© 2015 Pythian - Presentation for Oracle OpenWorld
Dates
36
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
APP
?, Vanessa Simons, Canada, 199$ ?, John Smith, Canada, 777$ ?, Scott Tiger, Canada, 333$
11:09:01 11:09:01 11:09:01
© 2015 Pythian - Presentation for Oracle OpenWorld
Same Value Index
37
• Requires Partitioning
• Changes everything
• Partition key selection is non-trivial
• Local Indexes
• Lots of combinations
• Primary key local or global
• Depends on partition key and type
• Lots of considerations
• Data locality
© 2015 Pythian - Presentation for Oracle OpenWorld
Partition 4
BLOCKS
1 2 3 4
5 6 7 8
IDX
Date
IDX
PK
Partitioning
38
Partition 1
BLOCKS
1 2 3 4
5 6 7 8
IDX
Date
IDX
PK
Partition 2
BLOCKS
1 2 3 4
5 6 7 8
IDX
Date
IDX
PK
Partition 3
BLOCKS
1 2 3 4
5 6 7 8
IDX
Date
IDX
PK
© 2015 Pythian - Presentation for Oracle OpenWorld
Same Value Index
39
• Multiple partitions
• Create multiple blocks that have the “current” date
• “current” block will still move around the cluster, but decreases chances of “collisions”
• Still far from ideal
© 2015 Pythian - Presentation for Oracle OpenWorld
Same Value Index - Other
40
• Alternative solutions
• Prefix same value index with customer ID
• Prefix same value index with INSTANCE_ID
• “fake” local index
• Relies on skip scan
© 2015 Pythian - Presentation for Oracle OpenWorld
APPAPPApps
42
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
APP
11:09:01 11:09:01 11:09:01
© 2015 Pythian - Presentation for Oracle OpenWorld
Apps
43
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
APP
Server
APP
Server
APP
© 2015 Pythian - Presentation for Oracle OpenWorld
Application partitioning
44
What if I told you it doesn’t have to be like that?
© 2015 Pythian - Presentation for Oracle OpenWorld
Application partitioning
45
• By Functionality
• Not all functions of the application require concurrent modifications
• Isolating by functionality is the easiest and fastest way to ‘fix’ RAC unfriendly activities
• Such functionality has inherently bad scalability properties, as number of cores increase, this will become bigger problem
• Cache fusion will mirror concurrently read data
• Multiple nodes do not ‘add’ to caching benefits
© 2015 Pythian - Presentation for Oracle OpenWorld
Apps
46
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
APP
Server
APP
Server
APP
Module(s)
With
Concurrent
Modifications
© 2015 Pythian - Presentation for Oracle OpenWorld
Application Partitioning
47
What if a single module exceeds the performance capacity of a single node?
© 2015 Pythian - Presentation for Oracle OpenWorld
More Performance
48
• Tune it
• What are you doing?
• Can you not do it?
• Can you do it more efficiently?
• Are you using the latest hardware?
• Should probably start with this
• Still need more?
© 2015 Pythian - Presentation for Oracle OpenWorld49
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
APP
Server
APP
Server
APP
Server
APP
Server
APP
© 2015 Pythian - Presentation for Oracle OpenWorld50
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
APP
Server
APP
Server
APP
Server
APP
Server
APP
Server
APP
© 2015 Pythian - Presentation for Oracle OpenWorld
Data Partitioning
51
• By Data
• Each application server processes a subset of the data
• By Customer ID
• By Region – data balancing challenges
© 2015 Pythian - Presentation for Oracle OpenWorld52
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
CPU
RAM RAM
Server
APP
Server
APP
Server
APP
Server
APP
Server
APP
Server
APP
Load Balancer
© 2015 Pythian - Presentation for Oracle OpenWorld
Data Partitioning
53
• Partition the Database to match
• Consider growth – have significantly more buckets than servers
• Local indexes everywhere
• Partition key and Primary Key generation can be challenging
• If partitioned by primary key, key generation from one node, must not send data to multiple partitions
• Secondary unique indexes still challenging
© 2015 Pythian - Presentation for Oracle OpenWorld
Data Partitioning – Primary Keys
56
• Unique Keys
• Primary Keys usually part of partition key
• Two level partitioning (range for historical, hash for concurrency) – global partitioned indexes
© 2015 Pythian - Presentation for Oracle OpenWorld
Sharded Data – why use RAC?
57
• Most apps will “just work”
• Few components require extra attention
• Not all data is sharded
• Reference / Lookup Data is shared
• Low concurrency data does not need to be sharded
• Redundancy – can ‘fail back’ to any node
• Caching (performance) implications
© 2015 Pythian - Presentation for Oracle OpenWorld
Sharded Data – why use RAC?
58
• Some data cannot be sharded one way
• Many to Many
• Buyer/Seller with portals for both customers and stores
• Manual sharding mistakes – loose data
• Non shardable queries? Code manually?
• Sharded queries and sorting
• Schema Changes/Indexing
© 2015 Pythian - Presentation for Oracle OpenWorld
Reports Across all Data
60
• No replication
• Cross shard consistency – even after data restore
• No data movement
• Reuse cache from each node
• Parallel queries can leverage all nodes
• MPP on OLTP data
© 2015 Pythian - Presentation for Oracle OpenWorld
Final Thoughts
63
• RAC Works
• With a little help
• RAC Scales
• Your CPU
• Sometimes your RAM
• Not your disk
• RAC can be
• share everything
• share nothing