Tuning Tips for Linux & AIX to run a Database - UKUUG Linux and UNIX support and tuning, ... MySQL,...

v

UK UNIX User Group – Databases and the Web

© 2007 IBM Corporation & UKUUG

Tuning Tips for Linux & AIX to run a Database

Nigel GriffithsLinux on Power & VirtualisationTechnical Leader, IBM Europe

Tuning Tips - Databases


© 2007 IBM Corporation & UKUUG2

Abstract:►Sitting behind any reasonable size website is some sort of

database to hold the data involved. ►This can be used to control the web output, user records or

profiles or regular database contents which are made available or manipulated via web applications.

► It is key then for the database to perform well in its support of the user experience.

►This session looks at the common causes of database performance problems with the aim to make sure you don't fall into these common traps.

►This is practical advice from the presenter's wide experience of Linux and UNIX support and tuning, and not database vendor specific.“




Agenda1.Scene setting and Assumptions 2.Configuration and Change control3.Use the latest OS version4.Burn in the box5.Spotting single-threadedness6.Balanced Systems7.Waiting is not a problem8.Disks * four …9.Memory – Use it or lose it!10.The problem with paging11.Understand your virtualisation tools & effects12.Classic bad thinking13.Know Your tools and nmon (blatant advert)




Assumptions

Database is on a server (not a PC) Multiple CPUs, Many GB of RAM, Many 100’s GB of disks and Many Processes / Users

You have many machines to look after Datacentre

I am biased toward what I know and have used► I work for IBM (15 years plus 10 before that)► Big UNIX systems: RS6000 / pSeries / System p / … ► AIX & Linux on POWER then Linux on x86 – Benchmarking + Sizing► RDBMS Oracle, then some DB2, MySQL, Sybase, Informix► I will try to keep the IBM content to a minimum – but it is hard for me to do




Scene Setting

Movie




Scene Setting

From my personal experience (your mileage may vary) Performance problems reported to IBM

► 50% poor disk layout + mgmt some disks 90%+ busy, while 50% not used at all► 10% poor setup of RDBMS tuning parameters relating to memory use► 10% single threaded batch applications► 10% poorly written customer extensions to standard applications► 5% system running with errors in the [errpt] log file (including CPU failures!!)► 5% paging on large RAM (>2 GB) systems [vmtune not use to set min/maxperm]► 5% OS problems already discovered and fixed but OS was not up to date► 4% badly ported App = not compiled with optimization or on old OS version► 1% genuine bugs in OS, RDBMS or commands

every single one of these was reported as a problem with the hardware!!!




What is “Quality”?

CPU► good 70%► bad 85%► ugly 90%+ (except batch)

Disk► good <30%► bad <40%► ugly <50%+

Memory► good zero paging► bad 10 pages per second per CPU► ugly any more paging+

Network ► good <30% bandwidth► Hubs bad >70%► Switched bad is MB/s near Mb /10

Run Queue► good < 2 * CPU's




Configuration and Change Control

Script to document the whole config1. CPU – number, type and speed2. Memory – size3. Adapters – number & type4. Network - number & type5. Disks – number & type6. Disk layout & file systems layout7. OS version & tuning options8. RDBMS versions & tuning options9. App (s) version & tuning options

Off machine – you need this anyway for disaster recover – right!

Do you know when the machine. OS, App, RDBMS was last changed?► By whom and why?► Exactly what they did?

Save config & performance data Before Make the change Save config & performance data After Makes problem determination

10 times easier and faster




Use the latest OS version Linux ► For advanced kernel features use 2.6 kernel and the latest rev► 100’s of man hour gone into the OS performance tuning ● why not take it for free?

► In 2.6.16 alone:● gcc, glibc, libhugetlbfs, 64KB pages = scaling (SLES10, RHEL5)

► No excuse for the 2.4 kernel now

AIX6► Major improvements and Power6 support at 4.7GHz CPU► Mobility to move a CPU partition between machines while live► Workload Partitions, JFS2 encryption & snapshots, RBAC




Burn in the box Burn in – first used in mainframe

► Larger the machine the more parts and larger every year

Hardware failures happen► Most failures are in the first two weeks► Then fairly random► Some occasions after 1 2 years in small batches

Therefore smart to burn in test for 24 hours Before you add your data► CPUs – just run something on each of them► Memory – get it hot► Disks – get them all spinning► Adapters – get them all busy

After this Disk protection & HA will then catch the remaining failures




Single Threaded Applications

Still finding applications like this!

The IT Industry has been SMP for over a decade ► There are no excuses► Its just lazy programming and no longer acceptable

If you have no choice …get the fastest CPUs in the industry … POWER6for you single process




Balanced Systems

Meaning a sensible ratio of CPU Memory Disks Adapters

For example my rule of thumb for POWER6► 1 CPU at 4.7GHz► 8 GB Memory► 1 TB Disk on 10 disks► 1 FC SAN adapter► 1 GB Ethernet adapter

Decide your own ratios for your platform If you are too far away from balanced you are

“stuffed”!




Waiting is not a problem CPU Wait – worries many people CPU Wait = CPU in Idle state

► But a flag is set indicating outstanding Disk I/O

20 years ago indicated ► Disks are holding back the machine

Now indicates► CPUs process the data faster than the

disks can supply it► Many applications just work this way

● pulls in lots of disk blocks but does not do much processing on the data

► Like BI, SAP RDBMS, SQL table scans

A benchmarker sees this as a opportunity to start more work !!

Is this machine 80 or 60% Busy?

Answer: 60% Busy with 40% unused CPU




Disks – the biggest problem 50% of performance problems are Disk Related

Going to assume you are running disk protection of some sort► Software Mirror► Hardware Mirror► Hardware RAID5► Fibre-Channel SAN sub-system based protection► Even iSCSI or NFS

If not you deserve every problem you get!

Also don’t forget to protect OS and paging space► If you lose these … you panic!




Disks – uneven use

Very often find this … Some times this … But never this …

Lots of times: Half the disks are doing nothing!!




Disks - Layout Old School►Separate Data, Indexes & Logs disks (+separate OS, App, paging)►This worked well when we had few concurrent SQL statements►Problem is we now have 1000’s of “in flight” transactions►Generates what looks like random disk I/O● So zero disk optimisation is possible

►We end up with some hot & some cold disks

New School►Make every disk participate – the “hose it all around” approach►Zero hot spots using your hardware to do the work (not people)► If extreme logging and 30 disks+ ● May have Dedicated Log disks 4 is the minimum




Disks – Too many disks also an issue

How many LUN’s can you manage?► 10► 100► 500► 1000► 2000► 5000

IMHO Too many LUN’s is a big mistake, that we will have to correct in the next 10 years.




Disks - Missing adapter stats No UNIX that I know does adapter stats They add the underlying disks




Memory – Use it or lose it!

If you buy 8 GB of memory use 8GB of memory!► Free memory (above a few 10MB) is a waste of money

Free memory is a tuning opportunity AIX will use free RAM in the filesystem cache

► Grow the RDBMS disk cache for faster SQL with less Disk I/O► Or larger in memory sorts

Tests (a few years ago) showed in memory data runs 10 times faster




The problem with Paging UNIX folklore Paging – just don’t do it. With high numbers of user systems, they page all the time►Asked my mainframe colleagues …

We have to live with it► Just make sure you spread out paging ►On to lots and lots of disks - what the back half of large disks is for !!

Paging many types

/bin/ksh

Memory Paging spaceFilesystem

Process

Code (ro)

Data, heap, stack

MemoryMapped file Shared

memory

Libraries

File




Understand your virtualisation tools and effects Virtualisation► Like: Xen, VMware, Advanced POWER Virtualisation (APV)

If you are in these environments you have to understand them well Can get very misleading in performance numbers Slow down might be due to other workloads not your tuning Do your homework► Read the manuals, articles, books and websites► Take the education

Run experiments► Generate workload► Can your tools clear tell you what is going on?

In APV, we now monitor Physical CPU use (not Utilisation)




Classic Bad Thinking 1 Filesystems are as fast as raw disks RAID5 is fast (the clue is in the I = inexpensive) RAID5 write will not be too bad DB does not do writes Using new and untried features Misreading I/O wait CPU statistics (covered before) Simplistic test that prove nothing on larger machines Assuming the defaults will be fine JFS raw + not adjusting RDBMS buffer pool size Using the tool you know, rather than the right tool Random OS parameters changes and forgetting them




Classic Bad Thinking 2 Running more than one database on a machine► expecting the OS to work out which is more important

Working out transaction rate from a yearly total and assuming 365 * 8 even hours of flat workload► Especially not true in the web business► Also basing a transaction rate on marketing estimates

Incremental upgrades & changes allow the quickest diagnosis► Do not upgrade OS, database and the application all together► Do not change software and H/W (like disk layout)► Do not change software or DB parameters and data placement

Parallel databases are easy (and cheap)► More complex need balancing more training, experience & management




Know your tools Regular UNIX/Linux Tuning tools

vmstat iostat, top (Linux) or topas (AIX)

Do your home work read those manuals► I regularly have to tell people what the columns really mean!

Not so good for larger machines = 100’s of disks or processes etc. Do get boring after an hour We need something better and graphs (not numbers)




nmon basics Freely available Performance monitor for

► AIX 5 & 6► AIX 4 (via older nmon version) ► Linux (POWER, x86 and mainframe)

Near zero installation Two modes

► Online see ► Save data to file then either:● nmon2rrd & rrdtool for .gifs/website● nmon Analyser Excel spreadsheet

– From Stephen Atkins– IBM, UK.– Requires Excel 2000– Very good chap.




Know your Profile (processes, CPU, RAM & Disk) Nmon Analyser default graphs Then you can understand when its wrong




nmon

Screennmonfile

0

50

100

150

200

Excel graphs

Stephen’s nmonAnalysernmon2rrd

C filter

Java Dynamic Graphs

rrdtoolopen source

Scripts - Create rrd - Load rrd - Graph rrd index.html

-f or –F optionnmon

Federico’spGraph

Website +.jpg graphs

Bruce’snmon2web

perl

rrdtoolopen source

Scriptsrrdtool scripts+ CPU & RAM aggregation

Stephen’snmon

Consolidator

0

50

100

150

200




nmon References and Download

Forum Performance Tools Forum

Mostly about nmon For questions & answers

Wiki AIX 5L Wiki “Other Performance Tools” nmon

Download FAQ Manual and lots more

http://www.ibm.com/systems/p/community/




Whitepaper: Oracle 10g Release 2 Performance and Tuning on an IBM System p™ Linux on POWER Platform► Chij-Mehn Chang, Ric Hendrickson, Michel Riviere, Arun Sar► June 30, 2007

IBM Redbook: Tuning Linux OS on System p► SG24-7338

Questions

Other References

Date post:	18-May-2018
Category:	Documents
Upload:	doanthien
View:	224 times
Download:	1 times

Tuning Tips for Linux & AIX to run a Database - UKUUG Linux and UNIX support and tuning, ... MySQL,...

Documents