Finding and Fixing Production Issues

Dave Stokes@stoker

David.Stokes@oracle.com

Gorilla in cancer slide

Recently an image of a gorilla was 'added' to slides of cancer cells and reviewed by a group of radiologists.

Gorilla in cancer slide

Recently an image of a gorilla was 'added' to slides of cancer cells and reviewed by a group of radiologists.

The gorilla was recognized only by 17% of the radiologists!

The gorilla responsed

Who? Me worry?

● If you relied on the work of the radiologists, would you be concerned?

● Lets assume form here on your LAMP stack is tuned, cached, tweaked optimally.

Who? Me worry?

● If you relied on the work of the radiologists, would you be concerned?

Do not forget

The output from one woman pregnant for nines months is much different than that of nine women pregnant for one month – Werner Von Braun

Problem Solvers

What are your trying to maximize/minimize● What is the goal?● Work systemically● Optimization needs to be a continuous process

● Be able to back out changes!!!!● Really think

Who said?"What is a system? A system is a network of interdependent components that work together to try to accomplish the aim of the system. A system must have an aim. Without an aim, there is no system. The aim of the system must be clear to everyone in the system. The aim must include plans for the future. The aim is a value judgment. (We are of course talking here about a man-made system.)”

Must have an aim

W. Edwards Demming

● What is the 'aim' of your LAMP stack?

● Hopefully not 'ready, fire, aim'

Yogi Berra

If you don't know where you are going, you might wind up someplace else.

Define

● What are your 'norms'– Avg/Min/Max times

– End to end times or per section

– Reasonable variances

– Know your service levels

H. L. Mencken

For every complex problem there is an answer that is clear, simple, and wrong.

Costs

● What is the cost of a query? Time, resources, and/or $

● What is the cost of storage/ retrieval?

● Growth deltas

Kano Model or you are constantly slipping

Finding problems

● Outside of norms– Change

causes

– Regression

● Re-norm-ing– New and/or

improved

– New benchmark

Pipeline Pig● Pigging in the context of

pipelines refers to the practice of using pipeline inspection gauges or 'pigs' to perform various maintenance operations on a pipeline. This is done without stopping the flow of the product in the pipeline.

– How could you adapt this idea for your environment?

Audit

● Random● Every N

– Start to finish

– Section

Tools

● Testing suites– $$

– No P-N-P

● Monitoring tools

● Utilities, general system– Systat, iostat, ps, show user

MEM

Similar products

● Solarwinds, ManageEngine, MySQL Monitor Tool

● MySQL Workbench, Percona tool kit

Re-Reengineer

● Service tiers● Shard, shard, shard● Pre-cache, move cache forward● Scale vertically, diagonally● Pray Moore's law is really true!

Virtual Developer Day● March 12

– beginners and advanced tracks– Free and Online

– Hands-on labs

– https://oracle.6connex.com/portal/mysql/login/?langR=en_US&mcc=OTNblog

A&Q

● David.Stokes@oracle.com● @stoker● NorthTexasMySQL.org● Slideshare.net/davestokes