Why Defragmenting Your Indexes Isn't Helping

Why Defragmenting Your Indexes Isn’t Helping

99-05: dev, architect, DBA

05-08: DBA, VM, SAN admin

08-10: MCM, Quest Software

Since: consulting DBA

www.BrentOzar.com

[email protected]

Free conference: GroupBy.org

sp_Blitz: FirstResponderKit.org

SQLServerUpdates.com

PasteThePlan.com

Github.com/BrentOzar

Personal blog: Ozar.me

You’re here because

performance sucks.

Success means

a performance boost

that end users notice.

The two kinds of fragmentation

Why the fix makes things worse

The better performance fix

When people insist on rebuilds

What to do in overnight windows

This deck: BrentOzar.com/go/defrag

“Fragmentation sucks.”

Page Header

Index OR

Data Rows

Slot Array

8KBData is stored

in 8KB pages.

Hall - Loggins Loggins -

MessinaMessina -

Oates

In a freshly rebuilt index, the 8K pages are full, and let’s say

they’re all in order on disk.

Location 4096 Location 4097 Location 4098

Hall - Loggins

He needs to go

here, but there’s

no empty space.

Loggins -

MessinaMessina -

Oates

McDonald, Michael moves to our town.

Split! New

Location 4096 Location 4097 Location 9144 Location 4098

Page split!SQL allocates another

page.

But it won’t be in

physical order.

Split! New

Location 4096 Location 4097 Location 9144 Location 4098

There are two kinds of fragmentation here.

Monitoring this stuff

To track page splits:

Perfmon counter SQL Server:Access Methods – Page Splits/sec

To track external fragmentation:

sys.dm_db_index_physical_stats – avg_fragmentation_in_percent

To track internal fragmentation:

sys.dm_db_index_physical_stats – avg_page_space_used_in_percent

Great! I’ll monitor for

page splits.

Create an empty table

CREATE TABLE TheHeart

(ID INT IDENTITY(1,1)

PRIMARY KEY CLUSTERED,

Groove VARCHAR(100));

GO

Check your page splits

SELECT * FROM

sys.dm_os_performance_counters

WHERE counter_name LIKE '%Splits%’;

Insert just one row

INSERT INTO dbo.TheHeart

(Groove)

VALUES ('The chills that you spill

up my back keep me filled');

GO

Check your page splits again

SELECT * FROM

sys.dm_os_performance_counters

WHERE counter_name LIKE '%Splits%’;

Adding a

new page

is also called

a page split.

Monitoring this stuff

To track page splits:

Perfmon counter SQL Server:Access Methods – Page Splits/sec

There are other more accurate ways to monitor page splits, but if this lying thing came as a

surprise to you, buckle up. It’s about to get worse.

To track external fragmentation:

sys.dm_db_index_physical_stats – avg_fragmentation_in_percent

To track internal fragmentation:

sys.dm_db_index_physical_stats – avg_page_space_used_in_percent

Is external fragmentation bad?

Pop quiz: if your pages are out of order

Will your maintenance tasks take longer?

(Backups, index & stats maintenance, CHECKDB)

Will queries reading from RAM be slower?

Will queries reading from disk be slower?

• What if you’re seeking?

• What if you’re scanning?

How do you fix it?








How do you fix it?

Systems Performance by Brendan Gregg










How do you fix it?

Fixing out-of-order pages

Reorganize Rebuild

What does it do? Shuffles individual pages around,

around, one at a time

Builds a whole new

copy of the index

Are all pages in physical order when

when we’re done?

No Maybe, but only if you use MAXDOP

MAXDOP 1

Can it be done online? Yes Enterprise Edition only

Can it be interrupted? Yes, and your

progress is kept

No

Is it a logged operation? Yes Yes

Does it slow down mirrors, AGs, log

AGs, log shipping?

Yes Yes

Users hate your

“fix.”








How do you fix it?

Wait! I’ll set FILL

FACTOR!

Data

Hall - Loggins

He needs to go

here, but there’s

no empty space.

Loggins -

MessinaMessina -

Oates

McDonald, Michael moves to our town.

Fill factor leaves empty space on the page.

Default set at the server level: 100% (and 0 means the same thing)

So the idea is:

• Set fill factor to something lower (like 70-80)

• Leave empty space on every page

• When new records are added, we’ll have space

so we don’t need to split pages

Let’s see a couple examples.

Data

dbo.Users

Clustered

index on ID

Users are ordered by ID, an

identity field

Do you want empty space

on this page?

• Inserts?

• Updates?

• Deletes?

What’s the right fill factor?

dbo.Usersnon-clustered on

LastAccessDate

Users are ordered by the last date they visited Stack Overflow

Do you want empty space on this page?

• Inserts?

• Updates?

• Deletes?

What’s the right fill factor?

The default fill factor of

100% keeps your database

small.

When you set fill factor <>

100%, guess what you’re really

doing.

When you set fill factor <>

100%, guess what you’re really

doing.

“Does that make me

a data scientist?”

Internal fragmentation

That’s right: setting fil l factor <> 100% actually causes …

Pop quiz: if your pages have empty space







How do you fix it?Data

Data

Pop quiz: if your pages have empty space







THE PROBLEM IS US.External fragmentation doesn’t really matter

But to “fix” it, we’ve been:

• Rebuilding/reorging our indexes daily,

which actually makes some things worse

• Setting fill factor, which

causes internal fragmentation

And internal fragmentation DOES matter

(and we’ve been the ones causing it all along!)

SQL Server needs a dog.







I’m so confused.

The short answer

1. Set fill factor at the server level back to 100%.

2. Use sp_Blitz and sp_BlitzIndex to check for index fill factors <80%, and set those

back up to 80% or higher (or 100%).

3. Rebuild your indexes once to atone for those past sins,

getting them back up to 100% fill factor.

4. Monitor the right metric,

and use the scientific method to improve it.

How SQL Server Schedules CPU

What’s Running Now What’s Waiting (Queue)


What’s Running Now

SELECT * FROM

dbo.Restaurants

(By Brent)

What’s Waiting (Queue)



SELECT * FROM

dbo.Restaurants

(By Brent)


SELECT * FROM dbo.Tattoos

(By Jeremiah)

SELECT * FROM dbo.Rabbits

(By Kendra)



SELECT * FROM

dbo.Restaurants

(By Brent)



(By Jeremiah)


(By Kendra)


What’s Running Now What’s Waiting (Queue)

SELECT * FROM dbo.Tattoos(By Jeremiah)

SELECT * FROM dbo.Rabbits(By Kendra)

SELECT * FROM dbo.Restaurants(By Brent)

SQL Server Waits For:

Resources:

CPU, memory, storage, network, latches, locks

Stuff outside of SQL Server (Preemptive):

COM, OLEDB, CLR

System Tasks:

Lazywriter, trace, full text search

Is this SQL Server working hard?


SELECT * FROM

dbo.Restaurants

(By Brent)


What about now?


SELECT * FROM

dbo.Restaurants

(By Brent)



(By Jeremiah)


(By Kendra)

Or now?


SELECT * FROM

dbo.Restaurants

(By Brent)


SELECT * FROM dbo.Tattoos(By Jeremiah)

SELECT * FROM dbo.Rabbits(By Kendra)

EXEC sp_WhoIsActive(By Adam Machanic)

BACKUP DATABASE SQLbits(By Ola Hallengren)

DBCC CHECKDB(By Paul Randal)

Some queries are simple

Some queries have a lot going on

These were simple little queries.


SELECT * FROM

dbo.Restaurants

(By Brent)



(By Jeremiah)


(By Kendra)

But one query looks more like this:


Index scan of Table1





So for each core, you’ll likely see:


One task running,

using CPU


Several

(or several DOZEN) tasks

stacked up waiting on stuff

If we do this for one second,

how many seconds of waits?


One task running,

using CPU


15 tasks waiting on storage

3 tasks waiting on locks

1 task waiting on CPU

Hours of wait time per hour

0

5

10

15

20

25

30

6:00 7:00 8:00 9:00 10:00 11:00

CPU

Locks

Storage

How hard is your server working?

Dynamic Management View (DMV) sys.dm_os_wait_stats

Tracked cumulatively over time

Trend on an hourly basis and break it out by:•Weekday vs weekend

•Business hours vs after hours

•Maintenance windows (backups, DB maintenance)

When Wait Time changes

This metric is linked to Batch Requests:the more work the server is asked to do, the more it waits.

Going up? Batch Requests could be going up, or the server is responding slower to the existing workload.

• Storage got slower, indexes were dropped, queries were tuned poorly, other processes are running on the server

Going down? Batch Requests could be too, or server is responding faster to the workload.

• Indexes or queries were tuned, more memory was added

Wait Time per core per hour

Near zero – Your server isn’t doing anything.

(Although that doesn’t mean every query is fast.)

1 hour of waits – You’re still not doing much.

Multiple hours per core – now we’re working!

And we should probably be tuning.

Free script: sp_BlitzFirst

Totally free open source diagnostic tool.

Installs in the master by default, but can go anywhere.

Runs in 5 seconds ideally, but can be more under load.

By default, shows waits for a 5-second sample now.

@SinceStartup = 1 shows waits since, uh, startup.

Typical output under load

Most useful parameters

@SinceStartup = 1

@ExpertMode = 1

@Seconds = 60

@OutputDatabaseName = ‘DBAtools’,@OutputSchemaName = ‘dbo’,@OutputTableName = ‘BlitzFirstResults’

Solve the emergencies,

then focus on waits, in order.

Some waits are like poison:

any occurrences of them have to be fixed first.

After that, work through the list.

You’ll never make them go away,

but you have to focus on them in order.

Wait stats decoder ring

Wait type Means sp_BlitzCache

@SortOrder =

CXPACKET Parallelism Reads, CPU

LCK* Blocking locks Duration, reads

PAGEIOLATCH Reading data from data files Reads

SOS_SCHEDULER_YIELD Waiting for CPU time (via CPU

schedulers)

CPU

RESOURCE_SEMAPHORE Query needs memory in order to

start

Memory grant







What if someone

wants to play doctor?

You’re on vacation when suddenly….

Performance is

bad and Ted says

it’s because you

don’t have an

index defrag job

The devs won’t

look at any code

till we add that job

and run it, is that

OK?

How to react

“The problem is high fragmentation.”

“So you guarantee that

rebuilding the indexes will fix it?”

“Uhhhh…”

“What metric shall we monitor to

know that the rebuild fixed it?”

“Ummmm…query runtime?”

“Great. Give me a query, and the current average

runtime, and what you expect it’ll be afterwards.”







So what do I do daily?

Lemme show you something awful.

Step 1: create database, back it up.

CREATE DATABASE BrandNewCar;

GO

BACKUP DATABASE BrandNewCar TO

DISK='BrandNewCar.bak';

GO

BACKUP LOG BrandNewCar TO

DISK='BrandNewCar_Log1.trn';

Step 2: put some data in it.

USE BrandNewCar;

GO

CREATE TABLE dbo.Options

(OptionName VARCHAR(50));

INSERT INTO dbo.Options

VALUES ('Leather seats'),

('Sunroof'),

('Racing stripes');

Step 3: back up our valuable car.

BACKUP LOG BrandNewCar TO

DISK='BrandNewCar_Log2.trn';GO

Step 4: look inside your car.

Shut down SQL Server (or set your database offline)

Open XVI32 – free hex editor

Open your database file

Do a control-F for Sunroof

Nice data you’ve got there.

Sure would be a shame if something happened to it.

What causes corruption?

Shared storage failures and bugs

What causes corruption?

Microsoft SQL Server bugs

Step 5: let’s cause corruption.

Change Leather Seats to something else

Save the file and close XVI32

Start up your SQL Server (or bring database online)

SQL Server looks the other way

SQL Server doesn’t detect the damage on startup because it doesn’t read every page from every

table.

Ways to discover the damage:

• SELECT * FROM dbo.BrandNewCar;

• DBCC CHECKDB();

• ALTER TABLE dbo.Options REBUILD;

Now, how do you fix this?

If this was our real production server,

we’d be taking our application down,

and restoring from backup.

In real life: BrentOzar.com/go/corruption

We get corruption sometime on Sunday

Full backup Sunday

12 am

(have this file)Data file corruption

at 10 am to one

clustered index

Log

backup

Log

backup

Log

backup

Log

backup

Successful

CHECKDB 10

pm

Corruption is detected – but not acted on

Full backup Sunday

12 am


at 10 am to one

clustered index

Full backup Monday

12 am (have this file)

Log

backup

Log

backup

Log

backup

Log

backupLog

backup

Log

backup

Successful

CHECKDB 10

pmPage is read,

CHECKSUM doesn’t

match

We only keep 24 hours of log backup files

Full backup Sunday

12 am

(have this file)

Full backup

Tuesday 12 am


at 10 am to one

clustered index

Full backup Monday


Log

backup

Log

backup

Log

backup

Log

backupLog

backup

Log

backup

Log

backup

Log

backup

Deleted log backup files

Successful

CHECKDB 10

pmPage is read,

CHECKSUM doesn’t

match

Log

backup

Will you lose data? How much?

Full backup Sunday

12 am

(have this file)

Full backup

Tuesday 12 am


at 10 am to one

clustered index

Full backup Monday


Log

backup

Log

backup

Log

backup

Log

backupLog

backup

Log

backup

Log

backup

Log

backup


Successful

CHECKDB 10

pmPage is read,

CHECKSUM doesn’t

match

Log

backup

User

finally

calls in

What could you do to lose less data?

Full backup Sunday

12 am

(have this file)

Full backup

Tuesday 12 am


at 10 am to one

clustered index

Full backup Monday


Log

backup

Log

backup

Log

backup

Log

backupLog

backup

Log

backup

Log

backup

Log

backup


Successful

CHECKDB 10

pmPage is read,

CHECKSUM doesn’t

match

Log

backup

User

finally

calls in

Ways to lose less data

Set up alerts for corruption errors: https://BrentOzar.com/go/alerts

Keep transaction log backup files longer

(dictated by your RPO, RTO, and CHECKDB frequency)

Run CHECKDB more often (and act fast on the outputs)

The more of these you do, the better your chances are to keep your job.

https://brentozar.com/go/alerts

DBAs don’t get fired

for slow performance.

DBAs get fired

for losing data.

Run CHECKDB as frequently as practical.

Forget shuffling indexes around every night.

It’s time to run CHECKDB in that time window instead.

Recap

Yay, it’s over!

What you learned

External fragmentation = pages out of order on disk(Only matters when you’re reading from disk)

Internal fragmentation = empty space on pages(Matters a lot, and fix this with judicious rebuilds)

Fill factor = internal fragmentation(Set this, and you’re making internal fragmentation worse)

Index maintenance is fine,

but it doesn’t solve most wait types.Find your top wait type with sp_BlitzFirst and focus on it.

What you’ll do tomorrow

1. Set fill factor at the server level back to 100%.

2. Use sp_Blitz and sp_BlitzIndex to check for index fill factors <80%, and set those

back up to 80% or higher (or 100%).

3. Rebuild your indexes once to atone for those past sins,

getting them back up to 100% fill factor.

4. Monitor wait time per hour,

and use the scientific method to improve it.

This deck & demos: BrentOzar.com/go/defrag

And stop reading advice

from the SQL Server 7.0 guys.

Date post:	22-Jan-2018
Category:	Technology
Upload:	brent-ozar
View:	2,922 times
Download:	3 times

Why Defragmenting Your Indexes Isn't Helping

Technology