1

Did you all get a chance to read that? As a public company we need to have Vista is the only solution built from the ground up on true enterprise

2

our disclosure statement before all presentations. If you have any questions

on what it means please speak with our General Counsel.

Vista is the only solution built from the ground up on true enterprise

technology --- allowing you to ensure that you continue to provide your

faculty and students an outstanding experience

3

4

5

6

7

Search Google and you will be amazed how little meaningful information you get by the words “performance forensics” in the context of computers and

8

get by the words “performance forensics” in the context of computers and software. One paper by Bob Sneed from Sun Microsystems (http://www.sun.com/blueprints/1203/817-4444.pdf) is out there, but very little else.

So you will have to trust me in my primitive definition of performance forensics. You might even offer to help make it better.

Performance forensics is like any other forensics process. It begins with collective evidence. If you are lucky and have a lot of tools in place you will have a starting point of data to sift through. More often then not, the data is not there. You are not always lucky to have the data when you need it and/or it might not be in the best format for getting to the root cause of a problem.

Evidence as we will discuss later can be collected after the fact. Techniques such as discrete simulation can be used to re-enact an incident. When that does happen, you have the ability to capture all of the data you want. You simply need to know what data to collect. It’s a circuitous loop of sorts…mainly because you might not know what data to collect to begin with.

It’s like when I look under the hood of my car. I have no idea what I’m looking at…Maybe it’s that smoking gun I’m in search of. Yeah, I guess if I see some kind of corrosive, smoke or leak it might be painfully obvious…but it never is. Not with today’s cars…Computers and software specifically are the same. Rarely is there that smoking gun sitting in front of your face waiting to be found. Thus evidence is critically important to the process. Interviewing is a big part of evidence gathering. It’s not a separate activity. As I will discuss interviewing is an art. You have to be able to assemble questions that will return meaningful answers. Equally, you have to be able to avoid diagnosis bias and value attribution that are often part of human nature.

Source: http://www.flickr.com/photos/turkguy19/1018419391/

9

http://www.flickr.com/photos/wwarby/3297205226/

10

11

Source: http://www.flickr.com/photos/t_squared/152270386/

12

http://www.flickr.com/photos/7563125@N08/2830710184/

13

14

Source: http://www.flickr.com/photos/turkguy19/1018419787/

15

Source: http://farm2.static.flickr.com/1330/3174009125_ec49351a6d_m.jpg

16

Source: http://www.flickr.com/photos/ale2000/1275120868/

17

18

19

20

21

Source: http://farm4.static.flickr.com/3396/3507282396_3756634f01_m.jpg

22

Source: http://www.flickr.com/photos/psilver/412264230/

23

http://crackerjackonlinemarketing.com/blog/wp-

content/uploads/2008/07/working-woman-with-octopus-hands.jpg

24

Source: http://www.flickr.com/photos/nickbush/450151862/

25

Problems are not always easily identifiable. When I say that I feel off or sick, I leave the listener desiring more information. They might infer that I have a stomach pain, a cold or a

26

listener desiring more information. They might infer that I have a stomach pain, a cold or a headache. It could be that I am tired or I have a broken arm. A more related example that I often hear is that my system is slow. What defines slow? Can you show me? Can I experience the slowness?

Is it always slow every single day and every minute? Are all of the components that make up the physical architecture necessarily slow? Are particular use cases experiencing latency? Do they always experience latency or is it at specific times? Is it specific users who experience latency? Are the users different is some kind of fashion? Does the problem happen after a particular interaction pattern? Does it happen with a particular piece of data?

When a problem is easily identifiable, define a clear, intelligible problem statement. The problem statement is used to aid the investigation so the forensics process can focus on collecting meaningful data to get to root cause analysis.

Narrowing down to a problem statement from the unknown can be an exhaustive effort. Start with questioning (not formal interviewing) in which your goal is to exclusively narrow down the chasm of possibilities. Start with the “Lassie Question: Can you show me?” Experiencing the problem first hand provides basic context. If the problem can’t be reproduced, try to provide supporting clues so that the unpredictable can become more predictable. You can’t necessarily replicate the performance problem at will. Do you have supporting data about your experience? Can you explain what happened to you? Do you know when it happened (smallest time window)? Has it happened before? If so when? Try to get down to the exact minute if possible. Has it happened to anyone else? What were they doing? Did it happen to them at the same time as you?

It comes off like you are asking dozens and dozens of questions, but in reality you are not. You are gathering basic context: Who, What, Where and When.

Be unwilling to announce a problem statement until you have confidence in the development of the problem statement (not the cause of the issue). Remember we are not diagnosing, we are simply collecting and announcing symptoms.

27

I’m not the creator of this methodology. I’m quite sure that others who are far more knowledgeable on the subject would tell you I’m possibly missing a step or that I am drawing out the process too far. A picture is truly worth a thousand words.

28

would tell you I’m possibly missing a step or that I am drawing out the process too far. A picture is truly worth a thousand words.

I will breakdown each element of the methodology in subsequent slides. I’ve designed a circular visualization for the obvious conclusion that I’ve come to over the years in which the process must revolve in order to come to root cause analysis.

Performance forensics doesn’t necessarily begin with evidence collection. Rather, it potentially begins long before an incident occurs. Let’s take an abstract example such as a person complains about chest pain. The person tells their spouse that at times they have unbearable pains, but eventually it goes away. It doesn’t happen enough and the pain isn’t so severe that it’s worth the time or the effort to go to the doctor. The process of convincing yourself that the symptoms you are experiencing is not what you really have is called diagnosis bias. I will talk about this in greater detail later.

This pain might go on and on for quite some time until it progresses. Analysis could be initiated at any point. More often then not, the complaints go unrealized and forensics is placed on hold. It comes back later on. The question is when. Typically when a terrible even occurs. It could be a heart attack or sadly a loss of life. The forensic engineer is tasked with tracing back why it happened, was foul play suspected and could it have been avoided.

I propose that at any time the methodology can be initiated. No major issue has to occur for performance forensics to begin. Symptoms do not necessarily have to show-up for the process to begin. You can call this what you want, but basically the collection of evidence, interviewing, modeling/visualizing and planning for the future is most commonly referred to as capacity planning. It’s not the much different from what we are trying to accomplish with performance forensics. The key difference is proactive behavior versus reactive behavior.

The methodology begins with the collection of data. We can call this data evidence. Evidence is collected in two ways: intended data collection and simulated data collection. When data is not available, we often go through the process of putting data collectors in place. The thought behind this is that if something happened once, it’s bound to happen again.

Interviewing is incorporated into the methodology. I will discuss techniques for interviewing. Understand that when humans are involved and asked to participate, you run the greatest chance for diagnosis bias and value attribution (two topics I will present in greater detail).

Next I will discuss why modeling and visualizing a problem can be critical at getting to the root cause of a performance issue.

29

30

http://www.stevesouders.com/blog/2009/06/30/firefox-35-at-the-top/

31

http://assets.en.oreilly.com/1/event/29/The%20User%20and%20Business%20Impact%20of%20Server%20Delays,%20Additional%20Bytes,%20and%20HTTP%20Chunking%20in%20Web%20Search%20Presentation.pptx

Source: http://www.flickr.com/photos/kaptainkobold/83359336/

32

http://www.phpied.com/image-optimization-7-mistakes/

33

Using pngrewrite to optimize this image

(http://entropymine.com/jason/pngrewrite/)

Cost Savings: 3KB or roughly 15%

34

Using optipng to optimize this image

(http://sourceforge.net/projects/optipng/)

Cost Savings: 4MB or roughly 47.50%

35

http://code.google.com/speed/page-speed/

36

http://developer.yahoo.com/yslow/

http://www.fiddler2.com/Fiddler2/version.asp

Great add-on to Fiddler2 is neXpert

http://videos.visitmix.com/MIX09/T53F

Great presentation highlighting differences between tools: http://assets.en.oreilly.com/1/event/29/Website%20Performance%20Analysi

37

http://assets.en.oreilly.com/1/event/29/Website%20Performance%20Analysis%20Presentation.ppt

38

39

40

41

42

43

44

Source: http://www.flickr.com/photos/pollyann/2877940383/

45

46

http://www.fiddler2.com/Fiddler2/version.asp

47

Demo videos: http://www.fiddler2.com/Fiddler/help/video/default.asp

Example of use as a reverse proxy: http://blogs.msdn.com/nexpert/archive/2009/06/04/capturing-http-with-fiddler-as-a-reverse-proxy.aspx

WebCast of neXpert: http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?EventI

48

http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?EventID=1032398774&EventCategory=5&culture=en-US&CountryCode=US

49

50

51

Can also consider using Microsoft VRTA: http://www.microsoft.com/downloads/details.aspx?FamilyID=119f3477-dced-

52

http://www.microsoft.com/downloads/details.aspx?FamilyID=119f3477-dced-41e3-a0e7-d8b5cae893a3&displaylang=en

53

54

55

56

57

JConsole on Steroids

58

Great presentation: http://www.javapassion.com/javase/VisualVM.pdf

Another Great Presentation: http://weblogs.java.net/blog/mandychung/archive/VisualVM-BOF-2007.pdf

59

60

61

62

63

64

65

66