A TRUE CAPACITY PLAN IS A PROCESSNOT AN EVENT

Joe BellJoe Bell

Senior Staff Regional Systems EngineerSenior Staff Regional Systems Engineer

Fujitsu Technology Solutions Inc.Fujitsu Technology Solutions Inc.

joe_bell@ftsi.fujitsu.comjoe_bell@ftsi.fujitsu.com

816-509-4084816-509-4084

04/29/03

2

Agenda

A brief history of Capacity Planning ProcessesA brief history of Capacity Planning Processes– Mainframe days

– “Open Systems” beginnings

– What’s changed?

– Why do we need Capacity Planning?

Basic Capacity Plan Process Components Basic Capacity Plan Process Components – Workload Analysis

– Forecasting

– Construct base plan and options

– Revision and Technology Inputs

– Procurement Analysis

– Reporting

– Refinement

3

Agenda -continued

Erector Set TheoryErector Set Theory– Planning

– Production

– Verification

– Implementation

– Maintenance

– Continuous Process Improvement

A Complete Process -top downA Complete Process -top down A Shortcut ProcessA Shortcut Process -bottom up -bottom up SummarySummary ReferencesReferences

4

A brief history of Capacity Planning Processes

Mainframe daysMainframe days– The Five Stages of Capacity Planning - Pat Artis

• Stage 1 - Vendor control– Reporting Level -> Corporate Decision Makers (CIOs, Directors)

– Written Documentation/Plans -> Marketing Proposals

– Relation to Corporate Budget -> Synchronized to budget cycle for hardware line items

– End User Considerations -> Little to none

– Continuity -> Long Term Vendor Marketing Plan

– Tools -> Rules of Thumb and Vendor Aids/Tools (proprietary)

• Stage 2 - Special Studies– Written Documentation/Plans -> Three to Five year Plans

– End User Considerations -> Initial Discussions, business overviews

– Continuity -> None - One Time Study

– Tools -> Rules of Thumb, Vendor Aids/Tools, Ad Hoc (build your own)

5

A brief history of Capacity Planning Processes -continued

Mainframe days -continuedMainframe days -continued– The Five Stages of Capacity Planning - Pat Artis

• Stage 3 - Technician– Reporting Level -> First Level Technical Supervisors

– Written Documentation/Plans -> Reams of detailed technical information, graphics, ..

– Relation to Corporate Budget -> Typically none

– End User Considerations -> Ineffective surveys or none

– Continuity -> Ongoing effort

– Tools -> Extensive tool development and acquisition

• Stage 4 - Organizational Development– Reporting Level -> Technical Services / Operations

– Written Documentation/Plans -> One/ two year plans distributed to management levels

– Relation to Corporate Budget -> Timed to serve the budget cycle

– End User Considerations -> Basic business requirements understanding

– Continuity -> SLRs created & maintained, long range strategies developed

– Tools -> Continued refinement

• Stage 5 - Mature– Reporting Level -> CIO

– Relation to Corporate Budget -> Current and future budget inputs

– End User Considerations -> Detailed business requirements understanding

6

A brief history of Capacity Planning Processes -continued

““Open Systems” beginningsOpen Systems” beginnings– Disjoint acquisition by end users and applications

• No single point of budget management

– Plethora of platforms for UNIX(es) and even NT

– Hardware cheap, just add more to solve CP and PM

– Learning new technologies and network techniques

– “Life Blood” applications generally still on the mainframe

– If any CP, then Stage 1, 2 or 3

7

A brief history of Capacity Planning Processes -continued

What’s changed?What’s changed?– Movement of disjoint departmental systems to the Glass House

• Server Consolidations for platform and management efficiencies

– Systems becoming more reliable and dependable• A handful of sophisticated high reliability, high performance Unix

platforms (HPUX, AIX, Solaris,..)

• Win 2000 and XP handling larger sophisticated applications

• LINUX !!

– Not safe to just throw resources at capacity shortages• Application, network and storage subsystem complexities

• Many examples of failures, just ask SPEs

• Just ask some .bombs and surviving .coms

8

A brief history of Capacity Planning Processes -continued

What’s changed?What’s changed?– Application workloads growing at higher rates on “open” platforms

– Movement of “Life Blood” applications to “open platforms”• Mainframes still do an excellent job, but at what cost?

• Most vendor R&D directed to “open platforms” and supporting software

– Hey, that hardware isn’t so cheap anymore!• Same for the software

• LINUX current exception

9

A brief history of Capacity Planning Processes -continued

Why do we need Capacity Planning?Why do we need Capacity Planning?– You can’t control what you can’t measure

• You can’t manage what you can’t control– Technology directions

• Can Contribute to Planned strategy versus chaos

– Budget expenditures

• It’s easier to see where you’re going if you know where you’ve been

• Instrumentation for critical management decisions

– Couple IT resources with Business Requirements• Better support of the corporate business environment

• Just in time resource control for reduced expenditures

– It’s Addressed in META Group’s “Best Practices

for Monitoring Web Applications”

– Improve the Total Cost of Ownership (TCO) Competitive Pricing

Well configured and balanced systems for top Performance

Fewer systems/Processors for simplified operations and Maintenance

Wahrehouse Applications

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10

Month of the Year

Pro

cess

ors NT Engines

Solaris Engines

10

Basic Capacity Plan Process Components

Construct a Plan and Work the PlanConstruct a Plan and Work the Plan– Mission and Objectives

• Resource types (processor, memory, Disk, IO, Network)• Long Term, Just-in-time• Types of Servers • User Community, Applications

– Define Functions and Activities• Reporting• Forecasting granularity• SLAs

– Determine Required Resources and Tools• People• Tool analysis and proposed acquisition• budget impact

– Construct Project Plan• Define Deliverables• Define Schedules

– OBTAIN MANAGEMENT BUY-IN & APPROVAL

11

Basic Capacity Plan Process Components-continued Workload AnalysisWorkload Analysis

– Define & Analyze current environment resource requirements• Processors, memory, Disk storage, Network bandwidth

– System OS, Database(s), subsystems

– Major application(s) (80/20 usually works)

• Types (OLTP, Background/batch, web enabled)

• SLAs

• Concurrent users

• Business drivers if available

– Performance Profile Requirements

• Peak versus Average utilization and ratios

• response time requirements for servers, storage, network and users

– Construct basic models

• Statistical correlation and prediction

• Analytical, simulation

– Can Workload Types be Balanced (CPU, memory , IO , network)

– Identify what’s important to the customer/user

• Availability

• Response Time

• Cost

• Scalability

12

Basic Capacity Plan Process Components-continued

Conduct Forecasting ActivitiesConduct Forecasting Activities– Historical analysis where appropriate

• Required for time series regressions

• Good to do anyway for information on historical activity

– Gather inputs from application development activities• New applications

• New application functionality

• They probably won’t have a feel for sizing, try similar and box car

– Get with Business Units that depend upon IT resources• A well run business activity will have it’s own forecasts

• There may be units that will correlate to application resources

– Interview Upper Level Management for other information

– Utilize statistical or/and Analytical models • Use Statistics for future based on history

• Use analytic for future based on current plus new event approximation

13

Basic Capacity Plan Process Components-continued Construct a basic Capacity Plan Construct a basic Capacity Plan

– The Way Things Are• History and current

• Annotated Graphs for events

– The Way Things Will Be• Forecasts by application/server

• Annotated Graphs with events

• Requirements on resources due to business and application changes

– Options Analysis• By resource you’re covering (server, memory, IO, Disk, Network)

• By vendor/architecture within resource

• Use those that satisfy all business requirements, stds.

– Create an Executive Summary with all important issues

– Include a pricing section if you have any cost data• Some organizations will keep this separate

• Some will require option pricing in the Capacity Plan

14

Basic Capacity Plan Process Components-continued

Review Plan with regard to Infrastructure and TechnologyReview Plan with regard to Infrastructure and Technology– System, Subsystem and Network Plans

• New component resource deltas

• Additional overheads from functional enhancements

– Adjust CP options based on Technology or Architecture Plan • New OS type(s), versions

• Three tier web enabled apps.

• ISCSI, NAS, SAN

• Gigabit network growth

– Include Disaster Recovery Requirements

Asset Management Review and Option Pricing AdjustmentsAsset Management Review and Option Pricing Adjustments– If no Asset or Procurement Dept for this then get help from

Financial Officer

– Include summary in Executive Overview

15

Basic Capacity Plan Process Components-continued

Finish Plan and make reports/presentationsFinish Plan and make reports/presentations– Present to all possible levels of management

• Tech management

• Application management

• business management

• Director IT or CIO for final approval– May have to go back and revisit options

– May have to perform budget cut activities against the plan

– Remember, “work the plan..”, because it’s not made out of concrete

– Stay flexible and keep emotion out of the final result.

Capture data for variances against your planCapture data for variances against your plan– Use root cause analysis for improvements to next plan

– 80/20 or 90/10, because little ROI on final percentage improvements

Time to begin again..continuous cycleTime to begin again..continuous cycle

16

Erector Set Theory

A Manufacturing Approach to CPA Manufacturing Approach to CP– Planning

• customer requirements, scope and content, schedules, critical success factors, planning aids

– Construction• Gather raw materials, create components, test individual components, distribute

analysis to concerned parties, combine components

– Verification• Component testability, discrete and measurable assumptions, coherency test,

rework time

– Implementation• Marketing and distribution for customer acceptance

– Maintenance• Expect it, planned enhancements, monthly reporting (forecast to actual), periodic

revisions

– Continuous Process Improvement -zero defect goal- process verification

17

Erector Set Theory -continued Underlying Tools and MethodologiesUnderlying Tools and Methodologies

– Benchmarking

– Customer Interviews

– Basic Regression Analysis

– Advanced Regression Analysis

– Queueing Theory and Analytic Models

– Financial Analysis

18

A Complete Process -top down

Capacity Plan ProcessCapacity Plan Process – Develop Master Schedule – Construct Base Plan

• Collect Historical CPU Usage • Collect Historical Memory Usage • Collect Historical DASD Usage • Build Baseline Models, Best/I • Itemize Application Events• Collect Business Indicators• Itemize Other Events• Size Application Events• Develop Business Units or NFU Forecasts• Size Other Events• Build Consolidated Forecast• Define Current Configurations• Input the Technology Plan• Determine Workload Placement• Recommend Hardware Platforms• Build Growth Models

19

A Complete Process -top down -continued

Capacity Plan Process -continuedCapacity Plan Process -continued– Construct Consolidated Plan

• Distribute Base Plan to all CP and Tech personnel• Input Disaster Recovery Plan• Input Financial Analysis• Input Facilities Plan• Input Performance Management Plan• Develop Plan Text• Inputs from Project Implementation Groups• Input Software Support & Maintenance Plans• Input Operations Plan• Input Network Plan• Input Hardware and Software Financial Analysis• Consolidate Financial Impacts• Consolidate Plan Document• Obtain Executive Approval• Distribute the Plan• Conduct process measures and variance analysis

20

A Shortcut Process -bottom up

STRUCTURED CAPACITY PLAN

> Executive Summary> summarize findings

> Business Analysis> business functions, drivers,history,forecasts

> Application Analysis> purpose, future events, history

> Resource Requirements> CPU, memory, DASD-history, forecasts

> Assumptions

> Recommendations> configurations, timeframe, list cost

> Alternatives> configurations, timeframe, list cost

> Benefits & Risks

> Attachments

Business UnitsMarketingDevelopment

ApplicationDevelopment,DBA, Sysadms,Resource Usage

Historical ResourceUsageData,New Inputs,Forecasting Tools

Asset Management,Technology Planning,Tech Support, Opns

UNIX SYSTEM UTILIZATION

0

5

10

15

20

25

1 2 3 4

MONTH

CPUBUSY

• Determine what the Plan Needs to Contain and Plan StructureAscertain the Inputs for each section and develop the section

21

Summary

A useable, accurate Capacity Plan Contains many coordinated activities.A useable, accurate Capacity Plan Contains many coordinated activities. A CP is not a one time effort, but a continuous process of refinement A CP is not a one time effort, but a continuous process of refinement

and verification and revision.and verification and revision. The depth of a CP will depend upon the amount of assets you are trying The depth of a CP will depend upon the amount of assets you are trying

to forecast and the accuracy level demanded by management.to forecast and the accuracy level demanded by management.

Construct a basic Capacity Plan

Workload AnalysisData Gathering

Integrate Infrastructure& Tech requirements

Pricing & ProcurementOptions

Reporting, Presentation& Approvals

Variance Analysisand Refinement

Begin

Forecasting & Modeling Activities

22

References

Carroll, J. R. & Cool, W. S., “Surviving My First Client/Server, Distributed Systems Capacity Study”, CMG94 Proceedings, 1994.

Bell, J.E. & McCord, D.W., “ESTABLISHING A UNIX MID-RANGE CAPACITY AND PERFORMANCE MANAGEMENT ENVIRONMENT”, CMG95 Proceedings, 1995.

Olsen, L., “Erector Set Theory, A Capacity Planning Approach”, KCCMG, 1993. Howell, J., “Workload Analysis”, Fujitsu Technology Solutions RSE Training, 2003. Bell, J.E., “The Origination, Evolution and Future of CRM”, CMG89 Proceedings, 1989. Artis, P., “The Five Stages of Capacity Planning”, CMG85 Proceedings, 1985. Thompson, G.I., “Six Levels of Sophistication for Capacity Management”, CMG2000

Proceedings, 2000.