Visual Studio ALM: Lessons Learned Through Dogfooding Brian HarryTechnical FellowMicrosoft Corporation

SESSION CODE: DPR206

AgendaLessons Learned Through Dogfooding

OverviewTFS at MicrosoftGetting started on a releaseDevelopment Tracking and ReportingValidation

Visual Studio 2010

Visual Studio ALM Progression

V1 - TransparencyV2 - Bringing together development & testV3 - ?

TFS at MicrosoftMay 2010 Growth TotalsUnique Active Users

+2,128 22,430

Total Active Users

+1,072 25,208

Projects +128 5,028Work Items -30,206 6,679,715Source Code Files

+1,773,773

91,337,562Metric TFS 2008 RTM

TFS 2010 RTM Ratio

TFS Instances 24 40 1.7XUnique Active Users

7,453 22,430 3X

Projects 1,209 5,028 4.2XWork Items 1,096,318 6,679,715 6.1XSource Code Files 24,952,519 91,337,562 3.7X

TFS at Microsoft

TFS in Developer Division

3,839 Recent users18.7TB Storage826,234,724 Files1,594,094 Checkins839,595 Work Items

At least 10 different locations, including 4 “major” ones

DevDiv TFS Topology

AgendaLessons Learned Through Dogfooding

Getting started on a releasePlanning

The Challenge

What We’ve Learned

Planning And Tracking

Is the organization aligned? How are we doing against biz objectives? Are we making progress on the right things?

Align the organization around key objectives Leverage TFS to reflect the planning process Use TFS Office integration for daily tracking Use TFS for end to end traceability and reporting

Our Process – VS 2010

Features

Feature Groups

Value proposition

Main objectives Pillars

Value Props

Feature Groups

Deliverables Deliverables

Feature Groups

Deliverables

Value Props

Feature Groups

Deliverables

Value Propositions

Deliverables

Features

Deliverables

Tracking Tasks In Microsoft Excel

Tracking Tasks In Microsoft Project

Iteration Reviews – Business

AgendaLessons Learned Through Dogfooding

Getting started on a releaseDevelopment

Organizing the WorkBranching

The Challenge

The Solution

Organizing the Work

Incredibly difficult to drive a release with 2,000 people. High degree of churn. Complex interdependencies. Inexorable schedule

“Feature Crews” Mini-development organization Semi-autonomous Accountable for delivering quality

Checkpoints Keep Focus

Checkpoint 0 – ScopeCheckpoint 1 – PlanCheckpoint 2 – DemoQuality gates & Merge

Establishing Architecture

SCRUM Is Very Popular

Quality as You Go

Development and test are simultaneousDevelopers & Testers share responsibilityDevelopers & Testers share tools

Test planningUnit testingFunctional testingManual testingPerformance testingStress testing

Test Planning

Unit Testing

Manual Testing

Performance & Stress Testing

Dashboards Provide Visibility

Builds

Every checkin built & verifiedBuildCode AnalysisCheckin tests

Nightly builds provide QA updates

Quality Gates Prevent Chaos

Carry no debt in feature developmentFeature Must Pass “Quality Gates” on Finish before

Feature completeTest CompleteAll Bugs FixedSecurity PlanStatic Code AnalysisCode CoverageNo Performance RegressionsLocalization TestingAPI Reviews

Feature may merge to active branch only when Quality Gates are met

2005 Debt vs. 2008 Debt

VS 2005 Beta 1Product Bugs only

VS 2008 Beta 1ALL bug debt

The Challenge

The Solution

Branch Strategy

Massive division – 2,000+ people Code churning at every level

Isolation and Integration Isolate major feature areas in branches Automate branch quality checks Rhythm of regular merges Handles integration complexity

Source/Branch Structure

1 branch per productGated checkin procedureVBLs build in parallelNightly test suites verify branch healthRegular schedule for merges Stringent criteria for merges to main

$/DEV10/Main

$/DEV10/PU/<BranchName>

$/DEV10/feature/<BranchName>

$/DEV10/PU/<BranchName>

$/DEV10/feature/<BranchName>

AgendaLessons Learned Through Dogfooding

Getting started on a releaseDevelopment Tracking and Reporting

Trustworthy TransparencyTop Tracking Metrics

The Challenge

The Solution

Trustworthy Transparency

Each team needs to optimize process differently for their needs Leadership teams need central rollup to trackprogress.

Fantastic hub-spoke federated model…All teams use TFS for central trackingEach team uses best process for its needsSingle reporting system

Iteration Reviews – Business

Iteration Reviews – Business

Iteration Reviews – Product Unit

Iteration Reviews – Product Unit

AgendaLessons Learned Through Dogfooding

Getting started on a releaseDevelopment Tracking and Reporting

Trustworthy TransparencyTop Tracking Metrics

Validation

The Challenge

The Solution

Validation

How do you make sure a public release is high quality?

Integration testing Overall Gut Feel “OGF” Stress testing Slowdown the churn

Shutdown Process

Full test passZero Bug Bounce (ZBB)Zero Resolved Bounce (ZRB)Tell ModeVerification test passAsk ModeEscrow

Tracking Bugs – Aggregate

Tracking Bugs – Rates

Tracking Bugs – By Feature Area

Tracking Bugs – By Person

Exit Criteria Summary

Summary

We have rolled our learnings back into our toolsWe have encountered problemsWe have learnedWe have rolled our learnings back into the tool

We Believe In These ToolsScales to Massive Commercial DevelopmentWidely tested

I hope you can learn and benefit from our experience

Track Resources

Resource 1

Resource 2

Resource 3

Resource 4

Resources

www.microsoft.com/teched

Sessions On-Demand & Community Microsoft Certification & Training Resources

Resources for IT Professionals Resources for Developers

www.microsoft.com/learning

http://microsoft.com/technet http://microsoft.com/msdn

Learning

Complete an evaluation on CommNet and enter to win!

Sign up for Tech·Ed 2011 and save $500 starting June 8 – June 31st

http://northamerica.msteched.com/registration

You can also register at the

North America 2011 kiosk located at registrationJoin us in Atlanta next year

© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

JUNE 7-10, 2010 | NEW ORLEANS, LA