Date post: | 26-Dec-2015 |
Category: |
Documents |
Upload: | elvin-heath |
View: | 216 times |
Download: | 0 times |
Terry LeeperTerry LeeperDirector Platform StrategyDirector Platform StrategyEMEAEMEA
The Microsoft The Microsoft Development ProcessDevelopment Process
Overview of TopicsOverview of Topics What to ship?What to ship? Team Structure, RolesTeam Structure, Roles How a milestone worksHow a milestone works Keeping builds stableKeeping builds stable Bugs!Bugs! End GamesEnd Games
Not CoveredNot Covered How MS recruits teamsHow MS recruits teams Product SupportProduct Support Shipping delaysShipping delays
Software BasicsSoftware Basics
Shipping TimelinesShipping Timelines
Multi-Month
Multi-Year
Real-Time, Daily, Weekly, Monthly, Quarterly Webzines, subscription fulfillment, open source
Web apps, Desktop software
Complex rich client apps, Operating Systems, Server Applications
General Microsoft General Microsoft PhilosophyPhilosophy
Common Microsoft PitfallsCommon Microsoft Pitfalls Team starts coding before the product Team starts coding before the product
vision is clearly defined, communicated, vision is clearly defined, communicated, and bought-in by peopleand bought-in by people
Vision is technology focused instead of Vision is technology focused instead of customer focusedcustomer focused
Verbal consensus on the vision, but Verbal consensus on the vision, but nothing written downnothing written down
Code reuse across companyCode reuse across company
Figuring Out What To BuildFiguring Out What To Build Most important and most dangerous part of a productMost important and most dangerous part of a product
Importance/danger grows exponentially with team Importance/danger grows exponentially with team sizesize
Failure to get crisp on product vision always produces Failure to get crisp on product vision always produces problemsproblems The release date will slip, and the feature-set won’t The release date will slip, and the feature-set won’t
be perfectbe perfect Clear product plan document provides team focus on Clear product plan document provides team focus on
the product visionthe product vision Helps prioritize goals, product focus, and overall Helps prioritize goals, product focus, and overall
feature-setfeature-set Helps provide road-map for a large team as to how Helps provide road-map for a large team as to how
everything fits togethereverything fits together Helps partners understand what you are buildingHelps partners understand what you are building
Product PlanningProduct Planning
Market NeedsMarket Needs Company Strategy and PartnershipsCompany Strategy and Partnerships
Supporting architectures, platformsSupporting architectures, platforms Supporting new technologiesSupporting new technologies
Customers’ Requests, ComplaintsCustomers’ Requests, Complaints Product Team VisionProduct Team Vision The Core FundamentalsThe Core Fundamentals
Security, Stability, Productivity, Usability, Security, Stability, Productivity, Usability, Performance Performance
Product PlanningProduct Planning
Before M0Before M0 Steering GroupSteering Group
Cross Functions, All LevelsCross Functions, All Levels Focus Groups, Surveys, Much MoreFocus Groups, Surveys, Much More
Answers Answers Where are we going?Where are we going? Map Product onto Long-term VisionMap Product onto Long-term Vision Define Release Specific VisionDefine Release Specific Vision Establish Product Release PillarsEstablish Product Release Pillars
Planning for DevelopmentPlanning for Development
Post-Mortem Previous Product CyclePost-Mortem Previous Product Cycle Development TimelinesDevelopment Timelines
CodingCoding StabilizationStabilization IntegrationsIntegrations Milestone Exit CriteriaMilestone Exit Criteria
Build and Release StrategyBuild and Release Strategy Checking in processesChecking in processes Build processesBuild processes Daily release processesDaily release processes
Cross-Coordination Cross-Coordination StrategyStrategy Working with Product TeamsWorking with Product Teams
Division TeamsDivision Teams Other Microsoft TeamsOther Microsoft Teams
Working with Vendors, CustomersWorking with Vendors, Customers DependenciesDependencies DeliverablesDeliverables
Product StructureProduct Structure GM, VPGM, VP Product GroupsProduct Groups
VS Core, VB, C#, C++, etc.VS Core, VB, C#, C++, etc. Office Core, Word, Excel,Office Core, Word, Excel, BaseOS, GDI, etcBaseOS, GDI, etc..
Team StructureTeam Structure
Products built at Microsoft by “Product Units”Products built at Microsoft by “Product Units” Contains 3 focused disciplines (Dev, Test, PM)Contains 3 focused disciplines (Dev, Test, PM) Product Units are run and report to Product Unit ManagersProduct Units are run and report to Product Unit Managers
Program ManagementProgram Management Responsible for product feature-set and feature designResponsible for product feature-set and feature design Customer advocate within teamCustomer advocate within team Communicators to other groups, customers, managementCommunicators to other groups, customers, management Box and Feature PMsBox and Feature PMs
DevelopmentDevelopment Responsible for implementation Responsible for implementation Responsible for architectureResponsible for architecture
TestingTesting Responsible for quality assuranceResponsible for quality assurance Responsible for writing customer scenariosResponsible for writing customer scenarios
Product Team StructureProduct Team Structure
Product UnitProduct Unit
ManagerManager
DeveloperDeveloper
ManagerManagerTestingTesting
ManagerManager
GroupGroup
ProgramProgram
ManagerManager
Dev LeadsDev Leads
DevelopersDevelopers
Test LeadsTest Leads
TestersTesters
PM LeadsPM Leads
Program MgrsProgram Mgrs
Other DisciplinesOther Disciplines
Build team Build team Product ReleaseProduct Release MarketingMarketing Product DesignProduct Design User EducationUser Education UsabilityUsability
CommunityCommunity LocalizationLocalization Product Support Product Support
ServicesServices Sustaining Sustaining
EngineeringEngineering
Microsoft’s Product Cycle Microsoft’s Product Cycle Model Model
Microsoft Product Cycle Microsoft Product Cycle ModelModel All teams follow this processAll teams follow this process
Implementation of the process variesImplementation of the process varies
Phases sometimes overlapPhases sometimes overlap Could have multiple cycles, in Could have multiple cycles, in
different stages, going in paralleldifferent stages, going in parallel Ideally All Teams Enter Cycle TogetherIdeally All Teams Enter Cycle Together Reality is That They Often Don’tReality is That They Often Don’t
SchedulingScheduling Milestone = unit of product cycle progressMilestone = unit of product cycle progress
Goal: Enable flexible Goal: Enable flexible planning/tracking/stabilization/feedbackplanning/tracking/stabilization/feedback
Milestone schedule: M0, M1…, Beta1, Beta2, RTMMilestone schedule: M0, M1…, Beta1, Beta2, RTM Enable accurate view of progress and distance leftEnable accurate view of progress and distance left
Intended to ensure that team is never “flying blind”Intended to ensure that team is never “flying blind” Features are scheduled in milestones in priority orderFeatures are scheduled in milestones in priority order
Enables schedule flexibility to respond to feedback Enables schedule flexibility to respond to feedback in later milestonesin later milestones
Milestone only “done” when quality “exit criteria” is Milestone only “done” when quality “exit criteria” is metmet Forcing function so team doesn’t go fast and looseForcing function so team doesn’t go fast and loose Ensure very stable, usable, complete product at Ensure very stable, usable, complete product at
milestone exitsmilestone exits
Development MilestonesDevelopment Milestones
Each is a Mini-ReleaseEach is a Mini-Release CodingCoding
Fixed Amount of TimeFixed Amount of Time Code Complete DateCode Complete Date
IntegrationsIntegrations Team code changesTeam code changes External team changesExternal team changes
A Linear View of the PCMA Linear View of the PCM
Plan, Design M1 M2 M3 Stabilize
KickoffCode
Complete RTM/RTWImplementation
Starts
Key Feature Milestone EventsKey Feature Milestone Events
Milestone EventMilestone Event DefinitionDefinition
Spec CompleteSpec Complete Date design specs for milestone features written & Date design specs for milestone features written & reviewedreviewed
Feature CodingFeature Coding Typically 8-9 weeks in length for feature milestonesTypically 8-9 weeks in length for feature milestones
Code Complete (CC)Code Complete (CC) Date all features scheduled for the milestone are Date all features scheduled for the milestone are finishedfinished
Test Plan Complete Test Plan Complete Test plans for all milestone features written reviewedTest plans for all milestone features written reviewed
ZBB Test Pass (ZBB TP)ZBB Test Pass (ZBB TP) All existing functional tests are run on current buildsAll existing functional tests are run on current builds
Zero Bug Bounce (ZBB)Zero Bug Bounce (ZBB) # of milestone bugs older than 48 hours = 0# of milestone bugs older than 48 hours = 0
Zero Resolved Bugs (ZRB)Zero Resolved Bugs (ZRB) # of resolved milestone bugs to be verified before # of resolved milestone bugs to be verified before closing = 0closing = 0
Test Sign-OffTest Sign-Off Final verification and media sign-off on the milestone Final verification and media sign-off on the milestone buildbuild
How do we define a How do we define a feature?feature?
Must be scheduled and testable!Must be scheduled and testable! Specing is reasonably balancedSpecing is reasonably balanced
PMs usually write design specPMs usually write design spec Devs write implementation specsDevs write implementation specs QA writes testing specQA writes testing spec
Performance goalsPerformance goals
Design SpecsDesign Specs Never write production code without designing up-Never write production code without designing up-
frontfront Excellent rule when working on a project just by yourselfExcellent rule when working on a project just by yourself Absolutely mission critical in a team projectAbsolutely mission critical in a team project
Feature-set driven by Program Managers at MicrosoftFeature-set driven by Program Managers at Microsoft Own the customer experience and make right thing happenOwn the customer experience and make right thing happen Drive leading feature team (PM, Dev, Test) during design Drive leading feature team (PM, Dev, Test) during design
processprocess Own writing the final design specification for each featureOwn writing the final design specification for each feature
A good feature design specification is:A good feature design specification is: Clear about the goals and non-goals of a featureClear about the goals and non-goals of a feature Clear about how feature is used by customers and partnersClear about how feature is used by customers and partners Precise about object model and architecture design of featurePrecise about object model and architecture design of feature Clear enough for separate developer, test, documentation and Clear enough for separate developer, test, documentation and
localization team resources to deliver it together without localization team resources to deliver it together without ambiguityambiguity
Example Design Spec Example Design Spec
CodingCoding Developers at MS use a variety of different tools for Developers at MS use a variety of different tools for
codingcoding Visual Studio, Emacs, VI, Source Insight, etc.Visual Studio, Emacs, VI, Source Insight, etc. No tool-specific files or code-injection allowed to be checked-inNo tool-specific files or code-injection allowed to be checked-in
Maintain single source tree for entire product lineMaintain single source tree for entire product line Can type “build -c” from root to build CLR, ASP.NET, .NET FX, Can type “build -c” from root to build CLR, ASP.NET, .NET FX,
Visual Studio and setup programs from scratch (takes 9+ Visual Studio and setup programs from scratch (takes 9+ hours)hours)
Can type “build -c” in any sub-directory of source tree to only Can type “build -c” in any sub-directory of source tree to only build/re-build a sub-portion of product (99% scenario)build/re-build a sub-portion of product (99% scenario)
Each team maintains a separate private “branch” off Each team maintains a separate private “branch” off main treemain tree Use internal source control system optimized for Use internal source control system optimized for
branching/mergingbranching/merging Minimizes check-ins into main tree and avoids build breaksMinimizes check-ins into main tree and avoids build breaks
CodingCoding C++ and C# the two common languages used C++ and C# the two common languages used
in Microsoftin Microsoft Emphasize efficient, clean, maintainable codeEmphasize efficient, clean, maintainable code
Avoid hacks, messy tricks, and stupid Avoid hacks, messy tricks, and stupid optimizationsoptimizations
Code we ship lives a minimum of 10 years Code we ship lives a minimum of 10 years (guaranteed support)(guaranteed support)
Documentation stored externally from source Documentation stored externally from source codecode Avoid documentation writers colliding with Avoid documentation writers colliding with
developersdevelopers All resource strings stored externally from All resource strings stored externally from
source codesource code Visual Studio localized into 8 languagesVisual Studio localized into 8 languages
CodingCoding All code check-ins must be code-reviewed by another All code check-ins must be code-reviewed by another
developer on the team prior to submitting any developer on the team prior to submitting any changes to the source treechanges to the source tree Applies to both the most junior and most senior Applies to both the most junior and most senior
member of the dev teammember of the dev team Dev responsible for check-in suites to exercise Dev responsible for check-in suites to exercise
implemented featuresimplemented features Moving to model where features can’t be checked-Moving to model where features can’t be checked-
in without 60% block-level code-coverage of all new in without 60% block-level code-coverage of all new code from developer written unit-testscode from developer written unit-tests
All product check-in suites must run and pass 100% All product check-in suites must run and pass 100% before any check-in can be submitted into source tree before any check-in can be submitted into source tree (2-3 hour process)(2-3 hour process) Applies both to code you’ve touched, and code you Applies both to code you’ve touched, and code you
haven’t touchedhaven’t touched ““Check-in mail” email sent to team after each submit Check-in mail” email sent to team after each submit
from the buddy machine summarizing the changes from the buddy machine summarizing the changes made, bugs fixed, and files modifiedmade, bugs fixed, and files modified
Producing Daily BuildsProducing Daily Builds A new “build” of the product is built and released every dayA new “build” of the product is built and released every day
Force engineering discipline, critical in the product end-gameForce engineering discipline, critical in the product end-game Build number indicates build date (40730 = July 30Build number indicates build date (40730 = July 30thth, 40810 = August , 40810 = August
1010thth)) Central build lab produces daily builds for entire division (~2000 Central build lab produces daily builds for entire division (~2000
people)people) Staffed 24 hours a day throughout the weekStaffed 24 hours a day throughout the week Each product team must have a build facilitation developer (BFD) on Each product team must have a build facilitation developer (BFD) on
call with beeper to help with any build problems with that productcall with beeper to help with any build problems with that product Build process:Build process:
Build team syncs source code tree (~midnight)Build team syncs source code tree (~midnight) Kick off end-to-end build (~4:00am)Kick off end-to-end build (~4:00am) Build complete (~1:00pm)Build complete (~1:00pm) Perform BVT (Build Verification Tests) run to verify build works Perform BVT (Build Verification Tests) run to verify build works
(~4:00pm)(~4:00pm) Pick up hot-fixes from BFD coordinator and re-build for BVT failuresPick up hot-fixes from BFD coordinator and re-build for BVT failures Repeat hotfix/BVT cycle until the build passes clean Repeat hotfix/BVT cycle until the build passes clean
What a build process!What a build process!
Lab Automation MachinesLab Automation Machines
TestingTesting Test team staffed by devs who are responsible for designing test plans, Test team staffed by devs who are responsible for designing test plans,
writing automated tests, and building the test infrastructurewriting automated tests, and building the test infrastructure Focus on driving up quality, preventing regressions, and enabling Focus on driving up quality, preventing regressions, and enabling
rapid analysis of different builds, variations, and language releasesrapid analysis of different builds, variations, and language releases Current Whidbey Test Status:Current Whidbey Test Status:
102,000 Functional Test Cases102,000 Functional Test Cases 505,000 Functional Test Scenarios505,000 Functional Test Scenarios 71 Stress Mix Variations71 Stress Mix Variations ~1000 servers in test lab to run all of this in an automated way~1000 servers in test lab to run all of this in an automated way
Internal test management system stores and manages test plans/runsInternal test management system stores and manages test plans/runs Allows user to easily add/remove/analyze test plans and casesAllows user to easily add/remove/analyze test plans and cases Allows user to remotely re-image machines within the labAllows user to remotely re-image machines within the lab Allows user to remotely kick off a test-run on a suite of lab Allows user to remotely kick off a test-run on a suite of lab
machines machines Allows user to remotely analyze results of test-runs and publish Allows user to remotely analyze results of test-runs and publish
resultsresults
Maintain ControlMaintain Control
TestingTesting Test plans are first step of the QA processTest plans are first step of the QA process
Documents completed before milestone exit Documents completed before milestone exit detailing all testing variations needed to cover a detailing all testing variations needed to cover a feature areafeature area
Target goal with tests is to reach > 85% product Target goal with tests is to reach > 85% product code coverage by end of the product cyclecode coverage by end of the product cycle We measure this statistic throughout the productWe measure this statistic throughout the product
Always on the lookout for “test holes”Always on the lookout for “test holes” When a bug is opened, the tester for that area When a bug is opened, the tester for that area
always makes sure that there is a corresponding always makes sure that there is a corresponding test-case to prevent it in the futuretest-case to prevent it in the future
Regular automated test runs catch regressionsRegular automated test runs catch regressions Rough metric is 1 regression per 3-6% of bug Rough metric is 1 regression per 3-6% of bug
fixesfixes
Bug TrackingBug Tracking Bugs and work-items tracked within internal bug-Bugs and work-items tracked within internal bug-
systemsystem Enables rich reporting and change history tracking on each Enables rich reporting and change history tracking on each
itemitem All MS products in same bug system (enables linking bugs)All MS products in same bug system (enables linking bugs)
Bugs “triaged” by feature leads and assigned priorityBugs “triaged” by feature leads and assigned priority Priority 0, 1, 2, 3, 4Priority 0, 1, 2, 3, 4 Bugs fixed in priority orderBugs fixed in priority order
Daily status mails sent to entire division tracking bug Daily status mails sent to entire division tracking bug statusstatus Key metrics to watch: incoming and resolve numbers + bugs Key metrics to watch: incoming and resolve numbers + bugs
per devper dev
After the final feature-milestone is completed, life for After the final feature-milestone is completed, life for the product team revolves around driving the numbers the product team revolves around driving the numbers to zeroto zero
Bug QueryBug Query
Bug DetailBug Detail
Ship Mode Glide PathShip Mode Glide Path Large projects “glide in” as opposed to end suddenlyLarge projects “glide in” as opposed to end suddenly
Like a super-tanker -- you can’t dock the ship abruptlyLike a super-tanker -- you can’t dock the ship abruptly Critical to get real-world feedback on product quality earlyCritical to get real-world feedback on product quality early
MS teams typically hold two public betas prior to RTM releaseMS teams typically hold two public betas prior to RTM release Many builds given to key partnersMany builds given to key partners
Key set of steps along the “glide path” to the “end-game”:Key set of steps along the “glide path” to the “end-game”: 1) Lock the feature-set and stop adding/changing design of 1) Lock the feature-set and stop adding/changing design of
featuresfeatures 2) Complete a full test pass to find all bugs in the locked 2) Complete a full test pass to find all bugs in the locked
designdesign 3) Push for a zero bug bounce (ZBB) on the locked design3) Push for a zero bug bounce (ZBB) on the locked design 4) Absorb bug-bounce over few weeks from ZBB4) Absorb bug-bounce over few weeks from ZBB 5) Triage all non-must fix bugs out of the system5) Triage all non-must fix bugs out of the system 6) Enter the “end-game” mode and start minimizing code 6) Enter the “end-game” mode and start minimizing code
churnchurn
StabilizationStabilization
Major Feature Work CompleteMajor Feature Work Complete Dependencies SatisfiedDependencies Satisfied
More Complex = More Time to StabilizeMore Complex = More Time to Stabilize
Focus on Fixing BugsFocus on Fixing Bugs Design Change Process in PlaceDesign Change Process in Place Justify Breaking ChangesJustify Breaking Changes Track Bug TrendsTrack Bug Trends
Incoming and Resolve RatesIncoming and Resolve Rates Trends Towards ZBB, ZRBTrends Towards ZBB, ZRB
ValidationValidation
Don’t “Bounce” Too HighDon’t “Bounce” Too High Focus is on Running Through Test CasesFocus is on Running Through Test Cases
Major Comprehensive “Test Passes”Major Comprehensive “Test Passes” Regression TestingRegression Testing Compatibility TestingCompatibility Testing
Visibility of Prerelease to CustomersVisibility of Prerelease to Customers Beta Customer ListsBeta Customer Lists Beta KitsBeta Kits Support for Beta CustomersSupport for Beta Customers If Final Release, RC Sign-OffsIf Final Release, RC Sign-Offs
End-Game ModeEnd-Game Mode As a product enters the “end-game” mode its “war team” takes overAs a product enters the “end-game” mode its “war team” takes over
War Team is made up of most senior team members – all with ship War Team is made up of most senior team members – all with ship experienceexperience
War Team makes final end-game calls and guides the ship in into portWar Team makes final end-game calls and guides the ship in into port War Team meets at least once a day – twice a day (or more) at the very endWar Team meets at least once a day – twice a day (or more) at the very end
Over the span of the final weeks before a release, the war team will Over the span of the final weeks before a release, the war team will steadily raise the “triage bar” and towards the end only allow steadily raise the “triage bar” and towards the end only allow “showstopper” bugs to be fixed“showstopper” bugs to be fixed If product is ready to ship, the # of showstopper bugs will slowly decline as If product is ready to ship, the # of showstopper bugs will slowly decline as
check-in rates dropcheck-in rates drop This is where shipping becomes less of a science and more of an art – This is where shipping becomes less of a science and more of an art –
the senior leaders need to “feel” that the product is right, have the senior leaders need to “feel” that the product is right, have confidence in the test coverage, the customer scenarios, and the confidence in the test coverage, the customer scenarios, and the overall teamoverall team
Once the rate of incoming bugs that stick gets to zero, we put the build Once the rate of incoming bugs that stick gets to zero, we put the build in “escrow” for a few days and wait and see. If it holds, ship it. If not, in “escrow” for a few days and wait and see. If it holds, ship it. If not, repeat until finally there.repeat until finally there.
Golden Rule of Commercial Software:Golden Rule of Commercial Software: ““2 years from now customers will not remember if you shipped a good 2 years from now customers will not remember if you shipped a good
product 2 months late. But they will absolutely remember if you shipped a product 2 months late. But they will absolutely remember if you shipped a poor quality product 2 months early.”poor quality product 2 months early.”
End GameEnd Game
Allow “Bake Time” After ValidationAllow “Bake Time” After Validation Focus on Finding “Ship-Stoppers”Focus on Finding “Ship-Stoppers”
Daily MeetingsDaily Meetings ““Ship-room”, “Tactics”, or “War-room”Ship-room”, “Tactics”, or “War-room” At Product Team LevelAt Product Team Level At Division LevelAt Division Level
Triage Late Incoming Critical IssuesTriage Late Incoming Critical Issues ““Bug Bars”Bug Bars” Manage Risk – Minimize ChurnManage Risk – Minimize Churn
Declare Release CandidatesDeclare Release Candidates Drive Product Sign-offDrive Product Sign-off
When to Sign-OffWhen to Sign-Off No Active “Ship-Stoppers”No Active “Ship-Stoppers” Meet Exit CriteriaMeet Exit Criteria
Key Scenarios Meet Quality ExpectationsKey Scenarios Meet Quality Expectations Product Features Meet Quality ExpectationsProduct Features Meet Quality Expectations Product Meets Release RequirementsProduct Meets Release Requirements
Security, Branding, Legal, Globalization, Security, Branding, Legal, Globalization, Logo, Etc.Logo, Etc.
Complete Test Coverage RequirementsComplete Test Coverage Requirements Automated and Manual TestingAutomated and Manual Testing All Supported PlatformsAll Supported Platforms All Supported LanguagesAll Supported Languages
Product Pre-ReleasesProduct Pre-Releases ““Practice” ShippingPractice” Shipping
Surface Any Late Integration IssuesSurface Any Late Integration Issues Shutting Down the ProductShutting Down the Product
““Practice” ServicingPractice” Servicing Setup Technology ChangesSetup Technology Changes Another Area of TestingAnother Area of Testing
Getting the Product in Customers’ Getting the Product in Customers’ HandsHands Proof of ConceptProof of Concept Real World CodeReal World Code User FeedbackUser Feedback
© 2003-2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.