Who are these guys, anyway?

Vibhu SrinivasanAgile Coach, Technologist.Doing XP since 1999

Ed KraayScrum Master, CSPAgile Coach

Doing scrum and XP since 2004

Hypothesis: Scale hurts due to increased communication cost

• Communication increases geometrically, while team members are added linearly.

• Open environment was not enough to solve the issue.

• Teams locally optimize

The context: An MMO game project0 Identity Management

Suite

0 Content Management System

0 Social Networking

0 3D Game

0 Back-end game infrastructure

The service platform

0 Technologies in Play

0 Java, Windows Communication Foundation , C# .NET 2.0, .NET 3.0, ASP.NET

0 PHP

0 A packaged product for content management

0 LDAP

EnvironmentScrum + XP

•Scrum Process•All 3 roles, artifacts, and ceremonies

•XP Practices•Paired Programming•Continuous Integration•Test Driven Development

•Open environment

5 teams, 9 projects, ~50 people

0 Multiple Team Gaming Program

Customer Team

7/18/2010All Rights Reserved – Vibhu Srinivasan

http;//agilefaq.net 8

How we organized our Meetings, and how that mapped to the 5 levels of

planning

TimelineTeam Forming:

March 07

First Release, Infrastructure:

August 07

Game Infrastructure

Teams: August 07

Program Retrospective

June 08

Planned Releases:

September 08 –December 08

The Timeline

How did we generate these insights?

• Held a cross team program retrospective

• 1 day facilitated event

• Set the Stage, Built a 1+ year timeline, generated insights and defined actions

• This was in addition to project, iteration retrospectives

Hurt: Undone work at the end of a sprint

0 Caused the next sprint to reduce velocity, reduced quality

0 Helped: Tracking the Right Metrics: Committed vs actual points, and worked to even them out.

Test Coverage

Committed vs

Completed Points

Hurt: Defects slowed velocity

0 Helped: Test Driven Development

0 Hypothesis: TDD reduces defects.

0

100

200

300

400

500

600

700Defects and Tests

Unit Tests

Acceptance Tests

Defects

Hurt: Multiple Backlogs0 Priorities handled on

the back end, mid sprint

0 Caused injection of priorities

Many Teams, many backlogs

Helped: chief product owner

Hypothesis: product definition team

0 A team of product owners, UX, stakeholders, working ahead of the teams, defining and feeding the backlog.

Hypothesis: Use one product backlog to reduce pain of multiple backlogs

Hurt: Multiple business priorities

0 Caused shifting focus on team.

0 Helped: Vision Statement

Hurt: Work inserted in the middle of the sprint

0 Would break the sprint, or cause delays

Helped: Synchronizing our sprints

Hurt: lack of cross team culture 0 Caused silo effect,

even when teams were feet apart from each other.

0 Helped: X-Team Events: Retrospectives, Lunch, Release Planning, Celebrations.

Team A

rules!

Team B

rules!

SwarmingHelped: Collaborative Environment

Helped: X-Team Release Planning

Helped: Ensuring all customer stakeholders are represented by

Product Owners

Hurt: Impediments

0 Hardware Requisition Process

0 Network Issues

0 Team Issues

Stop the line

7/18/2010All rights reserved Vibhu Srinivasan

26

Services are painful to consume

0 Teams are working in silos and when services are built, they are too painful to consume

Service Versioning

Well defined contracts

Service level agreements

Lack of design caused too much technical debt

0 Agile Myth = There is no design and architecture in Agile development

0 Going from story to code causes a lot of technical debt

0 UML is not enough to convey details

0 Teams are required to white board designs one day after the sprint planning with others

0 Use more spikes to understand a design before it can be used

0 More emphasis on pairs to talk about what they are going to do, before changing code

UML does not convey the details of the complex models

0 Some services had too many complicated scenarios that could not be communicated easily in pictures.

0 Use Fundamental Modeling Concept ( FMC ) models to convey the overall intent of the system0 Color Coded by Release

linehttp://www.fmc-modeling.org/

0 Invest in a great digital camera and a large whiteboard

SERVICE

Services Architecture

PrivisionCollection

Package services

Goals Accounts

Uniquq Rights

Client Systems

Database

RSERVICE API

R

Manage

CollectionR

Purchase

R

Open

R

Combine/

ConvertR

Unique

R

Use

In

Game

R

View

Administration

RSERVICE API

R

Redeem

R

Escrow

R

Destroy

R

Share

Security

R

Define

R

Package

R

Publish

R

Create

Collation R

Recipe

R

Consume

RLog R

AuditR

Manually

Override

Ownership

RSet

Inventory

Levels

RTrack

Ownership

Patterns

R

Audit

R

Repository

Feed

R

Health

Check

Release Line 1

Release Line 2

Release Line 3

Game

New Game

Game2

Game 3

Game 4

R

Trade

R

Run

Recipe

More Engineering Pains0 Common Engineering

Practices Missing

0 Code was growing too fast in too many ways across teams

0 Duplicated code

0 Customers had a core architecture group that had less and less visibility as the teams grew and this caused unease.

One solution to these pains

0 Introduce a cross team architect and embedded agile coach0 A servant thought

leader

0 Mentor and the readily available to pair

0 Shields teams from technical bombshells

0 Active listener and helper

0 Help inject technical stories

More People = Less Quality

0 Some teams were reporting lower bugs than others

0 Unit testing was not consistent

0 Acceptance testing was missing

0 Team did not see value in TDD

0 More debates and arguments were leading to unhappy developers

0 Bad smells in code

Hurt: Defects slowed velocity

0 Helped: Test Driven Development

0 Hypothesis: TDD reduces defects.

0

100

200

300

400

500

600

700

Defects and Tests

Unit Tests

Acceptance Tests

Defects

Better Smells

0 Teams were asked for mandatory testing . They could choose to implement the tests any way they want, but should have unit tests

0 Build systems were made smarter to publish test results

0 Category tags were used to group tests for quicker running.

0 Inject one or two senior developers who knew TDD well in every team- Peer pressure = great code

7/18/2010All Rights Reserved – Vibhu Srinivasan

http;//agilefaq.net 36

It works for me syndrome

0 One teams code does not interface with the other team well

0 Code working on local build fails to integrate

0 Its not easy to deploy the code to customer

Solution – Integrated build and deployment strategy

0

Odds and Ends

0 Mock objects were mockingly simplistic

0 Acceptance tests if not automated does not provide a whole lot of value

0 MOB Programming helps for cross team communication

0 A team agreement document is a must have tool.