Extending Perforce Scalability Using Job Content Synchronization Shannon Mann Software Developer Research In Motion

Outline

•  Problem Definition •  Brief History •  Investigation

•  Solution: Job Content Synchronization •  JCS Concept •  JCS Details

•  Launch •  Implementation •  Benefits •  Concerns

•  Wrap-up •  Conclusions •  Questions

Problem Definition

Brief History

•  Early 2000 – first Perforce use •  Mid 2005 – large scale automation introduced

•  Automated parallel builds •  Dramatic growth of continuous integration •  Performance issues began

•  Late 2006 – performance severe •  Decision to split

•  Mid 2007 – split not possible at this time •  Code inter-dependencies too high

Brief History

•  Mid 2007 - Alternatives sought •  Monitoring introduced •  Hardware replaced - Sparc to Opteron •  OS replaced – Solaris to Linux •  Performance improved 10X

•  Continued accumulated improvements by 100X •  Initial 2007 samples at 22 seconds, now .2 seconds

•  Userbase activities increased by 6X •  Early 2010 - Decision to split revisited

Investigation

•  Decision to split by duplicating db and depot data •  separate each depot’s trees with protections •  best done with explicit protections

•  Reduce cost with post-split cleanup •  clientspec cleanup reduces db.have size •  Offline obliterates completes split

•  leaves depot files as-is – expensive •  Depot cleanup needed – future research topic

Investigation

•  Split Challenges •  large number of tool integrations •  multiple groups with conflicting needs •  moving product delivery dates

•  Bug-tracking integration complex – show-stopper •  Bug-tracker itself integrated with many tools •  Integration complexity dramatically increased •  Alternative desperately needed

•  Goal - avoid changing bug-tracker integration

•  Job Content Synchronization concept saves project

Solution: Job Content Synchronization

JCS Concept

•  Middleman depot allows existing bug-tracking integration unchanged

•  JCS copies job data from middleman to production

•  Perforce Support gave enthusiastic support •  Produced reference solution in two weeks

JCS Concept

Bug Tracker

Original Depot

P4DTG

Bug Tracker

P4Jobs Depot

P4DTG New Depot

JCS

New Depot

JCS Details

•  Stand-alone depot has P4DTG integration with bug-tracker •  Needs P4Broker to redirect ‘p4 jobs’ •  Needs P4Auth to make redirect transparent •  Needs P4Change to make connections unique •  Needs triggers to request initial job data •  Uses P4DTG in a depot-to-depot mirror of job data

JCS Details

Bug Tracker

P4Jobs Depot

New Depot

P4Auth P4Change

P4Broker

Users

P4DTG

JCS

Launch

Implementation

•  Tested using full test implementation •  All parts must be present •  Users must test their tool integrations

•  Staged production release •  P4Broker, P4Auth and P4Change implemented first •  Split done at later date – reduces risk •  Additional depots added as needed

Benefits

•  Greatly increased scalability

•  Depot-side and bug-tracker side connections increase linearly and independently

•  Near-trivial low-impact scalability

•  Non-split depots benefit – bug-tracking connect for free

Concerns

•  Significant increase in deployed complexity

•  Expert knowledge of subsystems now required

•  Complexity extends investigations and corrections

Concerns

Bug Tracker

Original Depot

Users Original Setup

Concerns

Bug Tracker

P4Jobs Depot

Users

P4Auth P4Change

New Depot

One Depot w/JCS

Concerns

Bug Tracker

P4Jobs Depot

Users

P4Auth P4Change

New Depot

New Depot

Two Depots w/JCS

Wrap-up

Conclusions

•  JCS allows near-linear scaling of bug-tracking interconnects to multiple depots

•  JCS allows a reduction in performance impact

•  JCS eliminates split impact to bug-tracking system integrations

Conclusions (cont.)

• JCS also allows multiple non-split depots to share bug-tracking integration

•  JCS can be used just for bug-tracking integration sharing •  Splits aren’t required to consider using

•  P4Auth, P4Change and P4Broker can be used alone

•  Perforce Support are your friends. Seek their advice

Questions?