MERGE 2013 THE PERFORCE CONFERENCE SAN FRANCISCO • APRIL 24−26

   

Perforce Administrative Self- Services at SAP

Wolfram Kramer, SAP AG

Perforce White Paper

SAP changed the processes for user and group administration and for project management step by step from a purely centralistic approach to a much leaner one.

2 Perforce Administrative Self-Services at SAP Background SAP AG has used Perforce as its version control system for almost 15 years. The use of Perforce has grown over time; currently, SAP hosts one of the largest Perforce server installations in the world in terms of the number of licensed users: 10,000. In terms of the number of productively used Perforce server instances, it is the largest one, with approximately 100. Many of the developers accessing Perforce use it to work with more than one instance. Perforce implements a per-server user and group management for which instance-specific protection tables are defined. It is evident that on the one hand, a synchronization of users, groups, and also protection tables is necessary to ensure that any user gets access to all the servers where he or she expects it. On the other hand, it is also clear that such a synchronization of user and group data cannot be done manually on all the servers. For this reason, early in the 21st century SAP implemented a central user and group storage based on a relational database. The user and group data for each Perforce server instance are synchronized from there. Manual user and group administration directly on a Perforce instance (via p4 user or p4 group) was not done any more. The leading paradigm for the implementation of the central user and group storage was a centralistic approach of user management and any user and group management activities were done by a central team. Tightly coupled with the scheme of permission groups and protections is the centralized governed depot structure on any Perforce server at SAP. Versioned files are organized in so-called projects as folders on the second level on the Perforce server directly below the depots. The creation of projects and their code lines (organized in the third level below) is strictly standardized and can only be done through a central infrastructure. Perforce triggers prohibit adding new files outside an already existing project and/or code line. The design of the project and code line creation process followed the paradigm of central administration by a central team. Although central teams are good at providing services in a workflow-like manner for implementing and empowerment of governance, they are a hindrance in the context of lean and agile principles because they are bottlenecks for the flow. They produce waiting time for the requestors (project managers or developers) because the workflow still needs human interaction by the central team. This white paper describes how SAP changed the processes for user and group administration and for project management step by step from a purely centralistic approach to a much leaner one with—in the ideal case—no more waiting time for consumers.

Perforce Server Landscape and Centralized User Storage The Perforce server landscape at SAP consists of:

• Approximately 100 Perforce server instances • A relational database named “P4SAP” containing central metadata such as user, group,

and project data • The P4 Management System P4MS as a framework of tools and services for data

manipulation on the P4SAP database

3 Perforce  Administrative  Self-­‐Services  at  SAP  

• Tools and services for data synchronization from the P4SAP database to Perforce servers, also integrated into P4MS

• Self-services and web services around P4MS to enable user interaction

Figure 1 illustrates the architecture:

Figure 1: Perforce server and user storage at SAP

Users and groups as parts of the metadata are stored and maintained on the P4SAP database via the P4 Management System (P4MS). Each group that is stored in the database has a specific validity scope of Perforce servers to which it is synchronized. This is because the sources of certain larger products are distributed over more than one Perforce server. The definitions of groups affecting accesses to the code base therefore have to be the same on each Perforce server. This is ensured by the synchronization of the group from the P4SAP database to all Perforce servers within the validity scope of the group. Synchronization of user data to Perforce servers happens together with group synchronization. The protection scheme at SAP assumes that any user gets permission through a group in which the user is a member. As soon as the first group that contains the user as a member is synchronized to a dedicated server, the user data is synchronized as well. This synchronization ensures that any member of a group that exists on one Perforce server is also a valid user defined on the Perforce server. From a technical point of view, P4MS uses p4 user –i to create a user on the relevant Perforce servers. Permission groups control the access to files within Perforce depots. At SAP, a standard Perforce depot is structured along projects. A project is a location in the Perforce depot tree that hosts the code for a subsystem of an SAP application. Because subsystems can have their own release cycles, the code line level is below the project level, which ends up in a Perforce depot structure (see Error! Reference source not found.). Ruby General� 4/19/13 8:25 AM

Deleted: Figure 2

4 Perforce Administrative Self-Services at SAP Perforce creates folders as pure representations of the full path of versioned files, appearing whenever a file having a new path is submitted. To prohibit wild creation of folder structures at SAP’s Perforce servers, pre-submit triggers prevent the creation of new project and code line folders. Creation of new projects is therefore also a task for the central Perforce team that operates on request by the development teams. It has permission to circumvent the pre-submit trigger for the creation of a first, empty file in the path of the new project. The purpose of this first “createdir.txt” file is to make the path visible in the Perforce navigation tree. By this operation, the pre-submit trigger is deactivated for future submits on the project path. Currently, there are more than 9,000 projects; each has a lot of code lines. New code lines of a project are usually needed for release branching. A service offered in the P4MS framework automatically creates the needed code line as a folder in Perforce. Nevertheless, the creation of new projects is still a manual, central activity.

Figure 2: At SAP, a standard Perforce depot is structured along projects

Design of Perforce Access Groups and Permissions In the past, development teams followed a lot of different processes that slightly differed in requirements concerning code line patterns, their naming, and their permission groups. But one pattern applied to almost all variants contained two basic groups:

• A dev group that has permissions for writing to the dev code line where all new changes are created; developers are typically members of this group.

• A patcher group that has permissions for writing on the qual code line that contains the qualified, stable code; permissions to this code line usually have build operators and quality managers.

5 Perforce  Administrative  Self-­‐Services  at  SAP  

//product_X

project_A

dev

qual

dev  groupproject_A-­‐dev

patcher  groupproject_A-­‐patcher

write

read

write

write

Figure 3: Permissions are simplified for project admins at SAP

Although it is possible to apply this pattern on a per-project level, in most cases, this approach is too finely granular. In general, a product consists of several projects that are contained in one depot. The same people are working on any of the projects within the depot. Therefore, the group pattern is often applied on a depot level: instead of groups like project_A-dev and project B-dev, product_X-dev is used. This simplification means the project admin has to maintain dev and patcher groups only once and not for each project separately (see Figure 3.) Also the central Perforce team that creates new projects does not have to create new groups or make new entries into the protection table. The latter are covered by entries using wildcards such as: write group product_X-dev * //product_X/*/dev/... read group product_X-dev * //product_X/*/qual/... write group product_X-patcher * //product_X/*/dev/... write group product_X-patcher * //product_X/*/qual/...

Centralized User Management Shortly after the introduction of Perforce in 2001, SAP began to distribute projects to more than one Perforce server instance. After first developing intermediate solutions using script-based copy of users from one server to another one, the central user and group storage described above was introduced (see Figure 4). After that, it was possible for the central Perforce operating team to use a web front end to maintain users and groups in the central database. Groups were assigned to one or more Perforce instances. Publishing users and groups to the Perforce server was done instantaneously by pressing the save button.

Ruby General� 4/19/13 8:25 AMDeleted: Figure 4

6 Perforce Administrative Self-Services at SAP

Figure 4: A web front end is used to maintain users and groups in the central database

It was a great step forward because per-instance maintenance of groups was no longer needed. But still all requirements from developer teams were handled by processes based on workflows. Typical scenarios handled by the central operating team were:

• Adding a user to a permission group to grant read or write access to a project; keep in mind that the admin groups already mentioned did not yet exist.

• Creation of a new project including its permission groups.

There were of course tools that made the creation of a project as comfortable as possible for the operation team; creation of the project and the groups was possible in one step. But the strict centralization of any activity concerning user and project management turned out to be a bottleneck: it was not scalable with the increasing number of projects. Instead, it created waiting time for the developer teams and it ate up the time of the central Perforce operating team, preventing it from investing in improvements to the infrastructure.

Group Member Self-Administration The consequence was the insight that user management had to be decentralized. In fact, the decision whether permissions on a project shall be granted to a specific user was not done by the central team but by the responsible development manager of the project. This was when the admin group for a project was introduced (see Figure 5). The admin group has the purpose to modify the dev group and patcher group for a project and for itself. It exists only inside P4MS and is in general not synchronized to Perforce servers at all. Project managers are usually members of this group since they decide who should join as developer or build operator of the project.

7 Perforce  Administrative  Self-­‐Services  at  SAP  

//product_X

project_A

dev

qual

dev  groupproject_A-­‐dev

patcher  groupproject_A-­‐patcher

admin  groupproject_a-­‐admin

change

change

changewrite

read

write

write

Figure 5 An admin group was established to decentralize user management

P4MS has an own authorization concept that is able to assign P4MS-application specific permissions to groups. Members of the admin groups of a project can modify the dev and patcher groups in P4MS and thus in the P4SAP database. The synchronization mechanism described above does the job to make these groups productive on the dedicated Perforce servers. By what has been described, the use case “I as a project manager would like to define the group members who have access to my project” is covered. For the use case “I as a developer would like to get access for a dedicated project,” the mechanism is not sufficient. The assumption is unfortunately wrong that the developer could just write an email to his or her project manager to add him or her into the project dev group. Often the developer does not know who the project manager is.

Figure 6: Any employee is able to request access to a project in Perforce at SAP

To cover the second use case, P4MS offers a service by which any employee is able to request access to a project in Perforce. This can be done in two ways: either the user knows the name of the access group of the project he/she wants to be join or—and this happens in

8 Perforce Administrative Self-Services at SAP most cases because most people are not familiar with the group names—the user provides the Perforce path of the project he or she want to get access to. In the latter case, the system finds out by itself which project the path points to and what the name of the dev group is (see Figure 6). Then the system determines the admin group that is able to maintain the dev group, and an email is sent to members of the admin group. Via the link that is included in this email, any of the addressed administrators is able to open the maintenance dialog for the dev group the requesting user wants to join. After having pressed the button for approval, the user is added to the group within the P4SAP database. The synchronization of the group to the affected Perforce servers is scheduled and takes place within a few minutes asynchronously. What makes this dialog so powerful is that a new Perforce user is created automatically on any affected Perforce server where the user did not exist before. In this case, a strong random password is created, and the user receives a generated email to set a password of his or her choice. Hence the outlined workflow applies to already existing Perforce users who get access to an additional project as well as for people who did not have a Perforce user before. At this point, it is worthwhile to outline the procedure of setting passwords on SAP’s Perforce servers. As already mentioned, many of the users have to connect to more than one Perforce server. Letting them maintain and synchronize their passwords by themselves by applying the p4 passwd command on each server would create much frustration. Therefore, the password setting and distribution occur via a central service in P4MS that sets the password automatically on each Perforce server on which the user is synchronized. For security reasons, the passwords are not stored (not even encrypted) in the P4SAP database. Hence whenever users get access to an additional server, they have to go through this service once again (but with the advantage that, afterward, the password is the same on all Perforce servers).

Limited Local Perforce Project Control Group administration has been successfully decentralized, but the creation of new projects and code lines until recently was completely a central responsibility. But agility for the development teams also means being able to create very quickly the prerequisites to start a new prototype without any unnecessary loss of time. A project on a Perforce server belongs to the most basic infrastructural pieces that enable developers to do their work. A central workflow to create a project with all its waiting time and involvement of central people that does not directly add value to the product does not fit any more. Therefore, SAP introduced a self-service for project creation in a dedicated area where speed is crucial: projects hosting code for mobile applications.

Ruby General� 4/19/13 8:25 AMDeleted: Figure 6

9 Perforce  Administrative  Self-­‐Services  at  SAP  

Figure 7: Self-service project creation at SAP

The solution enables project managers to create a project by themselves without having to request project creation from a central team. The self-service asks the project manager for a project name; provided that the name is not yet in use, the project is created (i.e., the corresponding path on the Perforce server is created) (see Figure 7). So far, this self-service is only able to create the project in one single depot on one single server hosting all SAP mobile applications. Also the permission concept is so far rather pragmatically solved: for all the projects within this depot, there is only one dev group, one patcher group, and one admin group. As long as no special, more granular permissions on a per-project level are needed, this is sufficient. Otherwise, in rare cases, interaction by the central operating team is still required.

The Future: Full Local Perforce Project Control For the future, SAP plans to extend this offering to any kind of projects in any depot and on any Perforce server such that any project manager at SAP could get rid of lengthy workflows for project creation. But this will not be possible without solving a structural problem: the right choice of the hosting depot and Perforce instance for a completely new project still needs some central governance. Often the creation of a completely new depot is required; in the future, this will be a task of central administration. But luckily, that only happens when completely new products are invented. The much more frequent use case for a project manager is: “I need a new project in the same depot where all my other projects are already located.” Specifying an already existing project in a dialog or choosing it from a drop-down box is then an easy task for the project manager and a feasible approach of a generic self-service for project creation.

10

Perforce Administrative Self-Services at SAP Another challenge comes with the creation of protections on the Perforce server. Unfortunately, there is no common pattern for code lines and groups that have read or write access to them that fits for all development teams. Much worse, development teams need the ability to create new code lines (e.g., for prototyping that does not follow any name pattern). To avoid involving central manual interaction again, the protections on the Perforce servers have to be adapted as well: the self-service must be able to add lines into the protection table to grant permissions on a code line defined by the project administrator.