EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

SA1 – Maintenance and Support

Francesco Giacomini (INFN)SA1 Leader

1st EMI Periodic ReviewBrussels, 22 June 2011

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Outline

22/06/2011 1st EMI Periodic Review 2

• SA1 Objectives and Tasks in Y1• Key Performance Indicators• Lessons learned• Y2 Outlook

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611SA1 Objectives and Tasks

22/06/2011 1st EMI Periodic Review 3

• Software Maintenance: Ensure the efficient and transparent management of the maintenance process providing software problems analysis and resolutions

• Release Management: Ensure customers receive certified software releases of middleware services and components according to agreed release policies and quality of service attributes

• Quality Control: Increase the quality levels of the EMI software by contributing to the implementation of the Quality Control procedures defined by the project QA activity

• User Support: Establish the EMI User Support function and integrate it with the overall EGI, PRACE and VRCs user support channels

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

SA1 in EMI

22/06/2011 1st EMI Periodic Review 4

Project Management

NA1, JRA1

JRA1

JRA1SA1

SA1

SA2

Software & Services

Requirements

Defines

ImplementsCertifies

Release Candidate

Process definition

Process monitoring

NA2NA2

NA1, NA2

Collaborations, exploitation, sustainability

Training

Dissemination & Out-Reach

In-Reach

NA1

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

EMI Release and Software Timeline

22/06/2011 1st EMI Periodic Review 5

01/05/2010

Start EMI 1 EMI 2 EMI 3

Support & Maintenance

Support & Maintenance

Support & Maintenance

Supp. & Maint.

30/04/2011 30/04/2012 28/02/2013

Major releases

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Software Maintenance

22/06/2011 1st EMI Periodic Review 6

• Keep the stability of the software currently in production, introducing only the changes that are needed to keep the infrastructure running– Bug fixes or requested improvements– No backwards-incompatible changes

• Waiting for EMI 1 the Software Maintenance plan was applied as much as possible to existing software distributions– Following procedures inherited

from previous projects• Constant flow of updates

with no discontinuity betweenprojects

• Security vulnerabilities addressedfollowing the recommendationsof the Risk Assessment Team ofthe EGI Security VulnerabilityGroup

0

5000000

10000000

15000000

20000000

25000000

30000000

35000000

40000000

May

06

Aug

06N

ov 0

6Fe

b 07

May

07

Aug

07N

ov 0

7Fe

b 08

May

08

Aug

08N

ov 0

8Fe

b 09

May

09

Aug

09N

ov 0

9Fe

b 10

May

10

Aug

10N

ov 1

0Fe

b 11

May

11

# of jobs on EGI

Ensure the efficient and transparent management of the maintenance process providing software problems analysis and resolutions. [DoW] SO1.4, SO3.2Ensure the efficient and transparent management of the maintenance process providing software problems analysis and resolutions. [DoW] SO1.4, SO3.2

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Release Management

22/06/2011 1st EMI Periodic Review 7

EMI 1 – KebnekaiseReleased on May 12th, 2011

•54 products from ARC, dCache, gLite, UNICORE•Scientific Linux 5 (augmented with EPEL5), 64 bit, fully supported platform

– Selected products for SL5, 32 bit, available– Porting to SL6 starting soon

•333 binary packages– 86% of source packages– Digitally signed

•192 external dependencies– 165 from SL5/EPEL5, 27 managed by EMI

Ensure customers receive certified software releases of middleware services and components according to agreed release policies and quality of service attributes. [DoW] SO1.4, SO1.5, SO3.4

Ensure customers receive certified software releases of middleware services and components according to agreed release policies and quality of service attributes. [DoW] SO1.4, SO1.5, SO3.4

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

EMI 1 Preparation

22/06/2011 1st EMI Periodic Review 8

• Required a huge integration effort to put together in a coherent way all the existing software components– Common tools (ETICS, Savannah)– Common procedures

• Tools and procedures provided by SA2– Common external dependencies, same versions

• Globus libraries by IGE, taken from EPEL• Coordination of many people from different

experiences– Engineering Management Team (EMT)– Weekly phone conferences + e-mail discussions

• EMI 0 and four EMI 1 Release Candidates, with increasingly strict criteria, towards the final release– Integration builds run multiple times a day

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

User Support

22/06/2011 1st EMI Periodic Review 9

• Organization based on the EGEE experience, following reviewers' recommendations, and addressing ESFRI requirements

• Automatic notifications and reminders to SUs.• No discontinuity between projects

Establish the EMI User Support function and integrate it with the overall EGI, PRACE and VRCs user support channels. [DoW] SO1.4, SO3.1Establish the EMI User Support function and integrate it with the overall EGI, PRACE and VRCs user support channels. [DoW] SO1.4, SO3.1

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Quality Control

22/06/2011 1st EMI Periodic Review 10

• The activity performed by the Software Maintenance, Release Management and User Support tasks was reviewed by the Quality Control task– Following policies and check-lists contained in the

Software Quality Assurance Plan by SA2– Feedback was returned to SA2 to improve the SQAP

itself, about:• Alignment between SA2-defined metrics and KPIs• Tools to use to measure metrics and KPIs• Introduction of thresholds for metrics• Introduction of tolerance ranges for the checklists

• Coordination of the EMI 1 QC Verification– Joint activity with JRA1 and SA2

Increase the quality levels of the EMI software by contributing to the implementation of the Quality Control procedures defined by the project QA activity. [DoW] SO1.4

Increase the quality levels of the EMI software by contributing to the implementation of the Quality Control procedures defined by the project QA activity. [DoW] SO1.4

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611Security Assessment Plan

22/06/2011 1st EMI Periodic Review 11

• Under the responsibility of the SA1 QC task• Novelty wrt previous projects• Plan available and being implemented

starting from security-related software components– Based on the “First principles vulnerability

assessment” methodology• Collaboration with the EGI Software

Vulnerability Group and the University of Wisconsin

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611Service Level Agreements

22/06/2011 1st EMI Periodic Review 12

• One SLA signed so far, with EGI, in April 2011• It covers Software Development and Support

services needed for EGI– Provide clear description of roles and responsibilities– Provide a measurable description of the offered services

• E.g. target response time for GGUS tickets, depending on their priority

• Covered services– Software Releases– Requirements Analysis– Requirements and Software testing reports– Web-based support

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611Key Performance Indicators

22/06/2011 1st EMI Periodic Review 13

User Support•KSA1.1 – Number of incidents (i.e. tickets)•KSA1.2 – Incident resolution time

Software Maintenance•KSA1.3 – Number of problems (i.e. bugs)•KSA1.4 – Number of urgent changes•KSA1.5 – Change application time

Release Management•KSA1.6 – Number of releases•KSA1.7 – Number of release rollbacks

•Difficult to collect raw data in a uniform way for existing software– Some KPIs were postponed to after EMI 1 release

from GGUS

from RfC trackers

from Release tracker

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

KSA1.1 – Number of Incidents

22/06/2011 1st EMI Periodic Review 14

0

5

10

15

20

25

30

35

May 2010 - April 2011

Less Urgent Urgent Very Urgent Top Priority

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

KSA1.2 – Incident Resolution Time

22/06/2011 1st EMI Periodic Review 15

0

50

100

150

200

250

300

350

400

450

days

May 2010 - April 2011

Less Urgent Urgent Very Urgent Top Priority

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

KSA1.3 – Number of Problems

22/06/2011 1st EMI Periodic Review 16

0

1

2

3

4

5

6

7

Since EMI 1 Release

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Issues, Mitigation, Lessons Learned

• Putting together software developed in very different contexts is hard– EMI 1 represents the starting point

• Putting together the people who developed the above software is harder– Build consensus among project members before

approving plans and policies– It took a lot of time and patience– Agreed plans and policies are now available from

SA2• Application of Quality Assurance methods is

not straightforward– Improve communication

22/06/2011 1st EMI Periodic Review 17

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Issues, Mitigation, Lessons Learned /2

• Difficult/impossible to measure the performance of heterogeneous procedures in a uniform way– No KPIs for existing software– It is now easier, after the EMI 1 release

• Relationship with EGI Unified Middleware Distribution– We find their procedures overly complex but we’ll

do our best to comply with them• Not much effort/experience available within

the project for security assessments– Internal training– External collaborations

22/06/2011 1st EMI Periodic Review 18

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Outlook

22/06/2011 1st EMI Periodic Review 19

• Contribute EMI software to EGI UMD– According to the agreed procedures

• Support and maintain EMI 1• Continue support and maintenance of pre-

existing distributions– With diminishing effort and according to the service

end-of-life policy• Establish support relationships with other DCIs

(e.g. PRACE)• Work towards EMI 2 - Matterhorn

EMI 1

Apr 2011 Apr 2012Jun 2011 Feb 2012Jan 2012

Nov 2011Mar 2012

RC1

RC2

RC3

RC4

EMI 2

Development plans readyETICS infrastructure ready

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611

22/06/2011 1st EMI Periodic Review 20

Thank you