IAEAInternational Atomic Energy Agency

Sharing Best Practices for the Design and Management

of TC Projects

INT/0/085

3-5 September 2012

IAEA 2

Contents

Part I: OverviewPresentation 1

Part II: Survey Results

Part III: Selection and Validation Process

Presentation 2Part IV: Selection and Validation Results

Part V: Next Steps

IAEAInternational Atomic Energy Agency

Sharing Best Practices for the Design and Management

of TC Projects

Presentation 2 of 2

INT/0/085

3-5 September 2012

IAEA 4

Sharing Best Practices for the Design and Management of TC Projects

Part III: Selection and Validation Process

Presentation 2Part IV: Selection and Validation Results

Part V: Next Steps

IAEA 5

Sharing Best Practices for the Design and Management of TC Projects

Part III:

Selection and Validation Process

IAEA 6

Where Are We?

The BP initiative has 13 Major Milestones!

0.Consultative Process1.Collection2.Selection3.Dissemination4.Documentation5.Main Streaming

TCQAS

PCMF/In-Touch/External Web

TCDIR-TC-RDIR / RCA

4-TC Regions

Selection

DisseminationDocumentation

Start2011/2012

TC Departments/ Stakeholders

Collection

IAEA 7

III. Selection and Validation Process

A. Key PrinciplesB. BPs Selection Process

1. Composition of the BP Selection Committee (BPSC)2. Consolidated Document on Proposed BPs delivered to BPSC3. Individual Assessment of Proposed BPs4. Consolidation of Assessment Results5. Joint Validation of the Results of Individual Assessment by BPSC

C. BPs Selection Process Timetable

IAEA 8

A. Key Principles

• Transparency in the Selection Process

• Ensure a neutral and independent Selection Process

• Common Understanding on the Principles on how the Selection Process will be taking place

Agreed upon in stakeholders consultation workshop on the 16th & 17th January 2012

IAEA 9

B. BPs Selection Process

1. Composition of the BP Selection Committee (BPSC)

2. Consolidated Document on Proposed BPs delivered to BPSC

3. Individual Assessment of Proposed BPs

4. Consolidation of Assessment Results

5. Joint Validation of the Results of Individual Assessment by

BPSC

IAEA 10

B. BPs Selection Process

1. Composition of the BP Selection Committee (BPSC)The BPSC will be formed by:

• Regional Division Director• TCPC Director• TD Division Directors• Director MTPS

The number of BPSC Members will vary according to actual nominations received as follows:

The Chair for BPSC will be nominated by DIR TCPC. Members will choose the alternate Chair if required to act in the absence of the Chair.

TC RDs (8+) TDs (4) TCPC (3)

1 or 2 Representatives from the MSs (CP/NLO/Regional Chair or Coordinators) in each region = 4+ 1 PMO/SH from each Regional Division = 4

1 TCLOS/TOs each from NA, NE, NS and MTPS (procurement) = 4

3 TCQAS SH, QMS and Monitoring Officer (BP Consultant as an external observer)

IAEA 11

2. Consolidated document on Proposed BPs delivered to BPSC

The TCQAS focal point(s):• Reviews the Proposed Best Practices for completeness of information • Hides any identifying feature that may compromise the neutrality of selection and scoring• Prepares a consolidated document.

TCPC/TCQAS:

• Furnishes the consolidated document based on questionnaire, guidelines for selection process and other relevant documents, electronically, to the BPSC members at the set dates for individual assessment and rating according to the quality criteria.

B. BPs Selection Process

IAEA 12

3. Individual Assessment of Proposed BPs

Each BPSC Member receives a BPs Assessment Package electronically of via external website link from TCQAS-BPs Facilitator/Administrator containing the following:

• Consolidated document of coded, unidentifiable Proposed BPs• Best Practices Selection Guidelines with:

• Annex 1 Individual Assessment Form• Annex 2 Joint Assessment form• Annex 3 FAQs on Selection Process• Annex 4 List of BPSC members

Other supporting documents/items for further reference include:• Project Document of INT/0/085• Core BPs for the Design and Management of TC Projects• Best Practices Surevy Guidelines• Best Practices Survey Questionnaire• Presentation on Best Practices Survey Results (Pilot)• Presentation on Best Practices Selection Process (Pilot)• 2 videos of the above presentations (Pilot) 

The BPSC Member is requested to:• Read each BP submission thoroughly• Fill an Individual Assessment Report for each BP submission, to the best of his/her

skills.

B. BPs Selection Process

IAEA 13

3. Individual Assessment of Proposed BPs

The BPSC members:• Reviews and assesses each Proposed Best Practice based only on the criteria

indicated by the respondent on the submission.• Verifies the same through a list of guiding statements.

The BPSC Member/Assessor makes a tick mark under the relevant columns 3 and 4 as shown below to affirm the indicated criteria and to verify the same through the reading of the BP description

II. BPs Selection Process

1 2 3 4

No. Quality Criteria (Only the criteria indicated by the respondent(s) are to be affirmed and verified for the assessment.)

Affirmation of Criteria indicated by Respondent(s)

Independent verification of Criteria by the Assessor

IAEA 14

II. BPs Selection Process

3. Individual Assessment of Proposed BPs

At the end, the BPSC Member/Assessor gives a rating on a scale from 1-5.

For example if all indicated criteria is met, then the score is 5; if any of the indicated criteria does not seem to be fully reflected in the BP description, then the rating will proportionately be down scaled.

The BPSC Member/Assessor sends all his/her independently assessed reports in one-shot to the TCQAS-BPs focal point(s) by the set date (Pilot: 15 August 2012)

IAEA 15

4. Consolidation of Assessment Results

TCQAS, BP focal point(s) consolidates assessment results into one document. For consolidation, individual scores given to each Proposed BP are summed up and divided by the number of assessments to get an average score for the total independent assessments (for example 15 BPSC Members X 40 BPs = 600). Accordingly, a consolidated ranking report arranged in descending order for all 40 submissions will be prepared in anticipation of the face-to-face BPSC meeting.

III. BPs Selection Process

IAEA 16

5. Joint Validation of the Results of Individual Assessment by BPSC

• On receiving the consolidated assessment results matrix of Proposed Best Practices in this workshop, the BPSC members jointly review the results and, if required, do a random check of any number of submissions to revalidate the assessment.

• For this purpose, the Chair of the BPSC calls a meeting on a mutually agreable date within the set timeframe (upon mainstreaming).

• The Chair facilitates random check of scoring and deliberations on the top 10 ranking Proposed Best Practices leading to a final selection of a number of top ranking Best Practices with consensus. The BPSC will make an announcement of the results.

• Efforts of all participants will be acknowledged and a number of high rated Proposed Best Practices will be publicised for broader sharing.

III. BPs Selection Process

IAEA 17

5. Joint Validation of the Results of Individual Assessment by BPSC

• The BPSC Chair shares with the Members a report containing the top ranking BPs, with names of submitters, regional & country affiliations, BPs title, category (all relevant information that had been hidden during the selection process) etc.

III. BPs Selection Process

IAEA 18

BPs Selection and Validation Process (Pilot Phase)

Selection Process Date

1 Composition of BP Selection Committee (BPSC) May/June 2012

2 Consolidated document on Proposed BPs delivered to BPSC July 2012

3 Individual Assessment of Proposed BPs July/August 2012

4 Consolidation of Assessment Results August 2012

5

Joint Validation of the Results of Individual Assessment by BPSC in a Workshop/Meeting at IAEA, Vienna Agreement on dissemination methodology

September 2012

C. BPs Selection Process Timetable

IAEA 19

Sharing Best Practices for the Design and Management of TC Projects

Part IV:

Selection and Validation Results

IAEA 20

I. BPSC, Chair & Observers

Region Nr. Position Name RoleParticipation in Workshop

Submission BP

Assessment

TCAF

1NLO

AFRA NCMr Messaoud  Baaliouamer (ALGERIA) BPSC Member Confirmed Yes

2 NLA Mr Fidelis Iheanyi Obioha (NIGERIA) BPSC Member Confirmed Yes

3 PMOMr Michel Kiza (substitute for Mr Amha) --HQ

BPSC Member Confirmed No

4 NLOMs Jane Mubanga Mwamba Chinkusu (ZAMBIA)

BPSC Member Confirmed Yes

TCAP

5 NLO Mr Raja Jamal (MALAYSIA) BPSC MemberNot

participatingNo

6 TO Mr Long Nguyen--HQ BPSC Member Confirmed No

7 PMO Mr Sinh Hoang--HQ BPSC Member Confirmed No

8 NLO Mr Syahril Syahril (INDONESIA) BPSC Member Confirmed No

TCEU

9 PMO Mr Andrei Chupov --HQ BPSC Member Confirmed No

10 NLOMr Andras Molnar (HUNGARY) (Representative)

BPSC MemberNot

participatingNo

TCLA

11 PMO Mr Raul Ramirez--HQ BPSC Member Confirmed Yes

12 NLA Mr Eduardo Cortes (CHILE) BPSC MemberNot

participatingNo

13 TO Mr Alejandro Nader--HQ BPSC Member Confirmed Yes

14 CP Ms Elvira Gaytán Gallardo (MEXICO) BPSC Member Confirmed Yes

IAEA 21

I. BPSC, Chair & Observers

Region Nr. Position Name RoleParticipation in Workshop

Submission BP

Assessment

TCAF1   Ms Amira Bazouzi (ALGERIA) Observer Confirmed No

2 SH Mr Mulugeta Amha –HQ ObserverNot

participatingNo

TCPC Nr. Position Name Role

TCQAS 1 Independent Expert Ms Doha Abdelhamid Facilitator

TCPC Nr. Position Name RoleParticipation in Workshop

Submission BP

Assessment

  1 DIR Mr J.A. Casas Zamora Leader BP Initiative Confirmed   

TCQAS

2 SH Mr Javier Guarnizo Section Head Confirmed  

3 QMS Ms Khadija KhanFocal Point/BPSC chair

Confirmed  

4 MO Mr Max Kodjo BPSC Member Confirmed Yes

5 TA Ms Brigitte Liepold Focal Point Confirmed  

6 Intern Ms Carolina Rodriguez Focal Point Confirmed  

IAEA 22

II. Overall Individual Assessment Results

A. Criteria

Items Total QC Rating5. Effectiveness 23.5%4. Effi ciency 21.5%3. Sustainability 22.1%2. Ownership 16.8%1. Relevance 16.1%Grand Total 100.0%

24

IAEA 23

II. Overall Individual Assessment Results

A. Criteria Items

Total Assessors Ticks

Total Respondents Ticks

1. Relevance 24 182. Ownership 25 203. Sustainability 33 304. Effi ciency 32 305. Effectiveness 35 30Grand Total 149 128

IAEA 24

II. Overall Individual Assessment ResultsB. General:

IAEA 25

II. Overall Individual Assessment ResultsC. Few observations:

• Similarity in BP QC selection

• Some Assessors relied on final assessment

• Respondents vs. Assessors conflict of interest

• Regionality of BPs submissions and assessments

• Inaccuracies in understanding of some BPs features on

side of respondents noted (e.g. policy coherence; causal

relationships; BPs titles, template, definition & QC, etc.)

• Dual Scoring

IAEA 26

Sharing Best Practices for the Design and Management of TC Projects

Part V:

Next Steps

IAEA 27

I. Immediate Step: Validation of Consolidated Assessment

Average score is the total score gained/number of assessments. Scores are out of 5.

IAEA 28

II. Near Term Steps

A. Dissemination by BPSC √

B. Awards

1.Invitation

2.Presentation

3.Certification

C. Publication

D. Launch of BPs Community of Practice

E. Incentives √ } (Capacity Building, Expert Missions, Trophies, Medallions, Academic studies etc.)

F. Sustenance/Mainstreaming

IAEA 29

Sharing Best Practices for the Design and Management of TC Projects (Part 2 of 2)

Thank you!