A general perspective

Review 12 – 17 October 2015

CR-1 2015

• Introduction: key components of success

•The challenges: the challenges during the review and what opportunities they presented

•Good practices and tools: what was helpful in addressing the challenges

•Beyond the review week: how we kept the momentum going after the review week

Turning a good review into the greatA holistic approach:

• take a systemic view of the review

• apply project management techniques to LR work

• design and use a communication plan

The good is in details:

• early preparation and planning

• think through and plan for every step of the review (“what”, “so what”, “now what”)

Building a great team:

• communicate clearly

• share knowledge and responsibility

• have fun

Taking ownership in innovative solutions:

• identify the issues

• be proactive and solution-focussed

• take collaborative approach to solution design and testing

Review challenges: turning challenges into opportunities

The challenges of the review formed three major groups:

•New rules – due to changes in UNFCCC and IPCC guidelines as reporting and review rules

•Logistics – tools, templates, organisation & planning

•General – more generic challenges not related to review guidelines or technical issues

Turning challenges into opportunities: new rules

New guidelines for inventory reporting and review + IPCC 2006 methodological Guidelines:

the review process under the new rules had no precedents to follow

•An opportunity to develop a better understanding of the review process•A natural environment for in-depth group discussions•A great learning opportunity for new and experienced reviewers

Turning challenges into opportunities: tools and templates

New ARR template:

a streamlined table-based document with extensive internal cross-referencing

• An opportunity to apply innovative solutions to working with the ARR

• Saving time and efforts by applying the consistent approach to questions to the Party and potential recommendations in the ARR

• Design and test simple software tools to link wrap-up discussions with the new ARR

Tools: the usual software tools (like Locator, NK finder etc.) were not available

•Explore and understand both CRF data entry file and the output files

•Use software skills to make the use CRF input file more efficient and user-friendly

• Lesson learned: we can be less dependent on the standard review tools for data analysis and verification

Delay in the assessment report resulted in delays with the Party responses

•Apply a more flexible approach to the review planning and prioritise our work better

Turning challenges into opportunities: tools and templates

Communication: working across 22 different time zones was a challenge

•An opportunity to explore on-line meeting planning tools

•Use the time-difference to our advantage to schedule almost 24/7 non-stop work process

Language: one of the Inventories has been written in Russian with no English summary

•An opportunity to utilise our linguistic

skills and experiences

•Work more cohesively as a team

Turning challenges into opportunities: general

We have welcomed three new experts to the ERT (Energy, IPPU and LULUCF sectors)

•An exciting opportunity to meet the new people

•Strengthen our collaboration skills through group work, coaching and inter-disciplinary discussions

•Great learning opportunity for all experts through sharing knowledge and new ideas, responding to challenging questions and giving feedback

• Lesson learned: experienced reviewers working together with new experts produced excellent questions and insightful analyses all the way through the review

10

Turning challenges into opportunities: general

Good practices and tools

We found it a good practice to have a detailed plan for every part of the review, including:

•Timeline: Milestones schedule/ Hourly planning for the review week /Detailed schedule for completion of the ARR

•Roles and responsibilities

•Communications

•Q&A process

•Discussion process and tools

•ARR compilation process

all these will be introduced in the follow-up slides

Good practices and toolsWe have discussed, designed and tested agendas and tools for each aspect of the review week, making sure that every angle is covered:

•Role allocation (detailed)

•Team diary

•Meeting/conference planner (time matrix for more efficient communications – free on-line tool)

•Dashboard

•Agendas for: teleconferences; introduction meeting; daily work; wrap-ups

•Discussion tools (Wrap-up files 1 and 2)

Week: -5 -3 -1 0 0 0 0 0 0 2 3 4 6

Responsible ERT member (include

specific people once we know

responsibilities)

Access to VTR confirmed

(Due 14 Sept)

Prelim. Qs Sent (Due 28

Sept)

Sector-based ZOD provided

(Due 9 Oct) ZOD of ARR

LR comments on ZOD (by

17 October if possible)

UNFCCC comments on ZOD (by

17 Oct if possible)

Response to ZOD

comments (Due 17 Oct if possible)

Review transcript

(Due 17 Oct)

Preliminary main

findings (Due 17 Oct)

Inputs for consolidated ZOD (or FOD if respond to comments)

(Due 2 Nov.)

LR approved FOD (9

November)

Review officer

comments on FOD (Due

13 Nov)

Response to follow up

questions or comments

Submit to editing and QA (Due 27

Nov)

Respond to QA

comments

Send to Party (Due

25 Dec)

Response from Party

received (Due 22 Jan)

ERT response to comment (1

Feb)

Final ARR from ERT

(due 22 Feb)

Publication of final ARR (7 March)

Canada

General NA NA NA NA NA NA NA

Energy NA NA NA NA NA NA NA

IP NA NA NA NA NA NA NA

Agriculture NA NA NA NA NA NA NA

LULUCF NA NA NA NA NA NA NA

Waste NA NA NA NA NA NA NA

LRs NA NA NA NA NA NA NA NA NA NA NA NA

Secretariat NA NA NA NA NA NA NA NA NA NA NA NA

Ukraine

General NA NA NA NA NA NA NA

Energy NA NA NA NA NA NA NA

IP NA NA NA NA NA NA NA

Agriculture NA NA NA NA NA NA NA

LULUCF NA NA NA NA NA NA NA

Waste NA NA NA NA NA NA NA

LRs NA NA NA NA NA NA NA NA NA NA NA NA

Secretariat NA NA NA NA NA NA NA NA NA NA NA NA

Codes

Date finished

Not yet submitted

Not applicable

Good practices and tools: time-line (ex 1)The time line has been established in discussion between LRs, ERT & ROs

• First communication LR/RO – 7 weeks prior the review week• Long-term time-line created – 6 weeks prior the review week• Roles & responsibilities – 4 weeks prior the review week• Teleconferences – 3 weeks prior the review week

Good practices and tools: approach to communication

We approach this as a set of options

Elements of the plan included:

• Timing, frequency, triggers and methods (e.g., e-mail, teleconference, person-to-person call; workgroup/full ERT discussion);

• personal preferences in approach to communication; dealing with delays (plans A, B, C)

• On-line meeting planning tool was used to coordinate teleconferences across multiple time zones.

• Teleconferences were held in two consecutive days to ensure convenience for experts located across 22 time zones.

The plan was also used after the review week

Communication: teleconferences

•Timing could be challenging – we used an on-line meeting organiser (free) – matrix planner

http://www.timeanddate.com/worldclock/meeting.html

•Split group discussion to cater for different time zones (we had 2 groups)

•Sign-off process for each proposed teleconference option

•Detailed agenda

•Beware - teleconferences is an equipment challenge (BIG sound problems); detailed agenda and can help

Teleconference: sample agenda

• Introductions (LR)

• Overview of status of review tools (RO)

• Introduction of new processes and procedures (e.g. based on new guidelines or LR recommendations)

• Round of questions (all) – points of concern:• ZOD• Recommendation vs encouragement• Time-series consistency• Sector-specific• CRF-related technical questions

• Work plan for the review week

Good practices for role allocations (with example)

Expert fieldFamiliar with

the Party’s systems/NIR?

Knows Russian?

Party to review

Role options

Generalist (LR) (can help Energy & IPPU)

Yes (national system)

fluent Can

1. Primary (Canada)2. Help the other generalist with language, if needed3. Working with Energy expert on translation/ teleconference with Ukraine

• Timing: 4 weeks prior review

• The following attributes were taken into account:

– Field (extended for cross-sectoral collaboration)

– Experience

– Familiarity with the Party’s inventory/inventory systems

– Language skills

– Primary and secondary responsibilities

Good practice: Q & A process

•The process: all questions to the Party were peer-reviewed by

• fellow experts

•RO

• LRs

•The approach: “context-trigger-question” (CTQ) (this was initiated by LRs)

•CTQ was applied consistently to all

questions and later – to ARR entries

Good practice: CTQ (what? -> so what? -> now what?)

Each statement contained three elements:•The context where the issue was nested •A specific issue that triggered the question (a

contradiction, an identified gap, etc) •A question to the Party / a potential

recommendation that should help to resolve the matter

Why this was important:•This saved time on question clarifications with the

Party – practically, none were required• It helped to keep track the questions and potential

issues for the ARR

Discussion process and tools for the review week

We used process-based approach to ensure that the following aspects were given time and consideration:

• Individual work and quiet time for research & analysis.

•Work in small groups – when experts in each sector were given an opportunity to discuss their findings with other experts, peer-review each other’s work and discuss critical points with or get clarifications from the LRs.

•Whole group discussions (daily wrap-up sessions); the focus of each review day was factored in the discussion themes

Example: planning daily sessions during the review week

• Morning and early afternoon (9 am – 2 pm):

Individual/group work on questions and reports or/and teleconferences with the Parties

• Afternoon (2 - 4pm):

Expert group discussions of all issues for all Parties for each sector

• Late afternoon (4 – 6 pm):

Entire ERT discussion of day’s progress. This would include the ERT dashboard of identified issues and relevant questions, proposals for teleconferences with Parties and identifying potential recommendations.

• Evening (6 pm+):

LR finalised questions for the Party

AND we have finished each session in time

Q&A Review Transcript

ARR SummaryTable 2

Provisionalmain findings

Full ARR

Monday

Tuesday

Wednesday Last questions,evening if possible

Thursday

Friday Transfer RT issues to ARR documentSaturday Fully

elaborated

General Timeline of Review Week

Summary

table 2 may

help guide

wrap up

meetings

Example: flexible approach to review week schedule• We combined the timeline with review week agenda• The daily plan was not rigidly fixed – there was space for adjustments when

needed.

The review week: new discussion tools To further improve efficiency of discussions, two discussion tools were designed (file 1 and file 2)

File 1 (used on Tuesday-Thursday): •Based on the review transcripts • Issue identifier by: •Sector •Type (consistency, accuracy etc.)• Issue? (following the Issue flow chart) • Is it repetitive?

To streamline this process even further:

• Link RT to the discussion file

• Even better: direct link RT -> ARR

File 1: fields• Fields:• Issue ID • Category• Gas• Finding description• Is included in 2014 ARR? (incl. paragraph #)• Number of successive reviews when the issue not addressed

(Years X-X)• ERT assessment: Is the finding an issue?• Issue type (please select from the drop-list)• Recommendation/Encouragement?• Status (resolved, not resolved, not relevant, addressing)

• Process:

• Each expert populated a relevant sector/Party tab with the potential issues discovered during the day and sent them to the LR*

• LR compiled the sectoral entries in one file*

• ERT discussed the sector entries and discussed the status of each identified issue

• Product: a good ZOD for tbl 5 of the ARR (2015 template)

Review week: new discussion tools

File 2: •Consolidates and categorises the issues for tbl 2 of

ARR• It looks like tbl 2 of the 2015 ARR template•Compiled during the discussion•Best to be used for Friday’s discussion when the

majority of issues have been identified and discussed using File 1

The Process: •Experts name the issue identifiers as per File 1 to

populate each field of File 2•The ERT revisits some issues (if needed) and

finalises File 2•The product: ready-to-go ZOD for tbl 2 of the ARR

Review week: provisional main findings

•The final versions of files 1 and 2 provided a good summary of issues provisional main findings document

•We also applied the CTQ approach to improve consistency of the document and reduce its size

Files

1 & 2CTQ

Provisional

recommendations

Key lessons learned

• Each review comes with its challenges – turn them into opportunities to learn and grow

• Early planning saves time and helps the review to go smoothly

• Be creative – new tools and ideas could be helpful to more than one team

•New experts bring new ideas and ask good questions – we all can learn from them

• Collaborative work before, during and after review week helps to share the load, produce excellent solutions and makes the review a great fun