rmpex.orgrmpex.org/.../uploads/2016/08/RMPExTeam_Lead_Ref_G… · Web viewAs the leader of a...

Team LeadershipReference

GuideAugust, 2016

Table of ContentsForward..................................................................................................................................ivQuick References.....................................................................................................................vTeam Lead Checklist...............................................................................................................viIntroduction............................................................................................................................1

Representing Rocky Mountain Performance Excellence.................................................1Role of the Team Leader.................................................................................................1Tips for Leading Your Team to Peak Performance..........................................................2

Focus on Results and Productivity..........................................................................2Align People With the Criteria Areas Team Members Know the Best....................2Align People with the Criteria Area in which they are Passionate..........................2Put your Best Performers on your Biggest Opportunities......................................2Establish a Realistic Evaluation Schedule...............................................................2Trust Your Team Members – And Let Them Know It..............................................3Avoid Blame ......................................................................................................3Do Not Provide all the Answers – Help Your Team Members Think.......................3Build Consensus by Letting People Participate and Know ‘Why’............................3

Working with the Applicant............................................................................................4Application Issues...................................................................................................4Site Visit Dates ......................................................................................................4Hotel and Meeting Arrangements..........................................................................4Site Visit Schedule..................................................................................................4Results Updates (Peak Applications Only)..............................................................5

Establishing Teams and Accomodating Changes.............................................................5Team Size ......................................................................................................5Team Composition.................................................................................................5

Industry Orientation.......................................................................................................6STEP 1: Team Organization.....................................................................................................6

Get to Know Your Team..................................................................................................6Develop Initial Evaluation Schedule................................................................................7Prepare Initial Item Lead and Backup Assignments........................................................8

Understanding Your Team Member Strengths and Preferences............................8Aligning and Mentoring Team Members................................................................8Making Assignments..............................................................................................9

Conduct ‘Team Organization’ Conference Call..............................................................11STEP 2: Key Factor Development...........................................................................................11

Team Members Develop Key Factor List.......................................................................11Consolidation of Key Factor List....................................................................................12Review Consolidated Key Factor List.............................................................................12Conduct ‘Key Factor Consolidation / Finalization’ Conference Call...............................12

RMPEx Team Leadership Guide | List of Exhibits i

STEP 3: Independent Review.................................................................................................13Independent Review.....................................................................................................13

Independent Review Assistance...........................................................................13Scorebook Navigator............................................................................................13

Conduct ‘End of Independent Review’ Conference Call................................................14STEP 4: Comment Writing.....................................................................................................15

Second Training Day......................................................................................................15Comment Writing......................................................................................................... 15Key Theme Development..............................................................................................15

STEP 5: Consensus.................................................................................................................15Preparation Responsibilities..........................................................................................16Consensus Meeting Goals.............................................................................................16

STEP 6: Site Visit...................................................................................................................17Prepare for Site Visit.....................................................................................................17

Plan a Complete Schedule....................................................................................17Interview Order....................................................................................................17Interview Session Purpose...................................................................................17Pairing Team Members........................................................................................18Caucus and Walk-Around Time............................................................................18Results Chart Updates..........................................................................................18Hotel and Other Site Visit Arrangements.............................................................18Site Visit Planning Call with Applicant Contact.....................................................18

Pre-Site Visit Conference Call........................................................................................19Review of Site Visit Schedule................................................................................19Key Leader and Manager List...............................................................................20Document Request List........................................................................................20Interview Session Questions; Develop Site Visit Issue Worksheets......................20Site Visit Issue List................................................................................................20Walk Around Questions.......................................................................................21Key Themes Draft.................................................................................................21Consensus Comment Development Assistance....................................................21Other Issues ....................................................................................................21

Conduct Site Visit..........................................................................................................21Ground Rules and Behaviors................................................................................21Site Visit Dos and Don’ts......................................................................................23Sunday Team Meeting..........................................................................................24Opening / In-Brief Meeting Presentation.............................................................24Schedule and Time Management.........................................................................24Monitoring the Site Visit Issue List.......................................................................25Best Practices ....................................................................................................25

Finalizing Scorebook Comments...................................................................................25Key Themes ....................................................................................................26Returning All Documents......................................................................................27

RMPEx Team Leadership Guide | List of Exhibits ii

Site Visit Summation / Out-Brief Meeting Presentation.......................................27STEP 7: Support to the Judges...............................................................................................27

Significant Scoring Changes...........................................................................................27Scorebook Discrepancies..............................................................................................27Scorebook / Application Explanations...........................................................................28Team Leadership Call with Judges................................................................................28

STEP 8: Post Feedback Report Consultation Visit...................................................................28Attachments – Forms, Templates and Worksheets.................................................................29

Attachment 1: Team Member Information and Item Preference Template.......29Attachment 2: Evaluation Schedule Template....................................................29Attachment 3: Team Member Category Lead and Backup Responsibility

Matrix Template..........................................................................................29Attachment 4: Conference Call Sample Agendas................................................29Attachment 5: Site Visit Samples, Templates and Worksheets...........................29

List of ExhibitsExhibit 1 RMPEx Guidelines for Establishing Evaluation Teams..................................5Exhibit 2 Sample Team Member Information and Item

Preference Questionnaire............................................................................6Exhibit 3 Sample Team Member Category Assignment Matrix...................................8Exhibit 4 Common Process to Results Linkage for Peak Application Team

Assignments.................................................................................................9Exhibit 5 Pros / Cons of Announcing Assignments Before or After the

Independent Review....................................................................................9Exhibit 6 Sample Detailed Site Visit Schedule identifying Interviewees,

Interviewers, Scribes, and Time Blocks......................................................19Exhibit 7 Sample of a Key Site Visit Issues List Designed to Keep

Track of progress During the Site Visit.......................................................20

RMPEx Team Leadership Guide | List of Exhibits iii

ForwardAs the leader of a RMPEx evaluation team, the quality of the evaluation and feedback rendered by your team depends on how well the team members perform individually and collectively. It cannot be overstated how important your leadership role is for a constructive and valuable feedback report from your team – one that will assist your applicant’s performance improvement.

The RMPEx Team Leadership Reference Guide is an in-depth reference to the multiple issues a team leader might face to manage an examining team and optimize their experience as a RMPEx examiner. The tools in this guide are needed to complete a successful feedback report. The RMPEx Team Leadership Reference Guide is provided for the following purposes:

To provide an understanding of the RMPEx team leadership process; To provide an overview of the roles and responsibilities of a RMPEx Team Leader; To provide useful tips on leading a successful team, enhancing the ability to coach and

support new examiners and give them the best experience possible; To share opportunities to improve Team Leader effectiveness in the RMPEx

performance assessment process; To provide reference material and examples of reports or forms that may support Team

Leaders in managing the evaluation process and preparing their teams; and To give the best performance evaluation feedback to the applicant so that the

organization will use the information to improve its operations and achieve greater market success.

The INTRODUCTION provides useful information and tips for leading your Team to Performance Excellence, the role of the Team Leader, and working with the Applicant.

Following the INTRODUCTION, the Guide is aligned with the primary steps of the evaluation process:

STEP 1: Team Organization STEP 2: Key Factor Development STEP 3: Independent Review STEP 4: Comment Writing STEP 5: Consensus STEP 6: Site Visit and Final Report Development STEP 7: Support to the Judges STEP 8: Post Feedback Report Consultation Visit

Lastly, attached to this document are EXAMPLE Forms, Templates and Worksheets that are used during the evaluation process.

RMPEx Team Leadership Guide | Quick References iv

Quick References

Jim Walker Phone: 970 532 5191 (home) 970 222 8445 (cell)Email: [email protected]

RMPEx Web Site http://www.rmpex.org/

Scorebook Navigator Web Site

https:// scorebooknav.org

Site Visit In-Briefing Template

RMPEx web site Volunteers Examiners Team Lead Processes and Tools

Site Visit Summation / Out-Briefing Template

RMPEx web site Volunteers Examiners Team Lead Processes and Tools

RMPEx Team Leadership Guide | Quick References v

http://www.rmpex.org/

mailto:[email protected]

Team Lead Checklist

Due Date

Actions Guide Page

Notes

Review Team Lead Guide Introduction 1

Contact Team Members 6

Identify Assistant Team Lead, if appropriate 6

Develop Initial Evaluation Schedule 7

Create preliminary Item Assignments 8

Team Organization Conference Call; introductions, expectations, overall schedule

11

Team Develops Key Factor List 11

Monitor Key Factors List development progress; provide feedback

Assign or complete Key Factor Consolidation (SN Step2) 12

Contact Applicant - Introduce yourself; discuss site visit times 4

Key Factor Consolidation Conference Call (optional or provide KF list) 12

Team Performs Independent Review (SN Step 3) 13

Monitor Team Performance; provide feedback.

Contact Applicant; Finalize Site Visit Dates 4

Coordinate Arrangements / Schedule for Training Day 2 15

End of Independent Review Conference Call. 14

Move Team into Scorebook Navigator Stage 2 – Consensus (SN Step 4)

Day 2 Team Training Day 15

Team Performs Stage 2 Consensus (Comment Writing)

Contact Applicant; Send detailed Site Visit Schedule Draft, and Staff for Interviews (minimum three weeks prior to Site Visit) Confirm hotel arrangements

18

Monitor Team Performance; provide feedback, draft/send Document Request List.

Consensus Team Meeting 15

Export Pre-Site Visit Scorebook for Judge’s review

Pre Site Visit Conference Call 19

Conduct Site Visit 21

Judges Call with Team Leader 27

Post Feedback Report Consultation Visit 28

RMPEx Team Leadership Guide | Quick References vi

Introduction

REPRESENTING ROCKY MOUNTAIN PERFORMANCE EXCELLENCEYour team contributes significantly to the overall mission of RMPEx by serving as professional representatives for the program. Some guidelines on which your team members should be reminded:

Focus on the RMPEx Program as an educational program for achieving performance excellence.

Uphold the need to maintain confidentiality (especially applications and scorebooks), the appropriate use of examiner status, and the appropriate use of the RMPEx logo.

Communicate any significant issues, controversies, or changes that could impact RMPEx. Participate as a good team member, striving to achieve quality in your evaluation work

and to support team members in order to achieve the best possible feedback report and experience for the applicant.

ROLE OF THE TEAM LEADER The Team Leader guides the Team through the evaluation process, providing feedback and help to examiners along the way. The Team Leader also coordinates site visit activities with the Applicant.

Key activities accomplished by the Team Lead (or Assistant Team Lead) include: Establishing (and updating, as necessary) evaluation schedule. Coordinating team conference calls and meetings; preparation and dissemination of

meeting agendas. Assigning category leads and back-ups. Merging Key Factors into the Consolidated Key Factors List. Reviewing and monitoring Team Member progress in Scorebook Navigator (SN). Providing feedback to team members during each step of the assessment. Serving as facilitator for the Consensus meeting; acting as the Criteria “cop,” Key Factor

“cop,” Key Themes owner, or applicant champion during the Consensus meeting. Reviewing and organizing the issues documented on Site Visit Issue List. Coordinating Site Visit activities with the Applicant. Preparing Site Visit document and Key Leader list. Conducting Site Visit opening and closing summation presentations. Managing the flow and completion of work during the site visit. Maintaining positive control of Applicant-provided documents during the Site Visit. Assuming responsibility for Category 7 to ensure that all measurement data is reviewed,

understood, obtained, and commented on in the final feedback report. Serving as final editor of the Feedback report and being accountable for the accuracy of

the final Site Visit scorebook.

RMPEx Team Leadership Reference Guide | Attachments – Forms, Templates and Worksheets

A-1

Representing the Team as a resource to RMPEx judges on final scorebook issues and questions.

Participating in post feedback report conference with Applicant, if requested.

TIPS FOR LEADING YOUR TEAM TO PEAK PERFORMANCEAs the Team Lead, your team looks to you for organization, direction, and, most importantly, feedback. The following tips (lessons learned) have been compiled to help you avoid and/or overcome some of the perils of leadership.

FOCUS ON RESULTS AND PRODUCTIVITYSet clear goals and timelines in the evaluation schedule. Ensure that goals and timelines are clearly understood. Emphasize that to ensure an efficient use of time, team members need to show up on time and be available for team calls and meetings. Once these ground rules are established, let team members manage their own time. If you have a team member who is not performing, consider increased monitoring and frequent check-ins, give greater support such as suggestions for looking at case studies to improve performance, or consider whether the team member should remain on the team.

ALIGN PEOPLE WITH THE CRITERIA AREAS TEAM MEMBERS KNOW THE BESTEnsure you have the right team members in the right seats. While the “right fit” is important in all Categories, pay particular attention to Category 1 and consider leading it yourself. Evaluate the talents you have on the team and align responsibilities. Match those talents appropriately and the overall feedback to the applicant will be improved. Do your best to avoid overlaps of authority.

ALIGN PEOPLE WITH THE CRITERIA AREAS IN WHICH TEAM MEMBERS ARE PASSIONATEAnother part of getting team members in the right seats is finding what they are genuinely passionate about and seeing if they are ways to align them with responsibilities that let them channel some of that passion. Occasionally, that can mean putting a team member in an area where they do not have much experience. If their previous RMPEx work indicates that they can succeed in that role, it is usually worth it because their passion will fuel a strong desire to learn and give the best possible evaluation.

PUT YOUR BEST PERFORMERS ON YOUR BIGGEST OPPORTUNITIESWhen you feel that the applicant has a key area where feedback could propel the organization forward, step back and think about who is the best person to lead the charge. In addition to finding someone who has the talent for the work involved or who has a passion for the subject matter, you may need to support the assigned team member personally or with another team member. You should always put your best performers on your biggest opportunities.

ESTABLISH A REALISTIC EVALUATION SCHEDULECreate a culture of performance by establishing a realistic schedule and holding the team members accountable for their progress. The schedule should not be so aggressive that your


A-2

team members quickly fall behind or feel like they can never realistically achieve their responsibilities. Regularly check in with the team to make sure progress is being made so any team member difficulties can be identified and solutions found to assist the team member and maintain the team schedule. This includes use of the Scorebook Navigator for time tracking and team member follow-up if deadlines are not being met.

TRUST YOUR TEAM MEMBERS – AND LET THEM KNOW ITKnowledge workers typically have jobs that require creative solutions and decision-making. They need to stay sharp mentally to achieve top performance. The onus is on leadership to create an atmosphere that fosters and encourages that kind of creativity. One of the best things you can do is to let your team members know that you trust them and that you have faith in their ability to do the job, provide good feedback, and/or meet the deadline.

AVOID BLAMEThere are going to be times when team members do not perform as expected, and there will be things that simply do not pan out the way you had hoped. Remember that this is a highly intellectual challenge and it is likely that errors will be made, and should be dealt with privately. If necessary, inform the examiner clearly about your expectations. Do not publicly blame individuals – directly or indirectly – in meetings or team emails. If you do, you risk creating an atmosphere in which people are so afraid to make mistakes that they do not spend the time doing the proactive and creative work necessary to avoid future problems.

DO NOT PROVIDE ALL THE ANSWERS – HELP YOUR TEAM MEMBERS THINKWhile you are the leader, you don’t have a monopoly on all the good ideas. If your team members, especially new examiners, are hesitant to make decisions without asking your opinion first, you may not have properly empowered them or they may lack the confidence to proceed independently. We are asking a lot of new examiner and at times they may need additional direction and clarification from the team leader. The important thing is that a learning environment is created, rather than examiners being ‘paralyzed’ in fear of the process. Support new examiners by answering their questions and objectively provide guidance. If a pattern of ‘Ask First’ emerges, push the ball back into their court and ask them, “What do you think?” They might be surprised at first, however after you do that several times, they will, over time, be better prepared to discuss the matter and make a recommendation. That’s a good thing, as the more they understand the Criteria and application the better the evaluation and feedback will be. You need their opinions. You also need them to make some of their own decisions in order to fully participate during consensus.

BUILD CONSENSUS BY LETTING PEOPLE PARTICIPATE AND KNOW ‘WHY’One of your key leadership responsibilities is building consensus. For issues that rely on the team consensus and commitment, involve the whole team in the decision-making process. Strive for each team member to feel ownership of the final decision, solution, or idea. The more team members feel this way, the more likely they are to agree with and support the decision. Discuss the rationale, and allow all opinions to be expressed.


A-3

WORKING WITH THE APPLICANTDeveloping a good working relationship with the Applicant will enable your team to conduct a more efficient and effective site visit, which will improve the quality of your team’s feedback report. Contact the Applicant’s representative [identified in the Applicant’s Intent to Apply as the Applicant Contact Point] at the beginning of the evaluation process; develop a working relationship by reaffirming the purposes and importance of the RMPEx team’s evaluation, the overall evaluation schedule, and the site visit.

The applicant’s contact will help you with the following activities during the evaluation process: Application Issues Site Visit Dates Hotel and Meeting Arrangements Site Visit Schedule Results Updates (Peak Applications Only)

APPLICATION ISSUESIf there are any significant issues with the application, such as omitted pages, unreadable charts, etc., then bring this to their attention and ask for corrections.

SITE VISIT DATESFocus on the site visit and begin to discuss date ranges for the site visit. The more time you give the applicant to schedule the dates, the better chance that key applicant executives and staff members will be available for the site visit. Request that the site visit dates be confirmed in one to two weeks, and make sure you check with your team members to verify availability. The sooner the dates are set, the more precise your evaluation process schedule so that all team conference calls, meeting dates, and other key team issues can be addressed and established.

HOTEL AND MEETING ARRANGEMENTSThe applicant is responsible for reserving and prepaying hotel arrangements. Remind the Applicant’s contact that hotel arrangements close to the applicant’s site with an adequate meeting room and technical support should be considered early. Address the requirement for internet access to accommodate the Scorebook Navigator.

SITE VISIT SCHEDULEApproximately four to six weeks before the site visit date, email a preliminary site visit schedule to the Applicant’s contact for review. The schedule should identify by day and time the site visit topics, the staff titles that are requested for interviews, walk around times, team / group meetings, and the team’s completion schedule (Site Visit Schedule A-38 Template.) The preliminary schedule should be modified by the applicant to accommodate the schedules of their executives and staff. Coordination of the site visit schedule will be an iterative process; the final schedule, however, should be solidified two weeks prior to the site visit.


A-4

RESULTS UPDATES (PEAK APPLICATIONS ONLY)Peak Applicants are expected to update the data in Category 7 (results) prior to the site visit; request that the updates be provided to you at least one week prior to the site visit.

ESTABLISHING TEAMS AND ACCOMODATING CHANGES

TEAM SIZE The number of team members for an application evaluation and site visit is important to the quality of the evaluation and performance of the team. Even though the number of team members may vary per team and type of RMPEx level application, generally the team sizes established by RMPEx follow the guidelines in Exhibit 1.

Application Level Team Size Impacts on TeamPeak 6 to 8 members;

minimum of 5 members As Team Lead, you may elect to not be a Category

Lead, serving as the mentor and Category backup (depending on size of team)

The larger the team, the more coordination and communication required.

The smaller the team, the more time commitment required by each team member

Timberline 5 to 7 members; minimum of 4 members

Foothills 3 to 4 members; minimum of 3 members

Exhibit 1. RMPEx Guidelines for Establishing Evaluation Teams.

TEAM COMPOSITIONFor some applications, industry experience within the team may be a factor in the evaluation. 50% industry team member experience on a team would be a good goal but may not be possible, depending upon the number of applications and examiner expertise.

Changes. RMPEx team members are tentatively assigned before examiner training. There will always be situations where team composition may change

o Examiner Resignation- Examiners drop out of the RMPEx program due to personal, work, or other reasons.

o Conflicts of Interest. Although every effort is made to ensure no perceived or actual conflict of interest exists, sometimes examiners or applicants have a conflict of interest due to a prior relationship with the organization or the examiner works for a direct competitor. Contact JIM WALKER to resolve.

Performance. If an examiner is not performing and cannot meet individual responsibilities, team expectations, and deadlines, you may need to remove the examiner from the team. A poorly performing team member can have a detrimental effect on team morale and, more importantly, the quality of the team’s evaluation and feedback report. This situation should be discussed with JIM WALKER prior to making a decision.


A-5

Consider your team size and composition closely. If the size of your team falls below acceptable levels or you lose a team member with critical skills, contact JIM WALKER to discuss the situation. You may need to replace the team member, if necessary and possible, considering the available RMPEx examiner pool.

INDUSTRY ORIENTATIONIf a specific industry orientation is needed (healthcare, education, small business) you may want to consider using the appropriate industry orientation material as an educational exercise with your team. These orientations are not meant to be prescriptive or to bias the view of your examiners. Please caution your team members to use the material for general industry orientation purposes only. Industry orientation material can be found on the RMPEx website.

STEP 1: Team Organization

GET TO KNOW YOUR TEAMExhibit 2 is a sample “Team Member Information and Item Preference” questionnaire; the template provided at Attachment 1. The questionnaire is designed to capture information about your team members to aid in assigning lead and backup item evaluation areas and in developing the evaluation schedule. Distribute the questionnaire to all team members; request that responses be returned to you and then distribute to the entire team.

SELECT AN ASSISTANT TEAM LEAD Most teams will include one or more experienced examiners, in addition to you. At your discretion, you may invite one of these team members to serve as an assistant team lead


A-6

Exhibit 2. Sample Team Member Information and Item Preference Questionnaire.

DEVELOP INITIAL EVALUATION SCHEDULEUpon receipt of all Team Member Information and Item Preference Forms, draft your initial Evaluation Schedule. Identify key dates, dates that team members will not be available, and potential dates for the site visit. An annotated schedule template is provided at Attachment 2.

As a guide to developing your schedule, a national Baldrige team schedule is defined by the national office and contains the following assumptions:

Average time to complete one item (ie. 1.1, 1.2) during independent review is 2 hours Examiners will complete an average of one item a day during independent review. Item leads/backups will consolidate team findings and create comments for up to three

items in 3 weeks. Team members spend about 10 hours in the following week reviewing all comments.

Based on these assumptions, the following is a sample of completion schedules consistent with serving on a national Baldrige team:

Sample Peak Schedule:o Key Factor consolidation complete –9/15o Independent Review complete (17 items) - 10-15o Day 2 Training – 10/16o Consolidation/Comment Writing Completed (3 Items/6 team members)– 11/1o Final Comment Consensus – 11/13o Site Visit - 11/18

Sample Timberline Schedule:o Key Factor consolidation complete – 9/15o Independent Review complete (12 Items) – 10/7o Day 2 Training – 10/9o Consolidation/Comment Writing Completed (2 items/6 team members) – 10/28o Final Comment Consensus – 11/4o Site visit –11/11

Sample Foothills Schedule:o Key Factor consolidation complete – 9/15o Independent Review complete (6 Items) – 10/1o Day 2 Training – 10/8o Consolidation/Comment Writing Completed (2 Items/3 team members)- 10/22o Final Comment Consensus – 10/30o Site Visit – 11/4

The Schedule should be on the agenda for your first conference call. This high level description of the evaluation process, activities and proposed due dates will enable each team member to view all of their responsibilities and begin to coordinate their calendars with the schedule.


A-7

Most importantly, it reinforces the team members’ responsibility in the following key activity areas:

Key Factor development Independent review and the importance of pacing so that all Categories are completed Comment Writing / Consensus and being prepared to actively participate Team meetings / conference calls Tentative Site visit dates

Communicating clearly what is expected from each team member will lead to team efficiency and effectiveness and greater satisfaction for everyone. Making the RMPEx evaluation experience enjoyable and meaningful for team members is an important role of the team leader. It will produce a better feedback report for the applicant and encourage team members to grow in their knowledge and ability to provide relevant, actionable feedback and encourage the return of examiners in subsequent years.

PREPARE INITIAL ITEM LEAD AND BACKUP ASSIGNMENTSUpon receipt of all Team Member Information and Item Preference Forms, draft your initial Item Lead Assignments. Factors to consider when making assignments are listed below.

UNDERSTANDING YOUR TEAM MEMBER STRENGTHS AND PREFERENCESThis information is needed to make Category assignments for the entire process. It is important that your team members have the opportunity to let you know which categories and items they feel most comfortable leading during the Consensus and Site Visit processes.

On the Team Member Information Survey, team members are asked to rank their interest in and ability to evaluate the seven Baldrige Categories. Use this preference information as well as team member work background and experience to make your lead and backup assignments. You may want to consider being the backup for several categories if your team consists of mainly new examiners. Examiner work history and prior Criteria experience as well as individual interest in a specific Category will normally lead to a more comprehensive, quality review.

ALIGNING AND MENTORING TEAM MEMBERSIn addition to the mentoring that you and your Assistant Team Lead will provide to team members, the alignment of experienced examiners with new examiners in a mentoring role will create additional support to new examiners, if you have a sufficient number of experienced examiners to do this. The experienced team member will be able to give examination insights and suggestions that the new examiner may not have thought of or had not anticipated. This partnership truly accentuates the learning process for new examiners.


A-8

If you have a team that is primarily composed of new examiners, then you, the Assistant Team Lead, and / or other experience team member may need to provide mentoring of the new examiners. Some tips on managing this:

Assign half of the new examiners to yourself and half to the Assistant Team Leader Pay particular attention to first and second year examiners. Inform each examine regarding who his or her mentor is. Make sure examiners understand that they will be getting feedback on their

Independent Review with a focus on how well the Independent Review comments link to the Criteria and align to Key Factors.

Consider asking all examiners on the team to submit a few sample comments in advance of the Independent Review due date to gauge how well they are doing, provide feedback on their comments, and ensure they are on track to meet the deadline. Some Team Leaders have established a due date for each category that ensures the entire report will be done on time.

MAKING ASSIGNMENTSExhibit 3 is a sample matrix used to assign team member lead and backup responsibility and the matrix template is provided at Attachment 3. (This sample is based on a prior version of the Criteria.)

If you are evaluating a Peak application, then associating Process Categories with Results Category Items is helpful. The common approach to this linkage is provided in Exhibit 4.


A-9

Exhibit 3. Sample Team Member Category Assignment Matrix.

Team Members --> Nam

e

Nam

e

Nam

e

Nam

e

Nam

e

Nam

e

Lead Backup Category – Process Item 1. Leadership 1.1

1 6 6 3 1 5 Name Name 1.2

2. Strategic Planning 2.1 3 2 4 2 3 2 Name Name

2.2 3. Customer Focus 3.1

7 3 5 1 2 1 Name Name 3.2

4. Measurement, Analysis, & Knowledge Management

4.1 6 4 3 5 5 3 Name Name

4.2 5. Workforce Focus 5.1

5 7 7 4 4 7 Name Name 5.2

6. Process Management 6.1 4 1 1 7 6 4 Name Name

6.2 Category – 7. Results 2 5 2 6 7 6 [Industry] Outcomes 7.1 Name Name Customer-Focused Outcomes 7.2 Name Name Financial and Market Outcomes 7.3 Name Name Workforce-Focused Outcomes 7.4 Name Name Process Effectiveness Outcomes 7.5 Name Name Leadership Outcomes 7.6 Name Name

Process ResultsCategory 1 Leadership Item 7.4 Leadership and Governance ResultsCategory 2 Strategic Planning Item 7.5 Budgetary, Financial and Market ResultsCategory 3 Customer Focus Item 7.2 Customer-Focused ResultsCategory 4 Measurement, Analysis, &

Knowledge ManagementCat 7 Determine if 4.1 responses are

demonstrated in Cat. 7 resultsCategory 5 Workforce Focus Item 7.3 Workforce-Focused ResultsCategory 6 Process Management Item 7.1 (Product, Patient, Student Learning) and

Process Results

Exhibit 4. Common Process to Results Linkage for Peak Application Team Assignments.

There are two alternate approaches on when to announce Category assignments: BEFORE Independent Review or AFTER Independent Review. Exhibit 5 provides the Pros/ Cons of each. Since there is no right or wrong approach, consider which option you feel is most appropriate for the team. If you have a predominant number of new examiners, you may want to wait until Consensus. If you have a team of experienced examiners and talented new examiners in specific areas, then you may want to make the assignments at the start of Independent Review.

Assigning Categories to the Team Lead and Assistant Team Lead is another issue that should be considered and discussed. Some experienced team leads feel that their role is to manage the process and mentor other team members without being assigned a specific Category for Independent Review and Consensus. Other Team Leads feel they share the responsibility for Independent Review and Consensus to enhance teamwork, increases cooperation, and foster team cohesiveness. As mentioned earlier, the size of the team may dictate how this assignment issue is handled, as it may be necessary for the Team Lead to take a Category.

Announcing Assignments BEFORE Independent Review

Announcing Assignments AFTER Independent Review

PROs:Team members know their Category responsibilities in the overall planning process of the evaluation when beginning Independent Review; it allows the team members to become more comfortable with their assignment and may lead to an improved Consensus process.

PROs:Team members pay equal attention to all Categories since they do not know which Item(s) they will be responsible for during Consensus.As team leader, you have the opportunity to review the Independent Review analysis and can evaluate the quality of the Item review individually before making final assignments.

CONs:There is a possibility that examiners may pay less attention to the other Categories and prefer to put more effort on the Category for which they will be responsible during Consensus.

CONs:Fear of the unknown is worse than fear of the known – especially for new examiners; team members do may not have time to become comfortable with their assignments.

Exhibit 5. Pros / Cons of Announcing Assignments Before or After the Independent Review.


A-10

CONDUCT ‘TEAM ORGANIZATION’ CONFERENCE CALLYour first team conference call begins to develop the important team work necessary to produce a meaningful evaluation for the applicant. Attachment 4a provides a sample agenda for the Team Organization conference call.

Possible agenda items include:

1. Introductions. Let each team member introduce themselves, even though they have sent their brief biographical sketches to everyone.

2. Overall Schedule Review Draft. Discuss what the overall review schedule looks like and possible key in-person team meeting dates for Day 2 team training and consensus team meeting.

3. Item Lead and Backup Selections. If you decide to share this selection now instead of at the end of independent review, then review and discuss the assignments and make any necessary changes.

4. Industry Orientation. If needed, some orientation and discussion on the specific industry related to the applicant should be conducted.

5. Application Discussion. Some reminders to the team: (1) printing instructions on the application, if necessary; (2) reading the application completely; (3) any application problems needing correction by the applicant - like missing pages, unreadable charts, etc.; and (4) pacing the review so it does not create a major hardship when the review completion deadline approaches.

6. Development of Key Factors List. Discuss development of the Key Factor List, when it will be consolidated and by whom.

7. Access to Scorebook Navigator. Ask for and resolve any SN issues.8. Schedule for Next Conference Call.

STEP 2: Key Factor Development

TEAM MEMBERS DEVELOP KEY FACTOR LISTEach Team Member enters Key Factors into their Scorebook. Coaching new examiners is essential – they often feel as though they are doing something wrong by simply ‘cutting and pasting’ from the application. Reinforce that by compiling Key Factors, they are learning vital information about what’s most important to the applicant and familiarizing themselves with the applicant’s operating environment.


A-11

CONSOLIDATION OF KEY FACTOR LIST

After all team members have completed the creation of their Key Factors, the Team Lead moves the team into Step 2 in the SN. During the transition of Step 2, the Team Lead will assign one team member to perform the consolidation or perform the task themselves to ensure timely completion. In SN, a file on the scorebook page will appear that contains all of the Key Factors created by the Team. To consolidate the Key Factors, the assigned team member will review, align, and merge all key factors and eliminate the redundant factors.The objective is to consolidate and refine key factors.

REVIEW CONSOLIDATED KEY FACTOR LISTAfter the consolidation is completed, the Team Lead will share the consolidator’s Step 2 scorebook will all other team members. This will allow team members to review the consolidated list and provide feedback.Team members should review the list, and be prepared to discuss/ finalize the Key Factors during the next conference call if that is necessary.

OPTIONAL ‘KEY FACTOR CONSOLIDATION / FINALIZATION’ CONFERENCE CALLThe Key Factor Consolidation / Finalization conference call should occur about 1 week after the Team Organization conference call. This Conference call focuses on finalization of the Consolidated Key Factor’s list and progressing to the Independent Review. Attachment 4b provides a sample agenda for the Key Factor Consolidation / Finalization conference call. NOTE: This call may or may not be need. Possible agenda items include:

1. Finalization of the Consolidated Key Factor List. A line by line (key factor by key factor) review; further edits of the list (combining, deleting, rewording, etc.) as necessary. Note: team members can be viewing the shared Step 2 scorebook and the consolidator can make immediate updates to the Key Factor List.

2. The Independent Review Process. Review the issues that must be addressed in their evaluation of each Category in the application. Stress that Independent Review must be completed by the next conference call so Consensus can begin. The critical evaluation issues are: Key Factor References Evaluation Factors Gaps Overall Item Notes Scoring

3. Team Evaluation Schedule Review


A-12

STEP 3: Independent Review

INDEPENDENT REVIEWRemind your team members to begin reading the application completely before beginning to evaluate the application in the Scorebook Navigator. Remind the team members for Peak applications to print the results category items in color to be able to evaluate performance measure results properly.

Emphasize to team members that in their Independent Review they need to focus on how well the Independent Review comments link to the Criteria, align to the Key Factors, and adhere to the feedback comment guidelines detailed in the Examiner training material. If any new examiner had difficulty with the case study independent review for training and still feels uncomfortable, then you or the Assistant Team Lead may need to work with the examiner to provide additional guidance. Early support and training will add value to the team member’s future contribution to the assessment, site visit, and final feedback report.

INDEPENDENT REVIEW ASSISTANCEIf your team members have questions about independent review, the possible resources include the RMPEx Examiner Training Material, you and the Assistance Team Lead, and RMPEx leadership. Intervene early and do not let questions go unanswered. Examiner performance and satisfaction may be impacted and should be proactively managed to minimize this.

SCOREBOOK NAVIGATORThe Scorebook Navigator Manual provides complete instructions on team leader responsibilities and system usage. After reviewing the SN team lead instructions, it is optional for the team leader to schedule a preliminary call with JIM WALKER to ensure that all team members are set up in the system correctly and review key team lead system issues. It is suggested that this call be conducted either prior to or close to the date of initial team member contact.

Complete SN examiner training material was covered in examiner training and is immediately available on the RMPEx website for use by examiners. Reinforce the need to save comments frequently and make sure the examiner knows where to go for technical support from Jim Walker. Remember that new examiners are managing a lot of information and may need assistance with SN.


A-13

One of the great features of SN is your ability to monitor team member evaluation progress in Independent Review. You and the Assistant Team Lead should routinely check on progress and intervene with any team member who appears to be falling significantly behind in their review or is having evaluation quality issues. It is critical for all team members to complete their work fully and on time with the team evaluation schedule.

CONDUCT ‘END OF INDEPENDENT REVIEW’ CONFERENCE CALLAt the completion of Independent Review a team conference call should be scheduled to ensure all members are ready to move to Stage 2 – Consensus. Once SN is moved to Stage 2 – Consensus, the system will not allow the Team to return to the Independent Review scorebooks. Independent review responsibilities must be completed in order to move to Consensus in SN. Attachment 4c provides a sample agenda for the ‘End of Independent Review’ conference call.

Agenda items include:1. Status Check – is the team ready to move to Stage 2 – Consensus? If not 100% in

agreement that yes, the team should move to Stage 2, reset a realistic but hard deadline for completion.

2. Comment Writing Training Day and Consensus Review Dates. Remind the team to review RMPEx training materials on comment writing, to include the Writing Comments Dos and Don’ts document. Review the evaluation schedule, pointing out the timing for Consensus review and the date of the Team Consensus Meeting.

3. Consolidated Key Factors List Usage Review. You or the Assistant Team Lead should review the usage of the Key Factors during Independent Review and lead a discussion on whether the Key Factors List should be revised. If specific Key Factors are not being used, then there is no need to keep them in the Consolidated Key Factors list to be loaded into SN for use in Consensus.

4. Finalization of Category Lead and Backup Assignments. If there are changes requested to the initial Category lead and backup responsibilities, the changes should be discussed and clarified. If you did not make initial lead and backup assignments at the beginning of Independent Review, then the assignments would be made at this time.

5. Site Visit Schedule Format Review. This will give the team members a first look at the weekly schedule format of the Site Visit. Attachment 5a provides a sample Site Visit Schedule.

6. Team Evaluation Schedule Review


A-14

STEP 4: Comment Writing

DAY 2 TRAINING The second day of RMPEx training is one of the most important days of team preparation for a couple of reasons. The most important reason is that this is most likely the first time that all team members get to meet each other face-to-face and will begin to interact as a team. Making brief introductions again may be in order. The second important reason is training for comment writing. This all-day session is led by either the Team Leader or RMPEx staff and focuses on consolidating team findings, comment writing, and preparing for site visit.

COMMENT WRITINGDay 2 training is especially valuable because team members will be drafting comments for team review and discussion on the Category that they have been assigned to lead. Comments will be entered directly into the Scorebook Navigator. It is not only educational but practical in terms of completing actual consensus work for each team member.

KEY THEME DEVELOPMENTAlthough finalization of Key Themes will occur at the end of the site visit, it is good to remind all team members to be alert for possible themes that may have been identified during Independent Review as well as during the upcoming Consensus. It will make the development of Key Themes more efficient and effective for the final feedback report.

STEP 5: Consensus

The Consensus team meeting is the most important preparatory session for the site visit. It will be a day-long meeting at a facility* where all of the team members can use the Scorebook Navigator during Consensus Review. The outcome of the Consensus Team Meeting is the Preliminary Feedback report that will be used by the RMPEx Judging Team. *Contact JIM


A-15

WALKER to schedule the RMPEx Training facility or consider other sites such as public libraries or examiner workplace conference rooms.


A-16

PREPARATION RESPONSIBILITIESAt this point in the evaluation the following Consensus Review responsibilities will have occurred:

1. Assigned Category Consensus Comments and Scoring. In the Scorebook Navigator all team members have drafted their consensus comments and scoring of their assigned Criteria Items.

2. Backup and Team Member Observations and Comment Suggestions. In the Consensus comment development process, each Category Lead, not the team lead, decides when Category comments are ready for review and observation. Designated Category backups as well as other team members have given each Category lead observations on comments for review and use for comment modifications, if necessary.

3. Action Lists for Site Visit Issues. Any Item issues, documents, and possible applicant interviews have been identified in the Site Visit Issue List, Document Request List, and Site Visit Schedule preparation.

CONSENSUS MEETING GOALSConsensus Completion. It is extremely important that by the end of the Consensus Team meeting, all Consensus comments and scoring has been reviewed and agreed upon by the team. Establishing time limits and keeping track of the time taken to discuss and finalize Consensus is extremely important to reach completion. A time keeper may need to be appointed to keep the team on task.

The pre-Site Visit feedback report is sent to the Judges for their review, and the judges will complete a rubric on the quality of the Feedback Report that the team leader will share with the team. The judges will also compare this report to the final site visit feedback report and the team will be asked to explain any significant changes in the evaluation and scoring that occurred during the site visit.

Interview Questions Written. In addition to Consensus comment development, potential interview questions are also discussed at this meeting. The early identification of interview questions will prepare examiners for site visit interviews and allow team members to add interview questions from their area if pertinent.

Site Visit Schedule, Site Visit Issues, and Document Request List Review. The Team Lead or Assistant Team Lead will present drafts of site visit issues and the Document Request List compiled from team members or produced by the Team Leader. A draft site visit schedule is presented as requested by team members and team leadership observations during Consensus Review. Hotel and other logistical site visit arrangements will also be briefly reviewed.

It is important that these lists be nearing completion, as they can be shared with the Applicant Contact and used in the site visit preparation. The Applicant Contact will need sufficient time to


A-17

work with the lists and the tentative site visit schedule. Changes to the lists and schedule can be made up until the beginning and during the site visit.


A-18

STEP 6: Site Visit

PREPARE FOR SITE VISITIn order to plan an efficient and effective site visit, consider the following points:

Plan a Complete Schedule Interview Order Interview Session Purpose Pairing Team Members Caucus and Walk-Around Time Results Chart Updates Hotel and Other Site Visit Arrangements Site Visit Planning Call with Applicant Contact

PLAN A COMPLETE SCHEDULEUse the Excel scheduling template provided at Attachment 5b to prepare a detailed schedule to discuss in its entirety with the Applicant Contact to determine any conflicts or issues. This will likely be an iterative approach as the Applicant will need to schedule multiple people. Work closely with the Applicant Contact so that the schedule works well for your team and the Applicant.

INTERVIEW ORDERIn order to ensure that key executives participate in the interviews, schedule executives (Board members, Senior Leadership) as close to the beginning of the week as possible. This technique provides flexibility to re-schedule these critical interviews later in the week, if necessary. In addition, try to schedule as many of the new examiner interview sessions as possible in the early part of the week. It will give the new examiner additional time to address their site visit issues and develop their comments for the final feedback report.

INTERVIEW SESSION PURPOSETo make it easier for the applicant to understand the purpose of the interviews, provide some brief information on the topic to be addressed during the interview. For example, if you ask for a strategic planning meeting, make sure you note the purpose on the site visit schedule and work with the Applicant Contact to determine which executives and staff members are needed. You have to balance the risk of giving the applicant too much information about the Site Visit issue with the advantage of having the correct person being interviewed and being prepared to answer your questions.


A-19

PAIRING TEAM MEMBERSTeaming new and experienced examiners is advisable. It will especially be helpful if the experienced examiner is the person who provided backup support or mentoring to the new examiner in Consensus.

CAUCUS AND WALK-AROUND INTERVIEWSBe sure to schedule times for team caucuses (usually two per day) and lunch. Allow some open time to walk around the facility to conduct random employee interviews for Timberline and Peak applicants. Allow some time while on site to review documents and ask the applicant questions about those documents.

RESULTS CHART UPDATESThe Peak Applicant has the opportunity to provide the team with updated results for Category 7. Be sure to ask the Applicant Contact if there will be any updates and optimally these updates should be available before site visit.

HOTEL AND OTHER SITE VISIT ARRANGEMENTSWork with the Contact on the following issues:

1. Hotel. It is the responsibility of the applicant to book the hotel rooms and pay for them. Make sure that the applicant knows how many rooms are needed, if the rooms should be smoking or non-smoking, how many days/nights rooms are needed, and the specifications for a conference room for the team at the hotel.

2. Hotel Conference Room. The conference room for the team must be available and secure 24 hours a day and be large enough to accommodate the whole team plus their laptop computers, a printer, and all of the documentation provided by the applicant. If there are six members on the team, ask for a room and a table that will accommodate 12 comfortably. Access to a copier is also necessary.

3. Internet Access. Each team member hotel room, and the hotel conference room must have secure internet access for the Scorebook Navigator without any interference. Any charges for internet access are the responsibility of the Applicant.

4. Travel to Applicant Site. Find out the best travel directions to the Applicant’s site in advance. Car pool to the site each day.

5. Meals. Ask the Applicant for convenient suitable restaurants that provide both seated and takeout service. Takeout service may become a necessity as the site visit work progresses.

6. Applicant Site Meeting Room. The Applicant must provide a secure meeting room for use by the team during the site visit. It should have internet access and accessibility to copier and printing services.

SITE VISIT PLANNING CALL WITH APPLICANT CONTACTSend the draft site visit schedule, the requested document list, and key leader list to the Applicant Contact as soon as possible and in no case later than 2 to 3 weeks prior to the site


A-20

visit. Arrange a conference call to review the items mentioned above, and the Sunday team meeting with the Contact.

PRE-SITE VISIT CONFERENCE CALLThe objective of the Pre-Site Visit Conference Call is to review the final site visit plan and arrangements; all team members should leave this conference call with a good understanding of the site visit schedule, arrangements, and what must be accomplished to develop the final feedback report.

Attachment 4d provides an agenda for the Pre-Site Visit Conference Call; this final team call is generally scheduled 1 week prior to the site visit. The key issues covered on this call are:

Review of Site Visit Schedule Key Leader and Manager List Document Request List Site Visit Issue List Interview Session Questions Walk Around Questions Key Themes Draft Consensus Comment Development Assistance Other Issues

REVIEW OF SITE VISIT SCHEDULEThe detailed site visit schedule should be complete with all days of the schedule listing times, topics, as well as an interviewer and a scribe for each interview session. The number of interview teams will vary according to the size of your team and type of application. Exhibit 6 provides a sample of a detailed site visit schedule; a template for the schedule is provided at Attachment 5b. A schedule for a Timberline and Foothills application will vary in topics and length of the visit and would need to be modified. The excel template can be modified to develop any type of applicant site visit schedule.


A-21Exhibit 6. Sample Detailed Site Visit Schedule identifying Interviewees, Interviewers, Scribes, and Time Blocks.

KEY LEADER AND MANAGER LISTThe Key Leader and Manager list is a quick reference guide identifying the Applicant’s key executive, leader, manager, and staff names. The list is useful to your team members in the interview sessions and specifically useful during team discussions. Request the Applicant Contact to assist you in completing this as Organization Charts are often difficult to read. A template to assist with development of the key leader list is provided at Attachment 5c.

DOCUMENT REQUEST LISTVarious documents will be mentioned that explain in more detail how systems or processes work in the organization. Reviewing these documents will help the team clarify and verify explanations in the application. It is sometimes helpful to use appropriate documents in interview sessions to seek greater clarification so that correct interpretation and understanding occur on important site visit issues. If a document is requested, the team MUST review it. Remind the Applicant Contact that the documents have the same number as assigned on the Document Request List. A Document Request List template is provided at Attachment 5e.

INTERVIEW SESSION QUESTIONS; DEVELOP SITE VISIT ISSUE WORKSHEETSAs the schedule becomes refined, each team member leading a specific interview session must develop a list of interview questions to use at site visit for each interview they are assigned as the Interviewer. The Site Visit Issue Worksheet provided at Attachment 5d is designed to assist the team member to develop specific interview questions to clarify opportunity for improvement comments and to verify strength comments. An interview question should be developed on every comment in the Feedback Report. The Team Lead or Assistant Team Lead will review the interview questions prior to the site visit. Conducting interviews is important, both for the applicant and the image of the team and RMPEx. The items on the Site Visit Issue Worksheets should provide a list of interview questions for each interview session, and the Category lead will write each issue from the list as a complete question. The interview question list ensures that team members leave the session with critical questions answered and issues addressed. Make sure two copies of the Interview Questions are printed and that the format for the Interview Questions allows enough room for the scribe to take notes.

SITE VISIT ISSUE LISTCompile the Site Visit Issue Worksheets into a single list of key issues that will be verified and clarified at site visit by your team members. An example of the Key Site Visit Issues List is provided in Exhibit 7; a template for the list is provided at Attachment 5d.


A-22

Exhibit 7. Sample of a Key Site Visit Issues List Designed to Keep Track of progress During the Site Visit.

WALK AROUND QUESTIONSA list of questions should be developed and used by the team when walking around the organization to observe operations and when talking directly with employees. These questions are normally general in nature, such as where the employee works, their role, significant organizational changes, knowledge of or involvement with work process improvement teams, etc. A list of the Baldrige Program walk around questions is provided at Attachment 5g. An overview of how to interview applicant employees is provided at Attachment 5h.

KEY THEMES DRAFTAs a reminder, a Key Theme is:

1. A high-level summary strength or opportunity for improvement that is common (cross-cutting) to more than one Item or Category; and

2. Is significant in terms of the applicant’s Key Factors; and / or addresses a Criteria Core Value; and summarizes a consensus finding of your team.

The Key Themes will be included in the executive summary of the feedback report.

CONSENSUS COMMENT DEVELOPMENT ASSISTANCEIt is extremely important that a team effort on strength and OFI comment writing is expected and welcomed by all new and experienced examiners. With the time limitations of the site visit and the development of the final feedback report, all examiners acting together as a team - creating, modifying, and rewording comments - is part of every site visit. Be open to all suggestions in order to develop comments that are clear, meaningful, and constructive. Avoid extensive word-smithing due to time constraints.

OTHER ISSUESLosing a Team Member Prior to Site Visit. Be prepared for the situation where you may lose a team member due to an unforeseen work or personal reason. You and the Assistant Team Lead as well as other team members may have to modify Item assignments and site visit schedule arrangements to complete the site visit.

CONDUCT SITE VISITThe purpose the Site Visit is to clarify uncertain points in the application and to verify that the information presented by the applicant is accurate. During the Site Visit, your team of Examiners will reach agreement on the applicant’s Strengths and Opportunities for Improvement (OFIs), the resulting scoring ranges for each Category / Item, and the overall-scoring band for the application. The Stage 3 Scorebook is used by the Panel of Judges to determine the recipients of RMPEx Awards.


A-23

GROUND RULES AND BEHAVIORSFollowing are some ground rules and behaviors suggested for the team during the site visit. Please review them and discuss them with your team.

Working as a TeamThe objective is to conduct an outstanding site visit by getting the best collective effort from the team as a whole. Listen carefully to each other, and take full advantage of the diversity of the collective thinking and the broad experience of the team. Be flexible to pitch in and help each other throughout the process. Team members cannot leave until the entire job is complete. Working as a team also means that a uniform and united front will be presented during all interactions with the Applicant.

CustomersYour team has several sets of customers. The Judges are looking for an objective evaluation of the Applicant compared with the Criteria, so they can determine whether the Applicant is worthy of an RMPEx award. The Applicant is looking for insightful, non-prescriptive feedback that will help them reinforce strengths and identify OFIs. Your team members are looking for support in doing their tasks and, in most cases, a deeper understanding of the criteria and how to apply them.

Final Feedback QualityMake every effort to draft consolidated comments as if they were the final feedback report. Imagine that the Applicant is reading the comments as we write, and asking “so what?” after each one. Make it easy for the Judges to follow the trail from the application consensus report Site Visit Issues Final Feedback Report.

Category OwnershipNo one “owns” any Category or Item. Instead, try to get the best collective thinking. Cross-sharing of observations and input will be a necessity.

Time ManagementTry very hard to begin and end on time and stick to the overall schedule of events the team agrees on. It is important to document continuously. Site Visit Issues must be monitored and addressed each evening at the team meeting. If the team gets behind, it will be difficult to catch up. Therefore, try to avoid the late nights that produce fatigue and less than exceptional reporting.

Active Listening and ConsensusListen to other points of view, particularly those that are different, in addition to expressing our own. Team members can and should respectfully disagree, but the team is striving for consensus. Consensus means team members have heard, have been heard, and thus can and will support the team’s outcomes. The Team Leader must monitor time spent in discussion closely and may need to make a “last call” to bring the issue to conclusion.


A-24

Role Model BehaviorIt is important that the Applicant feels that the team is prepared, listens well and thoroughly understands their company, and that the team will be fair, neutral and objective in their interactions with them.

Have Fun and Learn from Each Other! Have fun and encourage humor at the same time the team is working hard. The team can even share some humorous moments with the Applicant, as they get to know them during the site visit.

SITE VISIT DOS AND DON’TS Do come prepared for a heavy schedule of activity beyond normal work hours. The

agenda is full, the schedule hectic, and the environment intense. Do plan to stay for the entire site visit. All team members must remain through the

completion of the final feedback report. Do exercise common sense when scenarios arise that you have not encountered

previously. Do what makes sense and is consistent with RMPEx policies, practices, and Code of Ethics. Do discuss issues with the team leader.

Do ask for whatever information is needed to clarify or verify your assigned issues. Ask spontaneous questions. However, be realistic, and do not place an undue burden on the applicant by requesting things that are not necessary.

Do adhere to the interview items, but be flexible. It is vital for the applicant to feel there were sufficient opportunities to “tell its story.”

Do be alert to any response or lack of response that may affect the Site Visit Team’s agenda or approach. Let the Team Leader know of the findings so that a change in the agenda or approach can be considered.

Do take thorough notes for documenting findings. Note the kinds of things that will help the applicant via the feedback report and will assist the Judges in understanding the applicant’s processes and / or results. When assigned as a scribe during an interview, take good notes.

Do participate in daily meetings and debriefings to share information and impressions, to ensure that all relevant information is obtained, to ask questions of team members about their interviews, and to adjust strategy as needed.

Do return all applicant materials to the applicant at the end of the site visit. Do bring your RMPEx pin and represent yourself only as an Examiner when at the

applicant’s site(s). Don’t contact the applicant before the site visit unless you are the Team Leader or

Assistant Team Leader. Don’t depart before the Site Visit Scorebook is finished. Don’t take cameras or video recorders to the applicant’s site(s). Don’t take notes if you are assigned as the interviewer. Keep good eye contact and let

the scribe take notes. Don’t discuss any of the following with the applicant:


A-25

personal or team observations, findings, conclusions, or decisions, whether in a critical or complimentary way

practices of other applicants team observations about other applicants names of or any other information about other applicants team member’s personal or professional qualifications or any personal information information about your own organization

Don’t drink alcohol until the Site Visit feedback report is completed. Don’t show or give verbal or nonverbal feedback during interviews. Do not let the

applicant’s representatives know your assessment of their responses. Don’t interview consultants, customers/students/patients, or suppliers unless an

exception has been identified by your Team Leader. Don’t hold debriefings, meetings, or discussions of the site visit in an open area. Don’t take applicant materials, reports, documentation, etc., off-site unless doing so is

essential and the applicant agrees to allow the materials off-site. Don’t write on any of the applicant’s materials. Don’t accept gifts of any sort. Don’t bring family members or friends on site visit trips. Don’t interact with the applicant after leaving the site.

SUNDAY TEAM MEETINGBe sure to plan the arrival of your team by 1 or 1:30 PM so they can get settled and ready for a 3 to 4 hour Sunday afternoon team meeting. The purposes of the meeting will be to:

Meet with the Applicant Contact. This will likely be the first time any team member will have personally met the Contact. It will also give the team and the Contact the opportunity to address any significant site visit issues.

Review the Schedule. If there are any schedule changes required on the first day, the Contact and team needs to know about these. The schedule goal for each day is for your team to know exactly who they will be interview and what topics will be discussed on the next schedule day.

Review Documents. The Contact should have all requested documents in the hotel meeting room by the time of the team’s arrival. The majority of the Sunday meeting time will be devoted to reviewing documents to address site visit questions and plan for the next day’s interviews that may require documents at the meetings. Make note of any missing documents or confidential documents that must be reviewed onsite.

Review Any Updated Results Charts. The Contact should have provided any updated results charts along with the documents in the hotel team meeting room.

OPENING / IN-BRIEF MEETING PRESENTATIONThe Site Visit In-Brief provides the ‘audience’ with a general overview and value of the Examination Process. The ‘audience’ is invited by the Applicant, and is usually comprised of the Applicant’s senior leadership team and key members of the application writing team. As the Team Lead, you will normally make the presentation. Be sure to introduce your team (name


A-26

only) and acknowledge the applicant’s leadership, especially the CEO and Contact. The template for the Site Visit In-Briefing is available on the RMPEx web site [http://www.rmpex.org/

SCHEDULE AND TIME MANAGEMENTAt the end of each day the Team Lead and / or Assistant Team Lead should meet with the Applicant Contact to review the next day’s interview schedule. Any changes suggested by a team member or the Applicant should be addressed and accommodated in the schedule. Additional interviews and documents can be discussed during these meetings. Be sure to print a new schedule for each team member if changes are made for the next day. Each team member should have a correct schedule at the beginning of the site visit day.

It cannot be over-emphasized that you stay on track with the schedule as much as possible. Moving interviews to a later part of the week may cause scheduling difficulties and a delay in completing the final feedback report. Your team’s objective is to complete the feedback report as early as possible the evening before the final days closing summation conference with the Applicant.

MONITORING THE SITE VISIT ISSUE LISTEither you or the Assistant Team Lead should be continuously monitoring the specific site visit issues for each examiner, making sure that all have been addressed during the interview sessions. This monitoring can become a normal part of de-briefing at the end of each day. For Peak applicants, by Thursday afternoon’s team session to begin finalizing the feedback report scorebook, all issues must be clarified and verified.

BEST PRACTICESIf applicable, lead the team in reaching Consensus on one to three best practices that seem to be evident in the application evaluation and verified during site visit. The purpose of this recognition is to identify mature practices that enable an organization to achieve superior results in key stakeholder areas. The organization may still lack maturity or deployment-learning-integration in other management system components.

The following Criteria are to be used when selecting best practices for consideration: The practice is documented and repeatable. It is measurable. It can be readily linked to a desirable result. If scored as a stand-alone item, it would attain a minimum score of 50%. The organization is willing to share the practice with external organizations. The practice would not reflect unfavorably on RMPEx or its processes. If the practice

were applied by another organization, it would be reasonably assumed to produce a predictable result.


A-27

FINALIZING SCOREBOOK COMMENTSThe Scorebook Navigator Final scorebook enables the team to work efficiently in team meetings at the hotel, at the applicant site, or in individual hotel rooms as strength and OFI comments are being finalized. Encourage each Category lead to begin drafting comment revisions each day during the site visit when time permits. As in Independent Review and Consensus, it allows you the ability to monitor team member work on revising final Item comments.

The Team Lead, Assistant Team Lead, and experienced backup examiners must provide as much assistance as needed and desired to the new examiners as their comments are being finalized. Sometimes the Category lead, backup or Team Lead drafting revised comments in a hotel room prior to presenting them to the team is productive, especially if the new examiner is struggling with a specific comment. The Team Lead can also provide guidance for examiners such as the use of Baldrige Case Studies to elicit verbiage that is useful to initiate comment writing if a new examiner is unsure of how to write a comment.

Be careful about sending new Examiners off to write feedback individually since it may not be an effective use of time. Writing final feedback is an art rather than a science and is learned through experience. A new Examiner may spend several hours developing feedback that still requires extensive editing by the team. This is a function of a lack of experience rather than a lack of ability. Remember that statements such as “it is not clear” or “it is not apparent” should be removed since the site visit will resolve these issues.

Managing the meeting time to finalize the scorebook is critical. The goal is to have comment revisions made during the site visit as issues are clarified/verified, so that the final evening will not be ending at midnight. All comments need to be clear, concise, and Criteria based. Comment writing guidelines must be followed.

There are several techniques and ground rules that can be used to effectively manage discussions during the Site Visit scorebook preparation process:

1. If a team member does not like a comment, the team member must suggest an improvement or remain silent.

2. The team will be responsible for creating a final feedback report that is suitable for review by the applicant.

3. Time limits and a schedule should be set for discussion on each item and comment. 4. Discussions must be limited to the item and comment being discussed. There can be a

tendency to get into lengthy discussions on interesting topics that have minimal impact on the final Site Visit scorebook. Make sure discussions are focused on the Criteria, Key Factors, and Site Visit findings related to the item being discussed.

If an Item score changes equal to or is greater than 15%, then the reason(s) for the change must to be thoroughly documented on the Site Visit Item Scoring Change Worksheet so the Judges


A-28

can understand why the large change has been made. The Site Visit Item Scoring Change Worksheet is provided at Attachment 5j.

KEY THEMESThe team should review, revise, and finalize the Key Themes developed during Consensus in the Scorebook Navigator. As a reminder, the Key Themes should summarize the most important strengths and significant concerns. The content of the Key Themes Worksheet should answer the following three questions:

What are the most important strengths or outstanding practices (of potential value to other organizations) identified?

What are the most significant concerns, weaknesses, or vulnerabilities identified? Considering the organization’s Key Factors, what are the most significant strengths,

opportunities, vulnerabilities, and/or gaps (related to data, comparisons, and linkages) found in its response to Results Items, if applicable to the evaluation type?

RETURNING ALL DOCUMENTSBe sure that all team members return any site visit documents to the secure hotel meeting room and notify the Applicant Contact the documents will be ready for pickup on the morning of the Summation or Out-Brief Presentation.

SITE VISIT SUMMATION / OUT-BRIEF MEETING PRESENTATIONThe purpose of the Site Visit Summation / Out-Briefing is to provide a very broad, top-level view of your final report. Scores (or scoring ranges) are not provided during the out-brief. The Team Leader generally prepares and delivers the briefing. The template for the Site Visit Summation / Out-Briefing is available on the RMPEx web site. If appropriate for the organization, share any stories or anecdotes about the Site Visit that might provide some humor and/or show the organization that you really learned about them. The intent of sharing stories is to break some of the tension at the end of a very hectic and stressful week.

Most importantly, encourage the Applicant to agree to a post-feedback report consultation after the organization receives its feedback report. At that time the details of the feedback report can be discussed fully so that the Applicant will clearly understand specific feedback issues. The consultation is described in Step 8.

Be sure to thank the CEO, the executive leadership team, and the Contact for their assistance and cooperation. Encourage their continuous journey to excellence and support for RMPEx and attendance at the Quest Event.

STEP 7: Support to the Judges

How can you and your team contribute to an effective judging process?


A-29

SIGNIFICANT SCORING CHANGESAs was mention in Step 6, document evidence from site visit to justify changes to final feedback report scoring changes equal to or greater that 15% from the initial Consensus scoring [Step 5] for any Item. Use the Site Visit Scoring Change worksheet at Attachment 5i to document changes. Most importantly, be prepared to discuss scoring changes on a conference call with the judges, providing rationale for changes and clarification of comments.

SCOREBOOK DISCREPANCIESEliminate discrepancies between Strengths and OFIs and the associated scoring for the Item. Ensure consistency between feedback comments and scoring so that both the Judges and the Applicant will understand the effect of Item comments on its scoring.

SCOREBOOK / APPLICATION EXPLANATIONSProvide other insights where the application may not have given an entirely correct or complete picture on the organization meeting specific Category Item Criteria. Examples discovered at site visit may include:

visible employee morale issues; interview explanations from the applicant that conflict with or omit key application

information; consistency of responses among applicant leadership and staff; and / or documentation that contradicts the application.

These observed insights help the Judges better understand your comments and scoring.

TEAM LEADERSHIP CALL WITH JUDGESThe Team Lead should be available for a mandatory call with the Judges before they make a final recommendation on feedback and scoring. If the Team Lead cannot be available for the call due to illness, vacation, or other important reason, the Assistant Team Lead or a very experienced team examiner can fill in for the Team Lead on the call with the Judges. The Judges will schedule the call.

STEP 8: Post Feedback Report Consultation VisitOne of the most meaningful experiences for the Applicant and the Team Lead is conducting a post- feedback report consultation. Many times applicant executives will not fully understand what is meant by or the impact an opportunity for improvement comment or Key Theme can have on the organization. This post-feedback report session allows you the ability to be “prescriptive” and to provide the detailed feedback explanations and examples that are often missing in feedback report comments. This discussion with senior leadership normally brings great insight into the importance and impact of the Baldrige Criteria on the applicant’s operation and performance improvement. The session also brings to light the important role that RMPEx plays in helping prepare the applicant to advance to its next level of higher performance and results.


A-30

A PowerPoint presentation should be prepared using the Out-brief template. Particular attention is paid to OFI’s so that is where the most time will be spent by the applicant. The Team Lead may want to consider using examples from Baldrige recipients to demonstrate an process approach which may be used by the applicant to improve performance. This information can be found on the Baldrige homepage, click on “Award Recipients” tab.


A-31

Attachments – Forms, Templates and Worksheets

ATTACHMENT 1: TEAM MEMBER INFORMATION AND ITEM PREFERENCE TEMPLATE

ATTACHMENT 2: EVALUATION SCHEDULE TEMPLATE

ATTACHMENT 3: TEAM MEMBER CATEGORY LEAD AND BACKUP RESPONSIBILITY MATRIX TEMPLATE

ATTACHMENT 4: CONFERENCE CALL SAMPLE AGENDAS a. Team Organization Conference Call Agendab. Key Factor Consolidation and Finalization Conference Call Agendac. Completion of Independent Review, Move to Consensus? Conference

Call Agendad. Pre-Site Visit Conference Call

ATTACHMENT 5: SITE VISIT SAMPLES, TEMPLATES AND WORKSHEETSa. High Level Site Visit Schedule Template b. Detailed Site Visit Daily Schedule Templatec. Key Leader and Manager Interview List Templated. Site Visit Issues Worksheet Templatee. Key Site Visit Issues List for Verification and Clarification Templatef. Site Visit Document Request Templateg. Sample Baldrige Program Generic Walk-Around Questionsh. How To Interview Applicant Employees j. Site Visit Scoring Change Worksheet


A-32

ATTACHMENT 1: TEAM MEMBER INFORMATION AND ITEM PREFERENCE TEMPLATE

Rocky Mountain Performance Excellence

Team Member Information and Item Preference

Name:

1. Current Employer (short description of current job/responsibilities):

2. Experience in the Baldrige Review Process (State and/or National):

3. Work Experience in the Baldrige Categories:

4. Please rank your ability to evaluate applications in the following areas: 1=best (use no number more than once)

__ Leadership__ Strategic Planning__ Customer Focus__ Measurement, Analysis, and Knowledge Management__ Workforce Focus__ Process Management__ Results (Peak Only)

5. Background/Experience Relevant to the Applicant’s Business:

6. Personal Note (optional):

7. Site Visit Week Not Available in:

August ____________________________________________________

September ____________________________________________________

After completing the form, please email it to all team members.


A-33

ATTACHMENT 2: EVALUATION SCHEDULE TEMPLATE

Date Time Call in # / Location Activity‘Team Organization’ Conference Call

Introductions Schedule Review

Complete Key Factor DevelopmentCompile / Distribute Consolidated Key Factor ListComplete Team Review of Consolidated Key Factor List‘Key Factor Consolidation / Finalization’ Conference Call

Consolidated Key Factor List Item Assignments Questions / Concerns Schedule Review

Complete Independent Review (ADLI – LeTCI Analysis)‘End of Independent Review’ Conference Call

Independent Review Progress Report Item Assignments (if not disclosed before

Independent Review) Questions / Concerns Schedule Review

Independent Review Closed in Scorebook NavigatorTraining Session Part 2 – Comment Writing

Comment Writing – How To Site Visit Planning Questions / Concerns Schedule Review

Complete Comment WritingTeam Observations Entered; Item Leads provides disposition on all observationsIdentify Site Visit Issues; complete:

Site Visit Issue Worksheet Document Request List

Consensus Team Meeting Review / obtain consensus on ALL comments Finalize Site Visit Issue Worksheets Finalize Document Request List

‘Pre-Site Visit’ Conference Call Detailed Site Visit Schedule Review Interview Questions Discussion Walk-Around Questions Accommodations and Travel Arrangements

Site Visit


A-34

ATTACHMENT 3: TEAM MEMBER CATEGORY LEAD AND BACKUP RESPONSIBILITY MATRIX TEMPLATE

Note: Excel spreadsheet; double click to access.


A-35

ATTACHMENT 4: CONFERENCE CALL SAMPLE AGENDAS

a. ‘Team Organization’ Conference Call Agenda

TEAM LEAD PREP WORK: Coordinate date / time / call in number with team members Distribute and have returned before meeting the ‘Team Member Information

and Item Preference’ data Prepare / distribute tentative evaluation schedule

AGENDA: Welcome Team Member Check In Introductions Access to Scorebook Navigator Key Factor Development – Questions / Concerns Review of Evaluation Schedule – Near Term Actions


A-36

b. ‘Key Factor Consolidation and Finalization’ Conference Call Agenda

TEAM LEAD PREP WORK: Remind team members (date / time / call in number) Consolidate Key Factor List, Share Step 2 scorebook for Review

AGENDA: Welcome Team Member Check In Review / Finalization of Consolidated Key Factor List Item Lead and Backup Selections (if disclosed prior to Independent Review) Industry Orientation Discussion (If Needed) Independent Review Analysis – Questions / Concerns Review of Evaluation Schedule – Near Term Actions


A-37

c. ‘Completion of Independent Review, Move to Consensus?’ Conference Call Agenda

TEAM LEAD PREP WORK: Remind team members (date / time / call in number) Review Key Factor Utilization Coordinate 2nd Training Session with Jim Walker / Kim Griffiths

AGENDA: Welcome Team Member Check In Status Check – is everyone finished with their Independent Review? Item Lead and Backup Selections and/ or Updates (if not disclosed prior to

Independent Review) 2nd Training Session – Everyone needs to be there Review of Evaluation Schedule – Near Term Actions

TEAM LEAD POST WORK: Move to Scorebook Navigator Stage 2 – Consensus on agreed to date.


A-38

d. ‘Pre-Site Visit’ Conference Call

TEAM LEAD PREP WORK: Finalize:

Site Visit Schedule and Arrangements with Applicant Consolidate Site Visit Issue List Consolidate Document Request Lists

Distribute to Team

AGENDA: Welcome Team Member Check In Review Site Visit Schedule Review Site Visit Issues List Review Document Request List Interview Session Question Progress Discussion Walk-Around Questions Key Themes Draft Hotel and Other Logistical Arrangements Site Visit – Questions / Concerns

TEAM LEAD POST WORK (ACTION ITEMS):


A-39

ATTACHMENT 5: SITE VISIT SAMPLES, TEMPLATES AND WORKSHEETS

a. High Level Site Visit Schedule Template



A-40

b. Detailed Site Visit Daily Schedule Template


DAY / DATETeam 1 Team 2 Team 3

Team Member Names Team Member Names Team Member Names7:30 Team Travel to Applicant's Facility7:458:00 Team meets with Applicant Representatives and Escorts; Review protocol (i.e., badging) 8:158:308:459:009:159:30 Interview with:9:45 Interiewer:

10:00 Scribe:10:1510:30 Interview with:10:45 Interiewer:11:00 Scribe:11:1511:30 Interview with:11:45 Interiewer:12:00 Scribe:12:1512:30 Team Lunch12:4513:00 Interview with:13:15 Interiewer:13:30 Scribe:13:4514:00 Interview with:14:15 Interiewer:14:30 Scribe:14:4515:00 Interview with:15:15 Interiewer:15:30 Scribe:15:4516:00 Interview with:16:15 Interiewer:16:30 Scribe:16:45 Team lead meets with Coordinator17:00 Team Travel to Hotel17:15

Team Break17:3017:4518:00

Team Dinner

18:1518:3018:4519:0019:1519:3019:45

Team Work Meeting20:0020:1520:30

Opening Breakfast MeetingCPEx Team makes 20 minute presentation - Team Lead

CEO makes 20 minute presentation (optional)


A-41

c. Key Leader and Manager Interview List Template



A-42

d. Site Visit Issues Worksheet Template

Item #:

Strength or OFI:

Comment / Issues / Gaps (from SBN):

Enter comment / issue / gap from SBN

Who to Interview: List positions or names (if known) of personnel to interview

Potential Questions to Verify / Clarify:

Draft questions which will enable you to verify / clarify the Strength or Opportunity for Improvement.


A-43

e. Key Site Visit Issues List for Verification and Clarification Template



A-44

f. Site Visit Document Request Template



A-45

g. Sample Baldrige Program Generic Walk-Around Questions

The questions below are samples from the Baldrige Program; these generic ‘walk-around’ questions aid the team understand the overall culture of the organization.

In what part of the organization do you work? How long have you been with the organization? How long have you been in your current position?

What are the most significant changes that you’ve experienced in the last 2 to 3 years?

Do you have direct interactions with your customers and/or stakeholders? How often? Could you describe a couple of examples of these interactions? In general, do you know what your customers and/or stakeholders expect from you?

What departments or groups do you depend on to do your job? How is the work allocated to you? Do you provide these groups feedback on what improvements could be made? How often? How (formal/informal feedback)?

Are you involved in community activities? Do these involve time off from work? Does the organization allow you time/pay you while you volunteer?

Are you currently participating on any work teams? How long have you been a member? What is the team’s mission/role? Are your team activities worth the time you spend? Why?

How do you share information with others or receive information from others to help you?

When was the last time you attended a formal training class? Topic? Length? What knowledge from the training were you able to use back on your job?

Do you receive information about the organization’s key strategic objectives? How (e-mail, newsletters, group meetings, etc.)? Which is most effective? Are there other ways you would like to get information and/or other information you would like to have?

Do you receive information on key organizational results? How do you use this information to make decisions?

How often do you see your direct supervisor? His or her supervisor? Under what circumstances?

If you could change one thing about this place, what would it be and why? If you want to change a process, where would you go to do this? What is the name of your performance improvement program?

Who are your major competitors? What do they do better than you? Are you aware of any efforts to improve in these areas?

What concerns do you have about the future of this organization? Have you shared these with your senior leaders? Have they asked you for input? How/how often?

How do you fit into the organization’s strategic plan?


A-46

h. How To Interview Applicant Employees(Note: interview in pairs)

Be prompt in starting the interview.

Introduce the team. " Hello, I'm __________ from the Rocky Mountain Performance Excellence site visit team."Explain the roles of all team members present during the interview (e.g., recorder, observer of the interview process, etc). Let them know you will be taking notes.

Ask the person's name if it is not offered and ensure the interviewee is an employee of the Applicant.

Ask the persons some questions about his / her background to help put them at ease.

Begin the interview by telling the Applicant that the Examiners are interviewing personnel to assist them in fully understanding the organization’s processes and approaches.

Ask: "Has anyone else from the site visit team spoken to you?"

Keep a separate page for each person with whom you talk. Note their name, their department or unit, and other pertinent information. Keep a separate page for each SVI.

Ask simple, straight forward questions. Ask questions using the Applicant's language. Avoid RMPEx or other types of jargon.

Do not ask leading questions and be careful not to inadvertently prompt answers. For example, you should ask: "How often does the planning team meet?" rather than, "Does the planning team meet every week?" Do, however, ask follow up questions that arise as the result of your prepared questions.

Be sensitive to how you are coming across to the interviewee(s), and seek to understand their non-verbal communication with you as well. Understand that the way you query will impact the Applicant and their response.

Avoid giving any indication about how the site visit is going. You may be asked, "how are we doing?" Examiners are not allowed to answer that question. The response should be: "We are not permitted to discuss the progress of the site visit, but as you probably know, we are here to clarify and verify what is in the application, and we thank you for your cooperation." You may compliment them on their hospitality, flexibility and cooperation.

Ask the person if they have anything they would like to add. You may have missed something the Applicant feels is vital.

Thank the interviewee for their input and communicate appreciation for the Applicant's efforts.


A-47

i. Site Visit Scoring Change Worksheet

Site Visit Item Scoring Change Worksheet

Use only for Item scores that have changed by 15% or more from Consensus. To explain scoring changes to the Judges, include observations, specific answers, and/or updated results found during the site visit that resulted in the scoring change. Be specific and include data as appropriate.

Item # Consensus Score Site Visit Score

Rationale for scoring change











RMPEx Team Leadership Guide | Attachments – Forms, Templates and Worksheets A-48

Date post:	07-Feb-2018
Category:	Documents
Upload:	vocong
View:	213 times
Download:	0 times