Examiner Training July 2011

The ONE Award will identify and promote best practices in non-profit organizations and recognize our region’s Organizations of Noteworthy Excellence

Agenda

• Overview• Roles, Responsibilities, Expectations

• 2011 Criteria

• Process• Independent Review

• Consensus

• Site Visit

• Questions, Next Steps

Handouts

• Slides

• Glossary of Key Terms

• 2011 Criteria

• Independent Review Worksheets

• Share Food – Organizational Profile

• Conflict of Interest/Code of Ethical Conduct

Process Timetable

• Applications Due – July 15• Applications Distributed – July 21• Consensus Review Report Due – August 25• Site Visit Training – August 31• Site Visits Conducted – September 12-30• Site Visit Reports Due – October 7

Process Timetable

• Honorees Selected – by October 28• Feedback Reports Delivered – by November 15• Recognition Event – January 2012

Key Roles

• Applicants• Examiners• Selection Panel• Award Process Team

Examiner Standards

• Confidentiality• Conflict of Interest• Code of Ethical Conduct

• Signed Statement Required

New in 2011

• Biennial Cycle• Streamlined Criteria• Shortened Application

• Two-page limit for Organizational Profile• Six-page limit (total) for Categories 1-3

• Applicant Workshop Series• Standards-Based Recognition

Criteria Characteristics

• Organizational Profile• Three Categories – People, Principles, Process• Approach-Deployment-Results (ADR)• “What” Questions• “How” Questions• Non-prescriptive• Results emphasis

Organizational Profile

• Key Characteristics• Programs/services• Purpose, Vision, Mission, Values• Customers and stakeholders• Suppliers/collaborators• Workforce• Accreditation, certification, regulators• Governance system• Major funder/foundation relationships

Organizational Profile

• Organizational “biography”• Accept at face value• Resource for evaluation of criteria responses• Two-pages, text or bullet format• Only source of “biography” information – cannot

go to web site to learn more

Organizational Profile

• Competitive Environment, Challenges• Key competitors• Key competitive success factors• Competitive/comparative data sources• Core competencies• Strategic advantages, challenges• Operational challenges• Human resource challenges

Categories

• Category 1 – People• Leadership, Workforce, Governance, Related Results

• Category 2 – Principles• Customer Focus, Strategic Planning, Organizational

Performance Review, Knowledge Management, Related Results

• Category 3 – Process• Value Creation, Support, Integration of

Suppliers/Collaborators, Related Results

Reviewing Smaller Organizations

• Size does not affect criteria applicability• Size does provide context for expectations• Systems likely to be less formal• Resource capacity may be limited• Have had to make choices due to resource

limitations

Important Terms/Concepts

• “Key”• System/process• Senior leaders• Results – actual data• Outputs and outcomes• Trends

Independent Review

Organizational Profile

• Become familiar with the organization by thoroughly reviewing Profile prior to working on the Category analysis

• Convert/condense to “key” bullet items (1-2 pages) to support analysis

• Applicant’s have been trained in what you need/expect – revise and further condense if necessary

• Share Food example

Category Review

• The applicant is the customer of your work• Selection panel depends exclusively on

examiner team’s work• This is not a compliance assessment – we are

seeking to help them understand where they have strengths to build on and opportunities for improvement

Category Review Process

• Review the criteria and keep them handy for your analysis process

• Use the Glossary to help you understand the criteria requirements

• Read the application completely prior to beginning to detail your comments

• Rely on the Organizational Profile for reference and background

Approach-Deployment-Results

• Approach – methods used to address criteria requirements

• Deployment – the extent to which the approach is implemented

• Results – outputs/outcomes achieved through approach-deployment efforts related to criteria requirements – these are actual data, not anecdotes or subjective statements concerning effectiveness

Category Review Tips

• Focus on criteria requirements

• Look for “action” verb in criteria questions to clarify what is expected

• Keep the key factors from the Organizational Profile in mind – what is important to this applicant

• Review the “small organization” tips

• Avoid generalized assumptions

• Benefit of the doubt

• Applicants want actionable feedback – they need to know what to work on

Worksheets – Analysis Process

Key Process Evidence of systematic approach Evidence of deployment

Deployment of Mission, Vision & Values

Two-way communication

Promoting legal and ethical behavior

Ensuring alignment with community goals

Creating sustainability

Item 1.1 – How do senior leaders lead?

Worksheets – Analysis Process

Key Process Evidence of systematic approach Evidence of deployment

Deployment of Mission, Vision & Values

Discussed at staff meetings, used as decision criteria in management meetings

Published in newsletters and brochures, staff members post in cubicles

Two-way communication

Forums held with employees to seek input, informal walk-around process

Monthly all-staff meetings, employee focus groups, talk one-on-one

Promoting legal and ethical behavior

Values include commitment to ethics, ethical standards documented

Discussed in staff meetings, shared at volunteer orientation

Ensuring alignment with community goals

Management reviews as part of organizational performance review

Discussed in staff meeting and in volunteer orientation

Creating sustainability

Key factors reviewed annually by Board and senior leaders

Leaders attend futurists conference, developing succession strategy

Item 1.1 – How do senior leaders lead?

Worksheets – AD ScoringItem 1.1 – How do senior leaders lead?

0-5% 10-25% 30-45% 50-65% 70-85% 90-100%

Approach Not addressed

No systematic approach, anecdotal

Systematic approach for basic

Systematic approach - all

Mature approach for all

National role model

Deployment No evidence

Early stages

Deployed in most areas

Fully deployed, few gaps

Fully deployed, no gaps

No gaps or weakness, role model

Worksheets – AD ScoringItem 1.1 – How do senior leaders lead?

0-5% 10-25% 30-45% 50-65% 70-85% 90-100%

Approach Not addressed

No systematic approach, anecdotal

Systematic approach for basic

Systematic approach - all

Mature approach for all

National role model

XDeployment No

evidence Early stages

Deployed in most areas

Fully deployed, few gaps

Fully deployed, no gaps

No gaps or weakness, role model

X

Worksheets – Results AnalysisItem 1.5 – People results

Measurement Provided/Not Provided

Levels Trends Comparisons

Leadership effectiveness

Workforce satisfaction, etc.

Fiscal Accountability

Compliance, legal and ethical behavior

Worksheets – Results AnalysisItem 1.5 – People results

Measurement Provided/Not Provided

Levels Trends Comparisons

Leadership effectiveness

360 evaluation results

Overall ranking of 4.2 on a 5 point scale

Increased from 3.8 in 2009

National average at 3.9

Workforce satisfaction, etc.

Employee and volunteer satisfaction

Volunteer satisfaction at 3.1 out of 5

One year’s data only

None provided

Fiscal Accountability

Audit and internal assessment

Zero exceptions on 2010 year-end

Internal review - fewer exceptions than in 2009

None

Compliance, legal and ethical behavior

Times in compliance, legal/ethical breaches

Four significant compliance violations

2009 and 2010 showed zero compliance issues

Similar providers averaged less than 2 issues

Worksheets – Results ScoringItem 1.5 – People results

0-10% 15-35% 40-60% 65-85% 90-100%

Levels No results Some good performance

Good performance for most areas

Good performance all areas

National role model performance

X

Trends No trend data

Little trend data, some adverse

Positive trends in key areas

Strong positive trends

Exemplary performance trends

X

Comparisons None provided

Little provided Some provided

Good comparative performance

Benchmark leadership

X

OverallXX

Consensus

Consensus Process

• One set of comments for the team• Strengths and opportunities for improvement

(OFI)• Single “score” for the team on each item• Identify site visit issues• Submit feedback report with draft of strengths

and OFIs by August 25

Comments – Feedback Report

• Most important Strengths• Most important OFIs• Criteria-based• Actionable but not prescriptive• Complete thoughts – 2-3 sentences• 4-6 per category for AD items (include both

strengths and OFIs)• 3-5 per results item – 1.5, 2.6, 3.4 (include both

strengths and OFIs)

Comments – Feedback Report

• Refer to “the applicant” but not their name• Avoid prescribing solutions or making subjective

judgments – state the specifics of the strength and/or OFI

• OFIs often need a “so what” or “so that…”• Watch out for contradictions• Don’t comment on application quality or

presentation• Make sure an independent reader can

understand your comment

Comments – Format

• All comments need an item number (1.1, 2.3, 3.4, etc.)

• Bold the most significant comments • Strengths first, OFIs next• Avoid “would,” “could” or “should” • Non-prescriptive – keep objectivity• Professional, positive tone

Developing Comments

• Preliminary comments based on independent review worksheets

• Final, revised comments based on site visit findings

Developing Comments

• Possible comments relate to:• Approach-related strength or OFI• Deployment-related strength or OFI• Gap in responding to criteria requirements (not

apparent that the applicant has a systematic approach to…)

• Potential role model practice – rare but important to highlight – make sure to present in bold type

• Acknowledgment of anecdotal response while lacking a systematic approach

Sample Comments

• Item 1.1 – The applicant’s senior leaders and Board members review the organization’s Vision, Mission and Values statements as part of the annual strategic planning process. Multiple communications methods, such as monthly staff meetings, one-on-one meetings and the monthly newsletter are used on a recurring basis to make sure that all volunteers and staff members understand these three key statements. (Strength)

Sample Comments

• Item 1.4 – The applicant ensures fiscal accountability throughout the organization through a series of periodic internal audits conducted by key staff as well as an annual independent financial audit. Audit results are reviewed within 10 days of receipt by senior leaders, and the Board reviews these results as part of each quarterly meeting. (Strength)

Sample Comments

• Items 3.1 & 3.3 – The applicant reviews its core competencies – Service Delivery and Service Design – during the Strategic Planning process and relies on these competencies to strengthen its 3 key value creation processes – needs assessment, service design and care delivery. These processes are improved annually utilizing input from customers, employees, volunteers and other key stakeholders, as well as data developed from customer feedback instruments. (Strength)

Sample Comments - Results

• Item 1.5 – The applicant’s annual workforce satisfaction survey indicates that overall employee satisfaction (those giving a score of “5” on a scale of 1 to 5) has increased from 41% in 2005 to 68% in 2008. Satisfaction among volunteers, who play a key role in the applicant’s service delivery process, has risen from 75% to 82% (scores of “5”) in that same time frame. (Strength)

Sample Comments - Results

• Item 2.6 – The applicant has reduced dependency on government funding through increased donations and other forms of private support as a source of revenue, addressing a key strategic challenge for the applicant. Private support has risen from 15% of its annual revenue in 2005 to almost 40% in 2008, while growing overall revenue by 30% in that same time period. (Strength)

Sample Comments - Results

• Item 3.4 – The applicant improved its process to report monthly performance to its funding partner, shortening delivery time from 30 days in 2006 to 15 days in 2008, six days less than the funder’s requirement of 21 days. The time made available by this improved process performance has led to the creation of additional capacity for providing service to customers. (Strength)

Sample Comments

• Item 1.3 – While the applicant expresses that its customers are satisfied, and indicates that it has received numerous letters from satisfied customers, it is not clear what systematic approach the applicant uses to determine the actual level of satisfaction of its customers with its services or how the applicant gathers data on customer experiences to develop actionable data on which to base improvement efforts. (OFI)

Sample Comments

• Item 2.4 – While the applicant indicates that it conducts monthly performance reviews, both by senior leaders and with its Board, it is not clear how or if the results of those reviews are shared with those individuals responsible for making decisions in the affected performance areas. (OFI)

Sample Comments

• Item 3.1 – While the applicant identifies its two key value creation processes, Needs Assessment and Case Management, it is not clear that the applicant has a process in place to assess the effectiveness of these processes in order to make improvements and incorporate emerging best practices. Lacking such a review and improvement process, it is not clear how these processes are kept current with evolving customer requirements, a key strategic challenge for the applicant. (OFI)

Sample Comments - Results

• Item 1.5 – While the applicant indicates that its customer satisfaction results have risen by 300% over the past three years (2006 through 2008), lacking actual data on the specific level of performance achieved, it is difficult to determine whether the applicant is achieving its goal of 95% customer satisfaction. (OFI)

Sample Comments - Results

• Item 2.6 – The applicant indicates that a key factor in its ability to compete effectively in its service delivery is minimization of delivery errors, yet its service delivery results actually reflect an increase in percentage of errors each of the past three years (2006 through 2008) from 6% to 8%. Given such a trend, it is not clear how the applicant will be able to ensure its sustainability in a highly competitive market. (OFI)

Sample Comments - Results

• Item 3.4 – The applicant indicates that it is critical for real-time data to be available at all times for effective service delivery in meeting customer needs, yet its system uptime remains at a level of only 85% in 2010. While this performance level has increased from 75% in 2009, it is not clear that the applicant can meet its customer requirements if data access is not available 15% of the time. (OFI)

Comment Improvement

• The applicant’s strategic planning process is highly effective due to the use of an outside consultant with much expertise.

Comment Improvement

• Item 1.2 – The applicant has excellent two-way communication due to its open-door policy and frequent e-mails from the leaders to the staff.

Comment Improvement

• Item 2.2 - It would be easier to assess the applicant’s approach to building relationships with customers if they would include more information about what they actually do in this area.

Comment Improvement

• Item 2.3 – The applicant does not include the results of its SWOT analysis making it difficult to assess the relevance and accuracy of its strategic objectives and goals. Without this information I cannot tell how the applicant will be able to improve its sustainability.

Comment Improvement

• Item 3.1 – The applicant could do a better job of improving its value creation processes if it would just take the time to map the processes and measure the performance of the process at each key step along the way.

Comment Improvement

• Item 3.1 – The applicant’s approach to managing its support processes is clearly a best practice that others should copy.

Comment Improvement

• Item 2.4 – The applicant would benefit from using a balanced scorecard approach as the over-arching framework for its performance management system. In doing so, the applicant would create better alignment of effort and produce greater impact. This would lead to greater likelihood of funding support increases, positioning the applicant for enhanced levels of sustainability.

Consensus Process - Review

• One set of comments for the team – seek to synergize comments, not just compile/combine

• Strengths and opportunities for improvement (OFI)

• Single “score” for the team on each item• Identify site visit issues• Submit single consensus feedback report with

draft of strengths and OFIs by August 25

Site Visit Process - Overview

• Review comments and identify site visit issues• Key site visit issues only• Verify and clarify on site• Review approach, deployment and results• Revise feedback report based on site visit

findings – new strengths and OFIs will emerge• Training for examiners and applicants on August

31

Examiner Training July 2011