Company Confidential Registration Management Committee What Are KPI’s Not Telling Us? July 22-23,...

Company Confidential

Registration Management Committee

What Are KPI’s Not Telling Us?

July 22-23, 2015

A. Michael McRandallNSF-ISR

RMC Workshop Charleston, SC July 22-23, 2015


Introduction

Mike McRandallBusiness Unit Manager, AerospaceNSF-ISR

Over 30 years experience in the aerospace industry in both quality and operations

2



Workshop Agenda• Process Identification (can’t measure ‘em if you don’t know ‘em)

• Process Effectiveness– What are we measuring?– Does it get us where we need to be?– Why aren’t OTD & Quality adequate measures for all?

• Options When Organization Is Process Challenged

• Case Studies/Examples

• Open Discussion

3



How Effective Are We In Applying The “Process Approach” To Auditing?

Not Very…

• We have been touting a “process-based approach” to auditing for over 15 years and we still have:− Auditees with no idea of what a process actually is

− Auditors in the same boat

− Industry “experts” just as confused and exacerbating the problem further

Our Methods are Not Overly Effective…WHY?

4



Process Identification

– Are processes properly identified?

– Is there a difference between an activity and a process?

– Are processes and procedures the same thing?

– Is every standard clause/element a process?

– Who identifies these processes?

5



Process Definition• Defined as*: “a set of interrelated or interacting

activities which transforms inputs into outputs.” These activities require allocation of resources (people, materials, equipment, etc.)

• A process must have defined:– Objectives– Inputs– Outputs– Activities– Resources

If you can’t describe what you’re doing as a process, you don’t know what you’re doing…

6



Activity or Process???

•An Activity is a unit of work to be completed within the execution of a process

•Activities can be performed in isolation - but a process requires a group of activities in conjunction with resources to actually be, a process

–9001:2008 Section 0.2 confirms: “…An activity or set of activities using resources, and managed in order to enable the transformation of inputs into outputs, can be considered a process.”

7



Procedure or Process…Really??

These are not interchangeable terms and do not mean the same thing

Procedure*: A specified way to carry out an activity or a process

Though not as common place as in the recent past this is still seen too often within an organization’s documentation.

8



Who Identifies These Processes?

● The organization, and only the organization,identifies (determines/defines) their processes (see 4.1a of the standard)

− It is not the place of the CB− It is not the place of the auditor

● The organization must also describe theinteraction between their processes - commonlyreferred to as the IOP (4.2.2c of the standard)

9



Are All Clauses/Elements of The Standard, Processes?Simple Answer: No

Remember what a process is and what it takes to fit the definition.

Also remember that an organization is required to monitor, measure and analyze their processes. So if they choose to call each element of the standard a process, then they need to be able to demonstrate how each element qualifies as a process and how those processes are monitored, measured and analyzed.

10



Assessing Process Effectiveness

● What is a KPI (Key Performance Indicator)?– Well, it is a metric, not a target

The target would be the objective. The KPIs show us how we are performing toward our objective(s) or targets.

● The objective should be directly related to the process being assessed

– It can, and should, tie into a more global business objective (OTD, quality, customer satisfaction, etc.)

– The KPIs should allow stakeholders (management, owners, auditors, etc) to assess the effectiveness of the process.

11



Will These Measures Get Us There?

● If the targets are either misplaced or disjointed from the overall company goals and objectives, the results might be, less than satisfactory.

● It is important to be able to show the tie-in between the process objectives (along with the KPIs used to track them) and the overall company objectives because…things may not always be as they seem.

Or as a wise man once queried: Do outputs really matter?

12



The Proper KPIs?

Let’s try a sports analogy. We’ll use the 2014 Colorado Rockies of MLB) as a shining example

13



2014 MLB Final - Team On-base Percentage (OBP)

14



2014 MLB Final - Team Total Runs Scored

15



2014 MLB Final - Team Batting Average

16



2014 MLB Final - Team Slugging Pct

17



As Individuals…

18



Some of Them Could Pick Too

19



So They Tore It Up…Right?

– Fourth overall in MLB, on base percentage

– Third overall in MLB, runs scored

– Second overall in MLB, batting average

– First overall in MLB, slugging percentage

20



Maybe Not So Much…

21



What Happened??

22



Best Starting Pitcher…72nd in MLB/ERA

23



Are We Looking At The Right Things?

24

What’s In It For Me??



Are We Looking At The Right Things?

What’s In It For Us??

25



How About OTD and Quality?

●They are required to be measured and addressed (5.2) – they must be good KPIs for every process, right?

– They are a means of having the organization maintain focus on the customer

– In most cases, they do not represent impactful data with regard to individual process performance

– KPIs point to symptoms, whether good or bad

– OTD and Quality are not symptoms – they are results

We’ll touch more on this during our open forum/case studies session

26



OTD or W/L

The Rockies supply us, again, with ammunition for this battle

● Company Goal – Make Playoffs (10 teams total)

– Objective – 88 Wins (think OTD/95%)

– Actual – 66 Wins (think OTD/70%)

Now making “Wins” your OTD equivalent and using Offense as one of your product realization processes, how accurate and adequate would it be to use Wins as your KPI for that process (Offense)?

27



Offense As a Process

Remember, The Rockies were:

● 4th in MLB for OBP

● 3rd in MLB for runs scored

● 2nd in MLB for batting average

● 1st in MLB for slugging percentage

But their KPI for this process (Wins), says that they failed, big time. So…what is the CA for this process?

28



What if The Organization is Process- Challenged?● They showed me their IOP and it doesn’t correspond with what they do…

– Start at 4.1a and work out from there√ Have they determined the processes? (Y/N)√ Have they determined the sequence and interaction? (Y/N)√ If yes to both, but still no match – 4.2.2c*

● They don’t have a process map or chart…– First of all, is this a problem?˚

√ If a narrative exists but does not meet the intent of the requirements then a discussion needs to take place to determine what element of the standard the finding will be written against.

29



Case Study / Open Discussion

You are auditing the purchasing process of Ellenem Industries and they have identified Customer OTD as their method for determining process effectiveness. Their target is 97% and their actual OTD was 75%. They also use OTD as the only KPI for this process. During the course of your process audit you find that they monitor and measure many things relative to purchasing such as: supplier quality, ppv, supplier delivery to contract requirements, time to receipt of quote, inventory turns and flow down escapes – all of which are performing to internal expectations. You found no nonconformances to the standard during your audit of this process.

•Seeing as how they missed their target of 97% (by a considerable margin), what type of actions would you expect to see they have taken to address this ineffective process?

•What actions, if any, would you as the auditor take?

30



Case Study / Open DiscussionYou are auditing the manufacturing process of Alacrity Amalgamated, a mid-size company that does short run jig grinding to customer specifications. They have identified product scrap as their effectiveness measure, with a target of < 2% of total volume. Their actual scrap rate is 4.2%. They have identified 2 KPIs for this process and they are: 1) the number of findings generated during internal audits (less than 3 / meeting target) and 2) the capacity scheduling of each work center (to be at 80% / meeting target). There were three findings against conformity to the standard identified during this audit:1) Failed to validate tools prior to release to production (7.5.1.3), 2) Undocumented changes to production processes (7.5.1.2), 3) No evidence that operations completed as planned (7.5.1h). Their response to missing their scrap target was to more closely analyze MRB minutes.

As an auditor, what, if any, actions would you take?

31



In Summation● Process Identification is Key

● KPIs Should Point to Symptoms

● OTD and Quality Are Not Great Individual Process KPIs

● Challenge Organizations to Understand Their Processes And to Identify KPIs to Monitor The Process

Thank you all for your time and patience

32

Date post:	13-Dec-2015
Category:	Documents
Upload:	melvin-stokes
View:	221 times
Download:	2 times

Company Confidential Registration Management Committee What Are KPI’s Not Telling Us? July 22-23,...

Documents