Getting the most out of Non Technical Skills (NTS) AssessmentHumanFactors

Dr Matthew ThomasHuman Factors & Safety Management Group

University of South AustraliaLevel 7 Playford Building - City EastGPO Box 2471 Adelaide SA 5001AUSTRALIA

ph: 0438 808 808e-mail: matthew.thomas@unisa.edu.au

Kia Ora

Adelaide

Research Context

Culture, Error and NTS on the line...

TABLE TWELVE Logistic Regression Analysis - Contributions of contextual factors and non-technical skills to crew's

failure to respond to an error during the descent-approach-landing phase.

Predictor S.E. Wald's 2 d f p e Captain as Error Origin 1.555 .608 6.531 1 .011 4.735 Assertiveness - .849 .339 6.253 1 .012 .428 Constan t .856 .979 .764 1 .382 2.354 Model 2 = 21.994, df = 2, p < .001, Nagelkerke R2 = .293, 70.8% correct classification.

Culture, NTS, and Error Management

If captain made the error, crew 4.7 times less likely to manage error

If crew scored high on assertiveness, 2.3 times more likely manage error

Culture, NTS, and Error ManagementStrong correlations between NTS markers and Error ManagementTop five markers: Leadership / Followership

VigilanceAssertivenessInquiryCommunication Environment

Instructional Use of Error and NTS...

Use of Error in Line Training

Corrected by Instructor

Directive from Instructor

Feedback from Instructor

Corrected by Trainee

None

0% 10% 20% 30% 40% 50%

45

10

5

14

26

Instructor Acknowledged Error

Debreifed in Simulator

Debriefed Post-Session

Detection of Error Discussed

Error Prevention Discussed

Error Management Discussed

0% 25% 50% 75% 100%

9

40

6

59

48

64

6

34

3

40

56

77

TEM in the Simulator

NTS

Identify

Debrief

Day One (Training) Day Two (Check) N = 277

“Gotcha” - error due to situational demands

Early recognition and management strategies.

“Random” - slip or lapse due being

humanDetection and response strategies.

“Deficit” - lack of knowledge or skill More practice!

“Mistake” - decision-making or judgement

Dig deeper and understand why. Need to identify where decision

process went wrong.

Error Type Training Focus

Summary

Some good data on the role of NTS in crew performance during normal operations

NTS are still not a real focus in recurrent training / checking programs

CRM in the classroom and NTS in the simulator / on the line are not yet aligned

TEM can be used as a focus for NTS in training

Regulatory Context

13 Jan 2009NOTICE OF FINAL RULE MAKINGImplementation of Safety Management Systems (SMS)and introduction of Human Factors (HF) Training andNon-Technical Skills (NTS) Assessment

Amendments to Civil Aviation Orders (CAOs) 82.3 and 82.5 (Air Operators Certificates - RPT

CASA - Civil Aviation Safety Authority (AUSTRALIA)

RPT AOC holders must “have a program, approved by CASA, to train and assess personnel in human factors and non-technical skills with the aim of minimising human error.”

Human Factors (HF) means the minimisation of human error and itsconsequences by optimising the relationships within systems between people,activities and equipment.

Non-Technical Skills (NTS) means specific human competencies, including criticaldecision making, team communication, situational awareness and workloadmanagement, which may minimise human error in aviation.

CASA - Civil Aviation Safety Authority (AUSTRALIA)

17 Sep 2009Project OS 09/15 - Minor Amendment of CAO 82.3 & 82.5 to remove the legislative requirement for assessment and to provide more time for Industry to conduct Human Factors TrainingProject approved.

CASA - Civil Aviation Safety Authority (AUSTRALIA)

FAA - Federal Aviation Administration (USA)

Funded some great research.

Got burnt by AQP?

Has produced some good Advisory Circulars (AC120-51E CRM).

EU-OPS and EU-FCL

Requirement for assessment of CRM performance

EASA - European Aviation Safety Agency (EUROPE)

Line checks

(i) Line checks must establish the ability to perform satisfactorily a complete line operation including pre-flight and post-flight procedures and use of the equipment provided, as specified in the Operations Manual.(ii) The flight crew must be assessed on their Crew Resource Management CRM skills in accordance with a methodology acceptable to the Authority and published in the Operations Manual. The purpose of such assessment is to:(A) Provide feedback to the crew collectively and individually and serve to identify retraining; and(B) Be used to improve the CRM training system.(iii) CRM assessment alone shall not be used as a reason for a failure of the line check.(iv) When pilots are assigned duties as pilot flying and pilot non-flying they must be checked in both functions.

EU-OPS 1.965 Recurrent Training and Checking — Pilots

NOTECHS

Summary

NTS Assessment is coming... (albeit very very slowly).

Australia is going backwards as quickly as it is going forwards.

European regulations are the most advanced, but what about implementation?

We should be setting the standards by worlds best practice rather than awaiting regulatory requirements.

Getting the most out of NTS Assessment / Evaluation

The core requirements...

Supported by standardized and customized tools.

Supported by good resources for instructors and crew.

Starts with good syllabus & scenario design for the simulator.

Provides focus for brief and debrief during training.

BEHAVIOURAL MARKERS

TRAINING & INTER-RATER RELIABILITY

INTEGRATION OF TRAINING

FOCUSSING ON SKILL DEVELOPMENT

Behavioural Markers and NTS Assessment

During the late 1990s the “Holy Grail” was a universal standardised set of

Behavioural Markers for NTS.

NOTECHS

Behavioural Markers for NTS

NOTECHS

Rating Scale

Training and Inter-Rater Reliability

Training

rwg=1-( Sχ2/σE2)

Calibration Monitoring

2 day program training records

Integration

!"!#$%#&'()*+#,-./#0-1(2(2)##31241'#

!015.677#128#92(+(1'#,'(:;

"#$%&'()*!+ ,)-!.#-'/(!"#0$1/%2!3--!*'*4-!$1(-!5#&!6-*1'472

"<=> &,0#!"!#$%#?03@

,-6AA/(28#015.677

,)-!8&#779'/6!(:'6-4'/-7!7)#9/!;-4#9!9-&-!6-&'<-6!*)&#:()!54'()*!*-7*!61*1=!

-/('/--&'/(!1/14%7'7=!1/6!54'()*!7'0:41*#&!-<14:1*'#/72

$6+.B >/('/-!7:&(-!81/!#88:&!9'*)!1!7*&#/(!8&#779'/6!#&!*1'49'/6!8#0$#/-/*!'5!

*1?-#55!*)&:7*!'7!7-*!;-5#&-!;&1?-!&-4-17-2!,)-&-5#&-=!*)-!&#44'/(!*1?-#55!

$&#8-6:&-!'7!7*&#/(4%!16<'7-6!9)-/!8&#779'/67!-@8--6!AB!?/#*7!#&!

*1'49'/67!-@8--6!CB!?/#*72

015.677#,-6AA/(28#%4(8.'(2.A

"&#779'/6!(:'6-4'/-7!1&-!/#*!8#/7'6-&-6!4'0'*1*'#/72!"&#779'/6!(:'6-4'/-7!1&-!

$&#<'6-6!*#!177'7*!#$-&1*#&7!'/!-7*1;4'7)'/(!*)-'&!#9/!8&#779'/6!$#4'8'-72

,1?-#55!8&#779'/6!(:'6-4'/-7!1&-!;17-6!#/!*)-!0#7*!16<-&7-!1'&$41/-!4#16'/(!D4'()*!

9-'()*!1/6!15*!8-/*-&!#5!(&1<'*%E!1/6!177:0-!1/!-/('/-!#:*!F,G!1/6!$&#$-&!$'4#*!

*-8)/'H:-2!G/!74'$$-&%!&:/91%7=!8&#779'/6!(:'6-4'/-7!1&-!1!5:/8*'#/!#5!&:/91%!

7:&518-!8#/6'*'#/2

IJ'/67!0-17:&-6!1*!KK!5--*!DCB!0E!*#9-&!)-'()*!1/6!1$$4%!5#&!&:/91%7!CLM!5--*!

DLN0E!#&!(&-1*-&!'/!9'6*)2

II!,1?-#55!#/!:/*&-1*-6!'8-!#&!7/#9!7)#:46!#/4%!;-!1**-0$*-6!9)-/!/#!0-4*'/(!'7!

$&-7-/*2

C42/1D#,628(+(62,-6AA/(28#,6:E62.2+#

?526+A@#F

/(+*64+#G#/(+*#/(2)'.+A

O&% KP!Q!KL

J-* AN

3*1/6'/(!J1*-&Q34:7) CN

3/#9!R!S#!T-4*'/(!II AN

U8-!R!S#!T-4*'/(!II CN

HI+6;.-#"=J#>KK!

!"!#$%#&'()*+#,-./#0-1(2(2)##31241'#

!015.677#128#92(+(1'#,'(:;

"#$%&'()*!+ ,)-!.#-'/(!"#0$1/%2!3--!*'*4-!$1(-!5#&!6-*1'472

&,0#!"!#$%#?03@ "<="

L(-.I+(621'#,62+-6'

U/'*'14!&:/91%!14'(/0-/*!1/6!70##*)!7%00-*&'814!*)&:7*!1$$4'81*'#/!&-7:4*!'/!(##6!

8&#779'/6!8#/*&#4!81$1;'4'*%!6:&'/(!*1?-#552!V'()*!5#&91&6!$&-77:&-!#/!*)-!8#/*&#4!

8#4:0/!6:&'/(!*)-!'/'*'14!$)17-!#5!*1?-#55!&#44!D;-4#9!1$$&#@'01*-4%!MB!?/#*7E!

'/8&-17-7!/#7-!9)--4!7*--&'/(!-55-8*'<-/-772!W/%!6-<'1*'#/!5&#0!*)-!8-/*-&4'/-!

6:&'/(!*)&:7*!1$$4'81*'#/!7)#:46!;-!8#:/*-&-6!9'*)!'00-6'1*-!70##*)!1/6!$#7'*'<-!

8#/*&#4!'/$:*72!30##*)!&:66-&!8#/*&#4!'/$:*7!8#0;'/-6!9'*)!70144!8#/*&#4!9)--4!

'/$:*7!&-7:4*!'/!1!/#&014!*1?-#55!9'*)!/#!#<-&8#/*&#44'/(2!V1&(-!8#/*&#4!9)--4!

'/$:*7!81/!)1<-!1/!16<-&7-!-55-8*!#/!6'&-8*'#/14!8#/*&#4!/-1&!XCDT"YE!6:-!*#!*)-!

166'*'#/14!6&1(!#5!*)-!-@*-/6-6!7$#'4-&72

$6+.B J'*)!9-*!#&!74'$$-&%!&:/91%!8#/6'*'#/7=!*)-!ZT!7)#:46!('<-!7$-8'14!

1**-/*'#/!*#!-/7:&'/(!*)-!-/('/-7!)1<-!7%00-*&'8144%!;141/8-6!*)&:7*!

'/6'81*'#/72!

C6+1+(62#128#015.677

T1'/*1'/!9'/(7!4-<-4!6:&'/(!*)-!*1?-#55!&#44!;%!1$$4%'/(!8#/*&#4!9)--4!

6'7$418-0-/*!'/*#!*)-!9'/62!O:&'/(!&#*1*'#/!8#/*'/:-!*#!1$$4%!8#/*&#4!9)--4!'/!*)-!

6'7$418-6!$#7'*'#/!*#!?--$!*)-!9'/(7!4-<-4!6:&'/(!4'5*#552!,)-!1'&$41/-!'7!'/!1!

7'6-74'$!9'*)!8&#77-6!8#/*&#47!1*!*)'7!$#'/*2!W!74#9=!70##*)!&-8#<-&%!5&#0!*)'7!

7'6-74'$!'7!188#0$4'7)-6!15*-&!4'5*#55!;%!74#94%!/-:*&14'['/(!*)-!8#/*&#4!9)--4!1/6!

&:66-&!$-61472

%4A+D#M(28#128#N+-62)#,-6AA/(28#,628(+(62A

\#&!*1?-#55!'/!(:7*%!#&!7*&#/(!8&#779'/6!8#/6'*'#/7=!:7-!#5!1!)'()-&!*)&:7*!7-**'/(!

*)1/!*)-!0'/'0:0!&-H:'&-6!'7!&-8#00-/6-62!J)-/!*)-!$&-<1'4'/(!9'/6!'7!1*!#&!

/-1&!]B^!*#!*)-!&:/91%=!*)-!$#77';'4'*%!#5!9'/6!7)'5*7!&-7:4*'/(!'/!(:7*%!*1'49'/6!

8#0$#/-/*7!6:&'/(!&#*1*'#/!#&!4'5*#55!'/8&-17-72!O:&'/(!*)'7!8#/6'*'#/=!8#/7'6-&!*)-!

:7-!#5!*)&:7*!7-**'/(7!84#7-!*#!#&!1*!01@'0:0!*1?-#55!*)&:7*2!,)-!:7-!#5!1!)'()-&!

*1?-#55!*)&:7*!7-**'/(!&-6:8-7!*)-!&-H:'&-6!&:/91%!4-/(*)!1/6!0'/'0'[-7!*)-!

1'&$41/-!-@$#7:&-!*#!(:7*%!8#/6'*'#/7!6:&'/(!&#*1*'#/=!4'5*#55=!1/6!'/'*'14!84'0;2

HI+6;.-#"=J#>KK!

Predictions for the Future

Integrated

Beyond Classroom

More Simulator and Line

Changes in Structure

Aligned

Targeted CFIT / ALARP

Skills. Full stop.

CRM = NTS = TEM

Still important

Skill Development

Emotions “HEA”

Fitness for Duty

Performance Protection

Changes in Content

Systems Thinking

Fatigue / Health /...

Frustration / Stress Confusion

Adapting to Situations

Safety Emerges from System

Some Good Resources

http://www.aavpasymposium.org

Dr Matthew J W Thomas

E-mail: matthew.thomas@unisa.edu.auPhone: +61 438 808 808 Web: http://www.unisa.edu.au/safety

UniSA © 2009