17
Evaluating New Hire Pilot Attitude and Behavior with Probationary Assessments Human Capital Management & Performance [email protected] [email protected] 1 November, 2017
Evaluating New Hire Pilot
Attitude and Behavior with
Probationary Assessments
Human Capital Management & Performance
1 November, 2017
Why is the probationary period becoming
more critical?
Pilot Supply < Pilot Demand
Regulatory Changes
US Pressures: Age 65
Retirements at Majors
As it becomes more challenging to bring pilots
through the front end of the selection process,
evaluating new hire pilots for safety and success
becomes increasingly significant
Pilots will be increasingly recruited from non-
traditional sources with varied backgrounds and
experience
Selection Process Philosophy
KSAPs Capabilitie
s: Pre-hire
Performanc
e: Post-hire
Applied
Knowledge
Personality
Characteristic
s
Cognitive
Skills
Training and
Job
Performanc
e
Attendance
Disciplinar
y Actions
Cockpit Fit
Probation is the most extended element of the selection process
4
Recruitment/Application
Screening
In person
screening
(Testing,
Interview, Applied
Exercises)
Job
Offer
Probation Training-IOE-Performance on the
line
Drug Test
Backgroun
d Check
Pilot Selection
Predictive Validity for Pilot Selection
Actual
problems * * * * * * * * * * * * * *
Predicted
problems
.
0
1
.
0
2
.
0
5
.
1
0
.
1
7
.
2
2
.
2
4
.
2
8
.
3
0
.
3
1
.
3
3
.
3
6
.
4
1
.
4
4
.
4
7
.
4
8
.
5
0
.
5
1
.
5
4
.
5
5
.
5
7
.
5
8
.
6
0
.
6
2
.
6
5
.
6
7
.
6
8
.
7
0
.
7
5
.
7
6
.
7
9
.
8
0
.
8
6
.
8
7
.
8
8
.
9
6
.
9
7
5
1. The Predictive Statistical Model Predicted Problems = A(“X” Cognitive ability measures) + B(“Y” Aviation
knowledge/problem solving measures) + C(“Z” personality measures)
2. Comparison of of the predicted performance versus actual
Red Predicted
Problems No problems
Real Problems 11 3
No Real
Problems 6 17
Black Predicted
Problems No problems
Real Problems 8 6
No Real
Problems 1 22
3. Optimizing the cost benefit
6
Crunching the Numbers: Rates of problems in training can be
predicted by performance on selection tests and strengthened by
the addition of background information
Category
Lowest to highest
(n=65 per group)
Ave. Training problems per pilot
1996-2007
Problem rate per 1000 training episodes
1996-2007
Total lost flying hours 2004-2007
1 (best predicted group)
.12 2 173
2 .11 2 187
3 .25 4 221
4 .48 7 232
5 .70 14 297
But are we hiring safe pilots? • Engagement
• Understanding
of principles
• Ability to multi-
task
• Teamwork
• Leadership
• Accountability
• Dedication to
continuous
learning
All of these behaviors and capabilities should be demonstrated during
probation and can be measured, but many are not
Traditional Monitoring During Probation Data available during this time period typically falls
into two categories: ◦ Approved and monitored with specific regulatory
reporting criteria
◦ Reports from Chief Pilots, Captain’s evaluations or other disciplinary letters
Although this information serves an important role for specific stakeholders, it is often treated as the basis for evaluation of probationary pilots, but this misses an important part of the story
By necessity, training is focused on proficiency
In the US, programs developed to meet FAA standards have some reference
to CRM and HF, but evaluate these things strictly from a procedural basis,
i.e. properly following checklists and leading debriefs
Performance
Dimensions
Reason Codes
Human Factors
Airmanship
Automation
Procedures
Knowledge
Systems
Knowledge Stick & Rudder
Procedures
Automation
Proficiency
CRM/TEM
The role of attitude in aviation training
A typical refrain from aviation instructors;
“ I would rather have a student with a good attitude who needed skills improvement than a student with good skills who needed attitude improvement”
This is true for a company making the investment in an employee, too
But conventional methods for evaluating newly hired
pilots focus primarily on procedural execution and technical proficiency
Probation assessments broaden the
scope of new hire evaluation during
training and IOE Prepared Undistracted Shows
Dedication Participates Exhibits
Confidence
Exhibits Passion Low Entitlement Takes Critique
Well Collaborates Communicates
These are the desired characteristics a new hire will bring to
training and the cockpit throughout their career
Design recommendations Link to proficiency scale
◦ Proficiency problems won’t cloud behavior evaluation
Use a scale that diverges from the typical training scale
◦ Greater range than error free, error captured
Provide behavioral anchors for scale
◦ Standardization
Focus on type of student, not just type of practitioner
◦ Will they be a lifelong learner?
Evaluate non-validation events
◦ Allows for more interaction, teaching environment
Get repeated measures
◦ Mitigate any personality clashes
Base evaluations on the job description
◦ Hold the pilot accountable for the displaying the capabilities required of the job
Sample Assessment: Student Focus
Does not allow outside distractions to disrupt the training environment (e.g.,
business, family)
1 Far short of Expectations: E.g. Preoccupied with trip home before training even starts. Clearly distracted during training without communicating reason, or withdrawing from session. Attitude that checking social media during training is acceptable. Lack of focus on training disruptive to other students.
2 Does not Meet Expectations: E.g. Personal problems or distractions not managed effectively.
3-Barely Meets Expectations
3 Reasonably Meets Expectations: E.g. Outside distractions not communicated, but have only minor impact on concentration. Engaged with material and not disruptive to other students.
3+ Completely Meets Expectations
4 Exceeds Expectations: E.g. No outside distractions, or aware of distractions, communicates them appropriately, conduct appropriate. Has made arrangements so that outside distractions will not interfere with training.
5 Far Exceeds Expectations: E.g. Intensely focused on training
Deployment
recommendations Training is key
◦ Inter-rater reliability and standardization
If the tool can be integrated into IT systems, automated alerts can be built in ◦ Early recognition of potential problems
Performance on this tool can be linked to selection performance ◦ Identify characteristics in candidates before hire, find
strengths and weaknesses of selection program
Can compare aircraft types as well as training and IOE ◦ Gives insight into differences across Training and
Standards
Linking Performance Back:
NEO PI-R Normal Personality
15
Neuroticism Extraversion Openness to
experience Agreeableness
Conscientious-
ness
Anxiety Warmth Fantasy Trust Competence
Hostility Gregariousness Aesthetics Straightforwardness Order
Depression Assertiveness Feelings Altruism Dutifulness
Self-consciousness Activity Actions Compliance Achievement Striving
Impulsiveness Excitement Seeking Ideas Modesty Self-Discipline
Vulnerability to
Stress Positive Emotion Values Tendermindedness Deliberation
Comparison of IOE outcomes to
Training
3.43.53.63.73.83.9
Training Mapped to IOE
3.43.53.63.73.83.9
Type of Skill: IOE Combined
Please feel free to contact us
with questions
01 919 602 2439
