7/28/2019 RAG - Measuring Employee Motivation
1/70
JUNE 2011 Draft v6 REVISED Report 1
Measuring Employee Motivation:
Design and Cognitive Test of Employee Motivation Questions
A report for:
Presented by:
Your contact person:
GfK Business
Colin Hand
Research Manager
Phone: +44 (0)20 7890 9654
Fax: +44 (0)20 7890 9222
E-mail: [email protected]
GfK. Growth from Knowledge
7/28/2019 RAG - Measuring Employee Motivation
2/70
JUNE 2011 Draft v6 REVISED Report 2
CONFIDENTIALITY
Please note that the copyright in the attached report is owned by OME and the
provision of information under Freedom of Information Act does not give the
recipient a right to re-use the information in a way that would infringe
copyright (for example, by publishing and issuing copies to the public).
Brief extracts of the material may be reproduced under the fair dealing
provisions of the Copyright, Designs and Patents Act 1988 for the purposes of
research for non-commercial purposes, private study, criticism, review and
news reporting.
NOTE ON QUALITATIVE FINDINGS
The findings in this document are qualitative in nature. They are derived from
a small sample and are not projectable in numeric terms to a larger
population. As such, these findings are directional and should be considered
indicative rather than conclusive.
7/28/2019 RAG - Measuring Employee Motivation
3/70
JUNE 2011 Draft v6 REVISED Report 3
CONTENTS
Page
1. BACKGROUND TO THE RESEARCH ........................................................... 4
2. RESEARCH APPROACH ............................................................................... 6
3. THE PROPOSED QUESTION SET FOR TESTING ........................................ 8
4. OVERVIEW ................................................................................................... 20
5. QUESTION 1: MEASURING MOTIVATION ................................................. 23
6. QUESTION 2: SELECTING FACTORS ........................................................ 26
7. THE LIST OF MOTIVATIONAL FACTORS .................................................. 32
8. QUESTION 3: RANKING MOTIVATIONAL FACTORS ................................ 40
9. QUESTION 4: ASPECTS OF PAY AND BENEFITS .................................... 45
10. CONCLUSION .............................................................................................. 51
11. APPENDIX I: RECOMMENDED QUESTION SET ........................................ 52
12. APPENDIX II: SIGNIFICANCE TESTS ......................................................... 59
13. APPENDIX III: COGNITIVE TESTING TOPIC GUIDE .................................. 62
7/28/2019 RAG - Measuring Employee Motivation
4/70
JUNE 2011 Draft v6 REVISED Report 4
1. BACKGROUND TO THE RESEARCH
The Office of Manpower Economics (OME) provides support for the six independent
public sector Pay Review Bodies, and the Police Boards. OME was set up to provide an
independent Secretariat to the Review Bodies, free from Government direction as to the
advice and support offered. It aims to provide high quality analytical, administrative andtechnical support.
The Review Body process is evidence-driven: stakeholders are invited to submit
evidence each year which the Pay Review Bodies consider when deciding upon their
recommendations, often alongside other, independently-provided information. Included
in most Pay Review Bodies terms of reference among several other important
considerations is to consider, when forming their pay recommendations, the need to
recruit, retain and motivate suitably able and qualified people.
In 2009 the Office of Manpower Economics (OME) commissioned GfK NOP to conduct a
research study aimed at providing authoritative guidance on, and a suite of standard
questions to measure, employee motivation. These questions and the attending
guidance were intended to be passed on to Review Bodies as a set of standard,
harmonised questions that could be used to provide in evidence to Review Body
Stakeholders comparable data across Review Body remit groups.
Three standard questions were presented to Review Bodies, however, they wereconcerned that the questions would not provide enough usable detail, and did not
differentiate between factors that motivate and demotivate. Also, the relative effect of
aspects of remuneration on motivation could not be drawn out, and placed in context of
motivation overall.
Since the first submission of questions, revisions have been made to the standard
questions, and a revised set was developed by OME. These questions underwent further
review by GfK before a new set of questions was proposed.
These new questions however were untested and required piloting with potential
respondents to identify unforeseen problems, and provide evidence for further
refinement.
1.1 Objectives
A cognitive testing phase was carried out to evaluate a draft set of employee motivation
questions, by observing potential respondents complete the questions, identifying
problems and gathering evidence for further revisions. The objectives of the testing
phase were:
7/28/2019 RAG - Measuring Employee Motivation
5/70
JUNE 2011 Draft v6 REVISED Report 5
To conduct a cognitive test of a revised set of employee motivation questions
amongst public sector workers
To test recommended approaches alongside alternative approaches
To produce an agreed, concise set of questions with the potential to improve the
quality of evidence on employee motivation available to Review Bodies
To document the conduct of the testing in order to provide justification for the final
proposed question set.
7/28/2019 RAG - Measuring Employee Motivation
6/70
JUNE 2011 Draft v6 REVISED Report 6
2. RESEARCH APPROACH
2.1 Cognitive depth interviews
Following the review process, 13 cognitive-depth interviews were conducted with
respondents representing the range of pay review body remit groups. Cognitive testing
explores the mental process by which respondents formulate their response to a
question. By investigating how respondents interpret questions, checking their
understanding of key terms, and exploring how judgments about answers are made, it is
possible to test whether the questions are working as intended.
The questions were administered as a pen and paper self completion survey, whilst an
interviewer observed how respondents answered questions. Respondents were asked to
think out loud, by vocalising their considerations as they completed the questions. This
enabled the interviewer to probe concurrently on problems as they arose, and explore
how respondents overcame them, as well as check that key terms were being registered
and understood as intended.
2.2 Sample
The majority of participants were recruited from GfK NOPs ePanel, with 3 respondents
being contacted through free found sources. Since respondents were not contacted via
their workplace or employer, the interviews were conducted in respondents homes or
other convenient place, outside of work hours.
The interviews sought to gain a representation of the Pay Review Bodies remit groups,
at junior and senior levels. The final achieved sample is summarised below:
Junior Senior
Armed Forces 1 1
NHS 1 1
Doctors and Dentists 1 1
Prison Services 1 1
School Teachers 1 1
Police Force 1 1
Senior Salaries 1
Each respondent was offered an honorarium of 50 as a token of thanks for their time. It
is not common practice to offer incentives to employees for employee research.
However, for this study we were not interviewing respondents as employees, but in their
7/28/2019 RAG - Measuring Employee Motivation
7/70
JUNE 2011 Draft v6 REVISED Report 7
capacity as people working in the public sector. The key differences to employee
research that give good reason for an honorarium were:
Interviews took place during their free time and likely in their homes
We were not introduced to respondents by an employer, and their employer was
not informed of their participation
Respondents were primarily recruited from GfKs panel of respondents, and
therefore it is common practice to offer an honorarium as a thank you for the
additional time they have given to any research study.
We also expected more success in recruiting willing participants if an incentive payment
were to be offered. Also, by using a cash incentive, this allows participants to donate this
to a charity of their choice or use it to replace their lost hour.
7/28/2019 RAG - Measuring Employee Motivation
8/70
JUNE 2011 Draft v6 REVISED Report 8
3. THE PROPOSED QUESTION SET FOR TESTING
The questions taken into testing followed the general structure proposed by OME.
Question 1. Identifying level of motivation on a scale of 1 to 10:
This question intended to identify where respondents currently are in terms of
motivation and provide a way of tracking motivation levels over time.
Question 2. Identify factors that contribute to motivation:
The question retained the original approach of identifying what contributes to
levels of motivation, but from a neutrally worded list of factors. Respondents
were asked to identify whether each factor, raises, has no influence, or lowers
their motivation.
Question 3. Ranking the most important factors:
The question then asked respondents to rank up to three of the most important
factors that currently raise motivation, and then up to three of the most important
factors that currently lower motivation.
Question 4. Aspects of pay and benefits:
Where respondents identified pay and benefits as a factor that either raises or
lowers motivation, they were routed to a question asking them to rank up to
three of the most important aspects of pay and benefits that either raised or
lowered motivation.
Alternative answer frameworks and question wordings were also tested during the
interviews. The full set of questions as they were tested is set out below. The final
revised set of questions is set out in Appendix I.
7/28/2019 RAG - Measuring Employee Motivation
9/70
JUNE 2011 Draft v6 REVISED Report 9
TEST 1) In relation to your current working life, on a scale of 1 to 10 where 1 is extremely
low and 10 is extremely high, how motivated are you to do a good job?
1 Extremely low
2
3
4
5
6
7
8
9
10 Extremely high
7/28/2019 RAG - Measuring Employee Motivation
10/70
JUNE 2011 Draft v6 REVISED Report 10
Below is a list of factors that may affect how motivated you are.
TEST 2) Again, in relation to your current working life, overall, how do the following
factors affect how motivated you are to do a good job?
Please provide an answer for each factor
Raises my
motivation
Neither
raises nor
lowers my
motivation
Lowers my
motivation
Dont
know
a. My Work Life Balance
b. The extent to which I am treated with respect in my workplace
c. My colleagues/team
d. The level of pride I have in working for this organisation
e. The level of control I have over my own work
f. The extent to which I am valued in my workplace
g. Material reward (salaries, pension etc.)
h. Access to flexible working policies
i. My line manager
j. The extent to which my work makes a difference to people
k. My prospects for promotion/progression
l. Status
m. Being challenged by my work
n. Relationship between my work and my personal values
o. Level of variety in my work
p. Extent to which I enjoy my work
q. The extent to which I am treated fairly and equally in my
workplace
r. Exercising authority
s. Job security
t. Extent to which I am able to develop new skills on the job
u. Opportunities for training and development
v. Communication at work
w. Other (please specify) _____________________________
7/28/2019 RAG - Measuring Employee Motivation
11/70
JUNE 2011 Draft v6 REVISED Report 11
TEST 3 a) Of the factors that you have identified that currently raise your motivation to
do a good job; please rank up to three in order of importance
Please rank only those factors that you indicated raise your motivation
Insert 1 for most important factor, 2 for the second, 3 for the third.
Raises my
motivation
a. My Work Life Balance
b. The extent to which I am treated with respect in myworkplace
c. My colleagues/team
d. The level of pride I have in working for this organisation
e. The level of control I have over my own work
f. The extent to which I am valued in my workplace
g. Material reward (salaries, pension etc.)
h. Access to flexible working policies
i. My line manager
j. The extent to which my work makes a difference topeople
k. My prospects for promotion/progression
l. Status
m. Being challenged by my work
n. Relationship between my work and my personal values
o. Level of variety in my work
p. Extent to which I enjoy my work
q. The extent to which I am treated fairly and equally in my
workplacer. Exercising authority
s. Job security
t. Extent to which I am able to develop new skills on thejob
u. Opportunities for training and development
v. Communication at work
w. Other (please specify) _____________________________________
7/28/2019 RAG - Measuring Employee Motivation
12/70
JUNE 2011 Draft v6 REVISED Report 12
TEST 3 b) Of the factors that you have identified that currently lower your
motivation to do a good job; please rank up to three in order of importance
Please rank only those factors that you indicated lower your motivation
Please insert 1 for most important factor, 2 for the second, 3 for the third.
Lowers my
motivation
a. My Work Life Balance
b. The extent to which I am treated with respect in myworkplace
c. My colleagues/team
d. The level of pride I have in working for this organisation
e. The level of control I have over my own work
f. The extent to which I am valued in my workplace
g. Material reward (salaries, pension etc.)
h. Access to flexible working policies
i. My line manager
j. The extent to which my work makes a difference topeople
k. My prospects for promotion/progression
l. Status
m. Being challenged by my work
n. Relationship between my work and my personal values
o. Level of variety in my work
p. Extent to which I enjoy my work
q. The extent to which I am treated fairly and equally in myworkplace
r. Exercising authority
s. Job security
t. Extent to which I am able to develop new skills on thejob
u. Opportunities for training and development
v. Communication at work
w. Other (please specify) ______________________________________
7/28/2019 RAG - Measuring Employee Motivation
13/70
JUNE 2011 Draft v6 REVISED Report 13
ANSWER QUESTION 4 only if material reward is selected in response to
Question 2
You identified "material reward" as a factor that contributes to your current
level of motivation
Below is a list of aspects that make up material reward
TEST 4a) Please rank up to threeaspects of material reward that currently
raiseyour motivation to do a good job, in order of importance
Please insert 1 for most important factor, 2 for the second, 3 for the third.
Raises my motivation
1. Basic salary level
2. Latest pay settlement / review
3. Performance-related pay elements, e.g. bonuses, profit Sharing
4. Access to pay increments/ structured pay progression
5. Other pay elements, e.g. overtime pay, shift pay premiums,
unsocial hours payments
6. Pension benefits being built up
7. other non-pay benefits, e.g. health insurance, car/car allowance
8. other (please specify) e.g. season ticket loan, childcare vouchers,
subsidised canteen___________________
7/28/2019 RAG - Measuring Employee Motivation
14/70
JUNE 2011 Draft v6 REVISED Report 14
TEST 4b) Please rank up to threeaspects of material reward that currently
loweryour motivation to do a good job, in order of importance
Please insert 1 for most important factor, 2 for the second, 3 for the third.
Lowers my motivation1. Basic salary level
2. Latest pay settlement / review
3. Performance-related pay elements, e.g. bonuses, profit Sharing
4. Access to pay increments/ structured pay progression
5. Other pay elements, e.g. overtime pay, shift pay premiums,
unsocial hours payments
6. Pension benefits being built up
7. other non-pay benefits, e.g. health insurance, car/car allowance
8. other (please specify) e.g. season ticket loan, childcare vouchers,
subsidised canteen
7/28/2019 RAG - Measuring Employee Motivation
15/70
JUNE 2011 Draft v6 REVISED Report 15
Alternative question 3
TEST 3a) Of the factors that you have identified that currently raise your motivation to do
a good job; please rank up to three in order of importance
Please rank only those factors that you indicated raises your motivation
Select one factor in each column
Most
Important
1
Second most
important
2
Third most
important
3
a. My Work Life Balance
b. The extent to which I am treated with respect in my workplace
c. My colleagues/team
d. The level of pride I have in working for this organisation
e. The level of control I have over my own work
f. The extent to which I am valued in my workplace
g. Material reward (salaries, pension etc.)
h. Access to flexible working policies
i. My line manager
j. The extent to which my work makes a difference to people
k. My prospects for promotion/progression
l. Status
m. Being challenged by my work
n. Relationship between my work and my personal values
o. Level of variety in my work
p. Extent to which I enjoy my work
q. The extent to which I am treated fairly and equally in my workplace
r. Exercising authority
s. Job security
t. Extent to which I am able to develop new skills on the job
u. Training and development
v. Communication at work
w. Other Specify _______________________
7/28/2019 RAG - Measuring Employee Motivation
16/70
JUNE 2011 Draft v6 REVISED Report 16
TEST 3b) Of the factors that you have identified that currently lower your
motivation to do a good job; please rank up to three in order of importance
Please rank only those factors that you indicated lower your motivation
Select one factor in each column
Most
Important
1
Second most
important
2
Third most
important
3
a. My Work Life Balance
b. The extent to which I am treated with respect in my workplace
c. My colleagues/team
d. The level of pride I have in working for this organisation
e. The level of control I have over my own work
f. The extent to which I am valued in my workplace
g. Material reward (salaries, pension etc.)
h. Access to flexible working policies
i. My line manager
j. The extent to which my work makes a difference to people
k. My prospects for promotion/progression
l. Status
m. Being challenged by my work
n. Relationship between my work and my personal values
o. Level of variety in my work
p. Extent to which I enjoy my work
q. The extent to which I am treated fairly and equally in my workplace
r. Exercising authority
s. Job security
t. Extent to which I am able to develop new skills on the job
u. Training and development
v. Communication at work
w. Other Specify _______________________
7/28/2019 RAG - Measuring Employee Motivation
17/70
JUNE 2011 Draft v6 REVISED Report 17
Alternative Question 4
You identified "material reward" as a factor that contributes to your current level of
motivation
Below is a list of aspects that make up material reward
TEST 4a) Please rank up to threeaspects of material reward that currently raise
your motivation to do a good job, in order of importance
Select one factor in each column
Most
Important
1
Second most
important
2
Third most
important
3
1. Basic salary level
2. Latest pay settlement / review
3. Performance-related pay elements, e.g. bonuses, profit Sharing
4. Access to pay increments/ structured pay progression
5. Other pay elements, e.g. overtime pay, shift pay premiums, unsocial
hours payments
6. Pension benefits being built up
7. other non-pay benefits, e.g. health insurance, car/car allowance
8. other (please specify) e.g. season ticket loan, childcare vouchers,
subsidised canteen______________________________________
7/28/2019 RAG - Measuring Employee Motivation
18/70
JUNE 2011 Draft v6 REVISED Report 18
You identified "material reward" as a factor that contributes to your current level of
motivation
Below is a list of aspects that make up material reward
TEST 4a) Please rank up to three aspects of material reward that currently lower
your motivation to do a good job, in order of importance
Select one factor in each column
Most
Important
1
Second most
important
2
Third most
important
3
1. Basic salary level
2. Latest pay settlement / review
3. Performance-related pay elements, e.g. bonuses, profit Sharing
4. Access to pay increments/ structured pay progression
5. Other pay elements, e.g. overtime pay, shift pay premiums, unsocial
hours payments
6. Pension benefits being built up
7. other non-pay benefits, e.g. health insurance, car/car allowance
8. other (please specify) e.g. season ticket loan, childcare vouchers,
subsidised canteen____________________________________
7/28/2019 RAG - Measuring Employee Motivation
19/70
JUNE 2011 Draft v6 REVISED Report 19
Alternative Question 4
4 Please rank up to three aspects of material reward that currently raiseyour
motivation to do a good job, and up to threeaspects of motivation that
currently loweryour motivation to do a good job in order of importance.
Please insert 1 for most important factor, 2 for the second, 3 for the third.
Raises my
motivation
Lowers my
motivation
1. Basic salary level
2. Latest pay settlement / review
3. Performance-related pay elements, e.g. bonuses, profit
Sharing
4. Access to pay increments/ structured pay progression
5. Other pay elements, e.g. overtime pay, shift pay
premiums, unsocial hours payments
6. Pension benefits being built up
7. other non-pay benefits, e.g. health insurance, car/car
allowance
8. other (please specify) e.g. season ticket loan, childcare
vouchers, subsidised
canteen____________________________________
7/28/2019 RAG - Measuring Employee Motivation
20/70
JUNE 2011 Draft v6 REVISED Report 20
4. OVERVIEW
Throughout testing, the questions performed well, with the majority of issues being
around understanding of key terms, signposting tasks clearly, and complex question
wording that led to confusion when ranking motivational factors. The questions provoked
thoughtful deliberation, and respondents commented that providing answers was not aneasy task. This was attributed to the challenge of considering how the range of factors
affected their working life rather than issues around understanding of the questions
themselves. However, testing revealed recurring issues that impacted on the immediacy
of understanding and ability to complete the questions in a way that felt satisfactory to
respondents:
The impact of mixed considerations: when deciding how one of the
factors affects motivation, the final answer may be subject to several
considerations or experiences, which can affect motivation both positively and
negatively.
Use of technical language: where terms such as factorand rankadded
an additional interpretative task on respondents
Grammatical complexity of questions: where questions were difficult to
follow, and had to be interpreted rather than immediately understood
Understanding the ranking up to three tasks: there were insufficient
instructions to make it clear to respondents how to progress where less than
three answers had been provided at previous questions.
Each of these issues affected all respondents to a greater or lesser degree, and had
significant impact on the ease with which respondents were able to navigate through the
questions. These recurring issues are explained and explored in further detail below.
Impact of mixed considerations
It was anticipated that the questions would present a challenge to respondents to
answer, particularly as the factors and aspects of pay identified could potentially both
raise and lower motivation over a period of time.
The purpose of currentand on balancein the questions was intended to help minimise
this difficulty and help respondents towards their answers. However, presented with
either a list of factors, or aspects of pay and benefits, respondents still found that they
had a mix of experiences and considerations that they could bring to bear on the
questions.
Further probing revealed that the terms current and on balance were not being
immediately registered within the question, diluting the impact each had on how
respondents formulated their answers. This was due to respondents interpretation of the
terms, the fit with the mental process of formulating an answer, and the grammatical
complexity of the questions themselves.
7/28/2019 RAG - Measuring Employee Motivation
21/70
JUNE 2011 Draft v6 REVISED Report 21
Respondents dealt with this dilemma in one of several ways:
Respondents reflected on their current situation in comparison with a
memorable period where the factor affected their motivation in a
particularly extreme way (either positively or negatively). For example a
police officer recently returned to work after maternity leave felt that her
Work/Life Balance raised her motivation since she was now able to benefit
from family friendly policies, whereas in the past, shift work has been
particularly disruptive to her work/life balance.
Respondents selected the neither raises nor lowers answer, as
considerations pulling each way neutralised the affect on motivation, although
this was considered to be an unsatisfactory answer.
Answered dont know. Again this was considered an unsatisfactory
answer, because You should know these things, [factor] either does affectmotivation or it doesnt. There was a mild preference for rewording dont
knowto cant say.
Use of technical language
Respondents were given pause by four terms used throughout the questionnaire;
factors, indicated, column and rank. However, these were not considered difficult
terms in themselves, but they did require interpretation within the context of the
questionnaire to be fully understood. For example, on seeing factorsfor the first time,
respondents immediately looked for any clue as to what the word factor was referring to.
Grammatical complexity
Respondents did not consider the questions to be difficult to understand and were able to
articulate what they were being asked to do in their own words. However, there was
evidence that questions were not immediately grasped, and often required re-reading
before respondents were comfortable they had understood the question correctly.
This was particularly the case with question 2a) asking respondents to identify how
factors affect their motivation and 3a) and 3b), asking respondents to rank factors in
order of importance:
2a) Again, in relation to your current working life, on balance, how do each of the
following factors affect how motivated you are to do a good job?
3a) Of the factors that you have identified that currently raise your motivation to do
a good job; please rank up to three in order of importance
Difficulties were caused by the presence of several concepts competing within thequestion wording. Terms such as currently, working life, on balance, up to three,
7/28/2019 RAG - Measuring Employee Motivation
22/70
JUNE 2011 Draft v6 REVISED Report 22
contributed to a degree of grammatical complexity that made the question difficult to
follow. This in turn led to any one or more of these terms being overlooked when
considering an answer.
In addition, this perceived complexity of the question, implied that providing answers
should be an equally complex task, prompting respondents to regularly double check to
ensure their understanding of the question was correct. However, even when doing so,
the key concepts; currentlyand on balance were overlooked by some respondents, as
they attempted to interpret a question as a whole.
Understanding the ranking upto three task:
This was an issue that affected both the ranking of motivational factors and the ranking of
aspects of pay and benefits. Essentially, respondents were unclear as to how to progress
where less than three answers had been provided at previous questions, or where less
than three answers were applicable. Again, due in part to the grammatical complexity of
the questions, the clue- up to threewas not always interpreted as no more than three.
Interestingly, where up to three was understood, in some instance respondents
uncomfortable with the notion of moving on to the next question, as it somehow felt
incomplete or unsatisfactory not to give a full answer.
4.1 Overall Recommendations
The way each question is currently worded does little to help respondents over the
difficult task of deciding how to answer where there are several considerations that affect
how they might respond. As a result, respondents are taking different approaches to
overcoming the dilemma, which risks introducing an element of instability into responses.
The recommendations below outline how each question has been revised to address
these overarching issues. The key overall recommendations that have arisen from the
cognitive test are:
Recommendation 1. Simplify the questions to make them easier to follow, by drawing
out key terminology into a separate clear instruction
Recommendation 2. Ensure simple clear instruction at each question and ensure that
guidance on completing the ranking questions is clear.
Recommendation 3. Place greater emphasis on directing respondents to answer on
the basis of how they feel at the current time
The following sections of the report will discuss each question in turn and provide more
detailed recommendations for revisions to each question.
7/28/2019 RAG - Measuring Employee Motivation
23/70
JUNE 2011 Draft v6 REVISED Report 23
5. QUESTION 1: MEASURING MOTIVATION
The first question required respondents to provide a self assessment of their motivation
at work along a 10 point scale.
CHART 1 TESTED QUESTION
The following questions are about your motivation at work.
1) In relation to your current working life, on a scale of 1 to 10 where 1 is extremely low and 10 is
extremely high, how motivated are you to do a good job?
1 Extremely low
2
3
4
5
6
7
8
9
10 Extremely high
Respondents had no problems in providing an answer for this question. The 10 point
scale was appropriate, with respondents tending to begin by answering at the positive
end of the scale and adjusting their answer as particular issues were recalled.
5.1 Motivation to do a good job
As in the previous research respondents had no problems understanding that thequestion related to their work, and no definition of motivation needed to be given. The
question taken into testing included the phrase to do a good jobwhich was added as a
result of the initial review of the question prior to testing. This addition was crucial to
ensuring respondents interpreted the question consistently. To do a good job was
associated with performance and outcomes, was very much considered to be an
objective focus of motivation. Alternative wordings were tested; to do your best in your
joband to do your bestat work, both of which seemed more subjective and associated
with effort, and self motivation. Respondents felt that to do a good job was both the most
appropriate measure, being the more tangible concept to consider, and easier to placeon a scale.
7/28/2019 RAG - Measuring Employee Motivation
24/70
JUNE 2011 Draft v6 REVISED Report 24
Recommendation 4. Retain the words to do a good jobas preference to to do your
best at work, or to do your best in your job.
5.2 Time frame: Current working life
The review process recommended that the word currentwas added to tighten the time
frame of reference around the term in relation to your working life. Respondents
interpreted current working life in one of two ways; either narrowly as my job, or in
slightly broader terms including my job as affected by mypersonal life. The addition of
currentalongside working life went some way to narrowing the frame of reference in
terms of theircurrent job, but did not explicitly encourage reference to the current point in
time.
As such, respondents answered the question based upon a broad range of time frames,from the time spent in their current job, to an arbitrary measure of the last 2 weeks. This
meant that respondents were applying an inconsistent time frame to this question, which
was then carried through and applied to the following questions.
Recommendation 5. Reposition currentwithin the question to break up the phrase
current working life.
5.3 Fit between question and answerAlthough not an issue raised by respondents it was apparent there was a mismatch
between the question and the answer framework. The question and answer can be read
out as follows:
Q. How motivated are you to do a good job...A. Extremely low / extremely high.
Although this was not an issue raised by respondents in this test, it may well become a
point of dispute in the future.
Recommendation 6. Reword the question to match the answer scale, by using the
phrase how would you describe your current motivation.
7/28/2019 RAG - Measuring Employee Motivation
25/70
JUNE 2011 Draft v6 REVISED Report 25
CHART 2 RECOMMENDED QUESTION
The following questions are about your motivation at work.
1) In relation to your working life, on a scale of 1 to 10 where 1 is extremely low and 10 isextremely high, how would you describe your current motivation to do a good job?
1 Extremely low
2
3
4
5
6
7
8
9
10 Extremely high
7/28/2019 RAG - Measuring Employee Motivation
26/70
JUNE 2011 Draft v6 REVISED Report 26
6. QUESTION 2: SELECTING FACTORS
The second question asked respondents to identify factors that influenced their
motivation positively and negatively from a neutrally worded list of factors. An answer
framework was designed that collected positive, negative and neutral views in one
question, rather than having to repeat the list of factors for each direction of influence.
CHART 3 TESTED QUESTION
Below is a list of factors that may affect how motivated you are.
2) Again, in relation to your current working life, on balance, how do each of the following factors
affect how motivated you are to do a good job?
Please provide an answer for each factor
Raises my
motivation
Neither
raises nor
lowers my
motivation
Lowers my
motivation
Dont
know
a. My Work Life Balance
...................................
w. Other Specify______________
6.1 Question and instruction wording
As discussed above, this question was considered difficult to follow and suffered from a
degree of grammatical complexity caused by inclusion of several concepts in the
sentence. Since respondents were continually referring back to the question if factors
required more consideration, this grammatical complexity failed to provide respondentswith the clear immediate understanding they were looking for.
The question and instructions sought to convey three key concepts:
To consider on balance where mixed considerations (i.e. where a factor
could both raise and lower motivation) could influence response
Consideration of theircurrentsituation, that is how they feel now
To answer once for each factor.
7/28/2019 RAG - Measuring Employee Motivation
27/70
JUNE 2011 Draft v6 REVISED Report 27
Minimis ing m ixed considerations:On balance
The review process recommended that a term such as overall or on balance, was
included at appropriate questions to overcome the anticipated difficulty of factors both
raising and lowering motivation. When probed, respondents were clear as to the meaning
of on balance within the questions, providing interpretations including all things
considered, oroverall. However, respondents also commented that balance may not
be the most appropriate concept. Those who found themselves bringing several
considerations to bear on a particular factor suggested that they were not weighing up
considerations, but referring to how they have felt at different times. When prompted,
respondents found it easier to reflect on their current situation, that is how they felt at
that moment in comparison to an earlier positive/negative period, than to attempt answer
on balance.
Reference period: current situation
As discussed at section 5.2, the placement of current alongside working life did not
explicitly encourage reference to the current point in time. As with question 1,
respondents responded to each factor based upon a broad range of time frames, most
commonly between 3 months (marking the New Year as a reference point) and 12
months (marking this time last year, as a reference point). When prompted to answer on
the basis of how they felt at that moment, respondents were able to come to a judgement
more easily, since it negated conflicting considerations originating from issues occurring
in the past. That is not to say past issues were discounted, as respondents noted that
events occurring even beyond a year ago still affect how they felt now.
Response instructions
The instruction to answer once for each factor was understood, and respondents all
provided an answer for every factor. Respondents assumed that only one answer was
allowed, but this was associated with the common convention of answeringquestionnaires rather than any interpretation of the instruction.
However, two issues emerged from testing question 2:
Factorwas found to be an overly technical term. It was understood, and it
was eventually clear what factors referred to, however it was an additional
interpretative task that respondents had to go through each time the question
was considered. Respondents own language commonly referred to issuesthat
affect their motivation
7/28/2019 RAG - Measuring Employee Motivation
28/70
JUNE 2011 Draft v6 REVISED Report 28
The instructions did not direct respondents consideration towards how they
felt at the current time. As discussed above, this was a key piece of guidance
that helped overcome difficulties formulating a response.
6.2 Answer framework
In general, the question approach worked well, and combining positive and negative
considerations into one question helped respondents make judgements about how to
answer against each factor. The impact of mixed considerations on response suggests
that to separate the question into what raises and what lowers risks encouraging
responses that indicate the same factors influencing motivation in both directions.
6.3 Neutral response options and opt outs
The answer framework included a neutral response Neither raises nor lowers my
motivationand an opt out to response as dont know. It was clear that the neutral
option was crucial to allowing respondents to answer both easily and in a way that
reflects how the factor actually affects them. There were however, subtle differences as
to how the neutral option, the dont knowoption, and a potential not applicable option
were interpreted.
Neither raises nor lowers my motivation
The neutral option was used in two ways:
The factor does not affect my motivation at all
The factorboth raises and lowersmy motivation.
This latter interpretation was symptomatic of the difficulty of having severalconsiderations for a factor, and was in effect, being used to express a polarising effect of
those considerations. However, used this way, the neutral option was the best fit as
without it, the only other way of responding would be to force respondents to answer
either raisesor lowers, or to respond as dont know.
Respondents who answered neutrally were presented with the prospect of having the
neutral option removed. Selecting instead, either raises or lowers felt to some
respondents to be a false response, and they did not feel comfortable with their answers.
They felt that that it discounted considerations being made at the other end of the scale
which may mislead those acting on the results. The dont knowoption was considered
Raises my
motivation
Neither
raises nor
lowers my
motivation
Lowers
my
motivation
Dont
know
7/28/2019 RAG - Measuring Employee Motivation
29/70
JUNE 2011 Draft v6 REVISED Report 29
by respondents as an entirely false alternative to a neutral option, as quite simply, they
did know.
Dont know
The dont knowoption was used in four ways:
As an alternative to the neutral option. In this case dont know became a
negative response, and there was preference for a clarification suggesting
cant say.
To express a genuine dont know response, particularly where respondents
felt they had too little information to give an informed judgement. For
example, at access to policies that allow flexible working, where respondents
were unaware of what policies were available to them, and what provision
was made for flexible working1.
Where the factor was not completely understood. An example here was
communication at work which was considered perhaps too ambiguous to
provide a certain answer to.
Where the factor did not apply to them at all, a good example being,
exercising authoritywith respondents suggesting they had no authority so
could not answer. In these cases, respondents suggested a response option
for not applicablewould be a better response.
Not applicable
A not applicable response option would certainly fit the way respondents wanted to
answer for some of the factors. The argument for its inclusion is that it would provide
some differentiation within the dont knowresponses. However, as a response itself, it
may also be of interest to understand what factors respondents think are not applicable
to them, for instance why is work/life balance not applicable to some employees?
The argument against its inclusion is its overlap with the neutral category. When probed
as to what not applicablemeans, respondents stated that it means it does not affectthem, or it is not relevant to their current job or lifestyle, both very similar to the
interpretation of the neutral category which was interpreted as having no effect on my
motivation.
Recommended approach for neutral and opt out options
The neutral and opt out options need to allow respondents to answer in a way that
expresses that a factor does not apply, that they are unable to provide an answer, or that
it has no affect. The evidence from the testing suggests that a not applicable response
1Ambiguities in factor wording are discussed at section 7.
7/28/2019 RAG - Measuring Employee Motivation
30/70
JUNE 2011 Draft v6 REVISED Report 30
may be confused or overlap with the neutral response (neither raises nor lowers).This
could be overcome by prefacing the neutral option with the issue applies to me but...
and including a not applicable answer option. This would distinguish the purpose of the
neutral option as being to identify where a factor has a neutral affect on motivation.
The question arises as to whether a not applicableoption should replace or sit
alongside a dontknowoption. The not applicableresponse option would address the
overlap in interpretation between dont know and the neutral option but removing the
dont knowoption does not allow a response where there is too little information or
knowledge to provide an answer.
The argument for including a separate dont knowresponse is to allow respondents to
answer where they have a limited knowledge or frame of reference to inform their
judgement, and want to opt out.
However, the argument for replacing a dont knowoption is that a not applicableoptionremoves the need for a dont knowoption. Arguably, if an issue is properly understood,
then it should either be relevant to a respondent or not. If relevant then, respondents
should be able to judge whether this issues raises or lowers motivation. As discussed
above, the dont knowoption in testing was used where a judgement could not be made,
but there was ambiguity as to whether the neutral or dont knowoption was the better
answer (driven in part by respondents aversion to answering dont know.
Therefore, the recommendation for these options is to replace the dont knowoption
with not applicable.
6.4 Recommendations
Recommendation 7. Replace the term factorwith issue, as this reflects respondents
own language. Reference should also be made to selecting a box
as opposed toproviding an answer.
Recommendation 8. Simplify the question to remove the grammatical complexity that
makes it difficult for respondents to follow. Remove the terms
currentand on balance.
Recommendation 9. Include an instruction explicitly guiding respondents to focus on
how they feel at the moment. The instruction should also
acknowledge that a bestanswer will enable respondents to
decide between the positive/neutral/negative responses, and
discourage ticking more than one response.
Recommendation 10. Ensure the instruction highlights that respondents should answer
for each issue, and make it clear that one box should be marked
for each issue.
7/28/2019 RAG - Measuring Employee Motivation
31/70
JUNE 2011 Draft v6 REVISED Report 31
Recommendation 11.At this point we would recommend replacing dont knowwith a not
applicable option so as to differentiate between neutral and not
applicable responses. Although there is potential for overlap and
confusion between the two, including the precursor Applies to me
but...resolves that ambiguity.
CHART 4 RECOMMENDED QUESTION
Below is a list of issues that may affect how motivated you are in your current job.
2) In relation to your working life, how do each of the following issues affect how motivated you
are to do a good job?
For each issu e, please tick the bo x that best descr ibes you r current s i tuat ion
Select one box only for each issue
Raises
my motivation
Applies but
neither
raises nor
lowers my
motivation
Lowers
my
motivation
Not
applicable
a. My Work Life Balance
...................................
w. Other Specify______________
7/28/2019 RAG - Measuring Employee Motivation
32/70
JUNE 2011 Draft v6 REVISED Report 32
7. THE LIST OF MOTIVATIONAL FACTORS
The factor list was considered long and somewhat daunting when first encountered.
However, once each factor had been considered, it was described as comprehensive,
relevant and resonated with the issues that respondents face in their own working lives.
7.1 Length of list and ordering of the motivational factors
The problems arising from the length of the list relates to respondents bringing mixed
considerations to bear on each factor. Where this was the case, respondents tended to
glance down, or back up, the list to see if their considerations had been covered
elsewhere in another factor. This did extend the time taken for respondents to complete
the list of factors.
There seemed to be few issues with ordering of the factors, but again there may be some
anticipation amongst respondents that like factors would be grouped together. However,
this may have an effect on response as respondents may be inclined to respond in the
same way within groups of factors. For this reason we recommend that, if used in online
surveys, the factors are rotated / randomised to ensure any potential order effect within
the list is minimised.
The testing did not find any evidence of ordering effects with the list on a paper based
questionnaire. Consideration has been given to grouping factors around similar themes.
However, this risks encouraging respondents to answer in the same way for groups
factors. Therefore, considering that no ordering affects were observed during testing, the
current list ordering is appropriate for paper versions.
Recommendation 12. Online surveys should rotate/randomise the order of the factors to
ensure any order effect within the lists is minimised. Paper
questionnaires should retain current ordering.
7/28/2019 RAG - Measuring Employee Motivation
33/70
JUNE 2011 Draft v6 REVISED Report 33
7.2 Wording of the motivational factors
The grid below outlines the original tested wording of the factors and any recommended
revision, along with commentary on how each factor was interpreted by respondents.
Tested Factor Revised Factor
My Work Life Balance No revisions
Respondents had no issues understanding this factor.
The extent to which I am treated with respect
in my workplace
No revisions
Respondents had no issues understanding this factor
My colleagues/team No revisionsRespondents had no issues understanding this factor
The level of pride I have in working for this
organisation
No revisions
There is a risk of inconsistent interpretation within review body remits as to what the
organisation refers to. Respondents felt that this organisation could be interpreted to
represent different levels of their workplace (for example, the NHS, this PCT, this hospital).
However, alternative approaches are likely to attract similar issues (such as ...in working
here ..in my place of work may be too limiting and still be ambiguous as to the level atwhich they should respond).
The level of control I have over my own work No revisions
More junior grades interpreted control to mean having control overwhat /how muchwork
they receive, whereas senior grades interpreted control to mean dealing with the work they
have. However, there was far greater variation in interpretation of an alternative managing
factor. The extent I am able to manage my work was interpreted to mean
competency/ability to manage workload.
The extent to which I am valued in my
workplace
No revision
Respondents had no issues understanding this factor
Material reward Pay and benefits
Before entering the testing phase OME suggested that material reward might not be well
understood and suggested testing pay and benefits as an alternative. Respondents
7/28/2019 RAG - Measuring Employee Motivation
34/70
JUNE 2011 Draft v6 REVISED Report 34
understanding of material rewardvaried, interpreting it both narrowly and broadly. Narrow
interpretations focussed on basic pay as the reward, whilst others included other elements
of pay such as bonuses, allowances and pension benefits. Amongst these respondents
material reward was a familiar term and was associated with pay. However, some
respondents were less familiar with the phrase, did not immediately associate the factor
with their pay or benefits. Instead, material reward, was interpreted as recognition or actual
awards presented by their employer, or even uniforms provided by the employer.
Pay and benefits was interpreted as being about pay, and essentially what respondents get
at the end of the month. This tended to include pension benefits, overtime payments and
allowances. It tended not to relate to bonuses or performance related pay, however these
are less frequently paid in the public sector.
Access to flexible working policies The extent to which I am allowed to
work flexibly
Respondents interpreted this factor in two ways. Flexible working policies were
understood to mean access to policies that allowed/promoted flexible working hours, home
working etc. However, because of the way the factor is phrased, it was also understood to
be referring to the flexibility of working practices in general.
An alternative suggestion made by OME was to revise to The extent to which I am allowed
to work flexibly, whilst an alternative generated from testing suggested Access to policies
allowing flexible working. The issue is what the factor is intending to measure: I can wo rk
flexiblyorthere are pol ic ies that al low me to work f lexib ly. The problems identified with the
former approach were that
1) when considering working flexibly, this was interpreted as a range of activities that
could or could not be conducted flexibly, as some aspects of a task are flexible, such as
managing workloads, whilst others are not, such as shift patterns and working hours. This
added to the difficulty making judgement as to how to answer.
2) that some professions such as policing, nursing, teaching work to stricter shift patterns,
and operational approaches making flexibility, (as interpreted as a work/life balance
concept) is not as relevant on a day to basis. However, such issues should be either
captured by a not applicable response, or as lowering motivation where the absence of such
flexibility impacts negatively.
The key to flexible working as a motivational factor was their organisations comm itment to
providing opportunities to employees to work flexibly if they needed it rather than the extent
to which they can actually do so themselves. This was the rationale behind a previous
suggested revision on Access to p ol ic ies allowing f lexib le working.
However OME have indicated that being allowed to work flexibly is likely to be a more direct
driver of motivation than whether flexible working policies are in place. Therefore the
recommended revision is The extent to which I am allowed to work flexib ly.
My line manager No revision
Respondents had no issues understanding this factor. Those respondents that did not have
a line manager as such, (for example Head Teachers) understood what the factor referredto and were able to transpose the concept onto a similar individual(s) that monitors their
7/28/2019 RAG - Measuring Employee Motivation
35/70
JUNE 2011 Draft v6 REVISED Report 35
performance.
The extent to which my work makes a
difference to people
No revision
Respondents had no issues understanding this factor. This was a key positive motivator forrespondents, and reflected respondents earlier consideration of what affects their
motivation.
My prospects for promotion/progression My prospects for promotion and
progression
Promotion and progression were understood to mean different things. Promotion referred to
a formal step up to a new pay scale, or new grade, and may be accompanied by a change of
title. Progression was interpreted as developing or moving up within a pay scale or grade.
Despite the two meanings, both were highly relevant, particularly for those respondents that
had no further immediate avenue for promotion. However, this broad interpretation comes
from placing the two words together. Respondents felt that two separate factors for
promotion and progression were unnecessary, as individually they were likely to be
understood to be referring to the same idea.
Status Status in the workplace
Status was considered vague, ambiguous and attracting a broad range of interpretations,
ranging from status in the workplace, status in the community, to current employment
status (e.g. contractor, temporary, full-time, part-time). Therefore, consideration needs to be
given as to whether the factor should refer to status in the work place or include status
outside of work.
If the factor is to refer to community status, the issue is whether this refers to a
respondents own community, or the community in which they work. Respondents
considered status in the workplace to be a relevant factor to motivation, and in some cases
more so than their social status in the community. Status in the community was also felt to
overlap with the factor referring to making a difference to people.
Being challenged by my work No revision
Respondents had no issues understanding this factor
Relationship between my work and my
personal values
No revision
Respondents had no issues understanding this factor. This was a key factor that
respondents were looking for, as it reflected their sense of self motivation to do their work
within the public sector. However, it was thought that any disconnect between personalvalues and work would need to be severe in order to warrant a negative response.
7/28/2019 RAG - Measuring Employee Motivation
36/70
JUNE 2011 Draft v6 REVISED Report 36
Respondents in this research found it difficult to visualise that kind of situation. Despite
this, there was no indication that the neutrality of the factor was compromised, as a negative
response could be envisaged, however severe the circumstances needed to be.
Level of variety in my work No revisionRespondents had no issues understanding this factor
Extent to which I enjoy my work No revision
Respondents had no issues understanding this factor
The extent to which I am treated fairly and
equally in my workplace
No revision
Respondents had no issues understanding this factor. Respondents tended to consider
whether they were treated fairly and equally, and if so, suggested that this should raise their
motivation, since it is naturally a positive thing to be treated as such. Equally, if they felt
they were not treated fairly and equally, this would lower their motivation. Respondents then
considered whether it actually affected motivation, or had become something that they
expected, and so had become a factor with little impact on motivation.
The difficulty appeared to be in mapping their self assessment of their treatment against its
affect on their motivation, rather than understanding of the factor itself.
Exercising authority No revision
Respondents had no issues understanding this factor; however, it was not thought to berelevant to more junior grades. In such cases respondents tended to answer in the neutral
as something that did not affect their motivation. Other alternatives were suggested, such as
making decisions that affect other people, as this was more inclusive and perhaps more
neutral. However, since this changes the meaning of the factor, (decision making vs.
authority) consideration will need to be given as to whether this alternative is an appropriate
measure.
Job security No revision
Respondents had no issues understanding this factor. Interestingly, a poor outlook on job
security, for some was described as a positive motivator. This was explained by one
respondent as a stick rather than acarrot, for example where employees have been warned
of a reduction in headcount, this can motivate to do a good job to ensure that they are
included in those that remain.
Extent to which I am able to develop new
skills on the job
No revision
Respondents had no issues understanding this factor. This was understood to mean
informal training and developing within a role, or learning from a close colleague or line
manager.
7/28/2019 RAG - Measuring Employee Motivation
37/70
JUNE 2011 Draft v6 REVISED Report 37
Opportunities for training and development No revision
Respondents had no issues understanding this factor. A key differentiatorbetween training
and development and new skills on the job is whether there is advance planning or a need
to book a course. There was potential overlap between development and skills on the job,
however respondents differentiated again by understanding development to be planned, and
perhaps in preparation for/enabling advancement, whereas skills on the job was
interpreted as being about developing skills to do their current job.
Communication at work Communication in my workplace
Communication at work was considered vague and potentially covers a range of
communication types. Foremost in respondents minds was work related communication
between colleagues and other teams. Respondents had to be prompted to consider whether
this could refer to internal communications about whats going on in the organisation, or
communication from the leadership in the organisation. Respondents felt the bigger issue
for their motivation is what they are told by their senior managers, particularly at a time of
change and uncertainty within the public sector.
However, Senior Managers interviewed, stated this either excluded them from the question,
or was asking them to evaluate their own communications, and that of their counterparts
within their organisation.
Despite some ambiguity/vagueness around the wording of the factor, all respondents were
able to provide and answer. Therefore the current recommendation is to adjust the wording
to reflect communications within the workplace, to give some clarity/direction as to what is
being asked,
Recommendation 13. Make changes to the factor list as indicated in the factor grid
above
Recommendation 14. Consideration needs to be given to four factors to confirm what
needs to be measured before agreeing the factor wording:
Exercising authority, (authority vs. decision making)
Status (workplace vs. community)
Communication at work (work related communication vs.
internal communication about the organisation/from
leadership)
Access to policies allowing flexible working
(organisations commitment to flexible working vs. ability to
take advantage of f lexible working)
7.3 Additional factors
Respondents struggled to suggest any further factors to add to the list. However, a small
number of factors emerged that were considered noticeable omissions or commonlycited by respondents. These were:
7/28/2019 RAG - Measuring Employee Motivation
38/70
JUNE 2011 Draft v6 REVISED Report 38
My personal well being: there was a strong feeling from one respondent that health and
well being will have an effect on motivation. Considering the focus that government has
given to the cost of sickness and absence on the economy, we recommend this factor is
given consideration for inclusion. OME suggested that this could be extended to My
personal well being and that of my family. However, there was no evidence from thetesting phase that suggests this is an addition that warrants inclusion.
The leadership of this organisation: The factors ask about relationships with
colleagues and line managers, but respondents noted an absence of opportunity to
comment on how senior management decision making and leadership affects their
outlook on their work. Again, in the current context, we recommend this factor is given
consideration for inclusion.
My working environment: This was not a factor mentioned explicitly by respondents,
but it has been suggested amongst the project team that working environment may be an
issue to cover. Since interviews were conducted at respondents home, their working
environment may not have been at the forefront of their minds. We agree that this
recommendation should be given consideration.
Recommendation 15. Give consideration to additional factors, given that respondents
considered the list reasonably long already:
My working environment
My personal well being
Leadership of this organisation
At present, on the basis of a small number of interviews, and in the
absence of quantitative testing we do not recommend that these
are included in the factor list; however, these issues may become
more relevant over time.
7.4 Other specify
The other specify option suffered from two issues:
i) Visibility: it did not stand out and was almost entirely overlooked, particularly after
what seemed to respondents an exhaustive list. Therefore pertinent information would
not have been collected, without being prompted to do so.
ii) Multiple entries: the succinct one line specifier and answer boxes assume that only
one other will be provided. This lead to respondents manually drawing in lines and
boxes, in order to complete the question.
7/28/2019 RAG - Measuring Employee Motivation
39/70
JUNE 2011 Draft v6 REVISED Report 39
Recommendation 16. Encourage respondents to give consideration to other factors, by
expanding the other specify factor, setting it apart from the list,
and provide space for more than one other to be recorded.
CHART 5 REVISED OTHER SPECIFY
w. If there are any other factors that affect your motivation pleasespecify these below
7/28/2019 RAG - Measuring Employee Motivation
40/70
JUNE 2011 Draft v6 REVISED Report 40
8. QUESTION 3: RANKING MOTIVATIONAL FACTORS
Questions 3a and 3b asked respondents to select and rank the three most important
factors from their previous answer (to question 2) those that raise motivation (3a) and
those that lower motivation (3b).
As with the task of identifying the direction of influence of each of the motivational
factors, ranking the top three was considered a challenging task. However, this difficulty
was exacerbated by:
confusion when trying to follow and unpick the question wording
understanding what the task required them to do
and how to record responses.
It was at this point in the questionnaire that some frustration set in amongst respondents
who did not have three answers to select from.
CHART 6 TESTED VERSION 1
3a) Of the factors that you have identified that currently raise your motivation to
do a good job; please rank up to three in order of importance
Please rank only those factors that you indicated raise your motivation
Insert 1 for most important factor, 2 for the second, 3 for the third.
Raises my
motivation
a. My Work Life Balance
...................................
w. Other Specify______________
7/28/2019 RAG - Measuring Employee Motivation
41/70
JUNE 2011 Draft v6 REVISED Report 41
8.1 Understanding the question
When asked about their initial response to seeing the question, the majority of
respondents were checking whether the list of motivation factors was the same or had
changed in any way. Respondents could spend up to a minute checking to see if the
factors were the same, supposedly in case of any tricksor attempts to trip them up.
However, this action did have the effect of drawing the eye down the list and away from
the question and instructions. Essentially, the question can be the last piece of
information on the page to be read. This, perhaps combined with a familiarity with the
factor list, led to a tendency to skip read the question and move straight on to recording
responses in the answer grid. Both versions, either the different response options in
Version 2 or the single column in Version 1 prompted these respondents to check the
question.
The question itself was understood, with respondents picking up that they needed to
refer back to their previous answers. Respondents could rarely recall how they answered
and so needed to physically refer back to check.
The term rankwas understood, but considered an extremely rigid term to apply to a
series of motivational issues that essentially could not easily be ordered along a single
continuum. Respondents articulating their understanding of the question referred tochoosingwhich factors were most important to them.
It also required an additional step of interpretation that contributed to respondents
consistently overlooking the crucial phrase up to three, seemingly distracted by either
trying to recall how they answered the previous question, or searching back to check,
and then beginning to decide which where the most important answers.
Recommendation 17. Introduce the question with a bold statement indicating that this
question is asking respondents to do something different to
previous questions
Recommendation 18. Replace the word rankwith choose, reflecting the language
used by respondents.
Recommendation 19. Reword the question by removing up to threefrom the question
and create an explicit note to say they can select up to three.
Recommendation 20.Add the word issueto the end of each response option for
clarification
7/28/2019 RAG - Measuring Employee Motivation
42/70
JUNE 2011 Draft v6 REVISED Report 42
8.2 Understanding the task
In general the task was understood. Respondents referred back to previous answers,
they deliberated as to which were most important and they recorded answers
appropriately. In the main this was completed successfully by most respondents for
raising motivation, yet respondents were still misdirected by confusion as to how torecord answers and what to do if they could not. The problems encountered related to:
Not being able to rank their answers, either because they are all equally
important or unimportant in one way or another
Failing to give consideration to their other options and including them in the
ranking process
In Version 1, marking the boxes with a tick or cross, rather than marking
numerically
Ticking more than one box per column, essentially expressing multiple factorsas the most important, second most important etc.
In all cases respondents eventually realised their mistake without probing, usually by
noticing an instruction or registering threein the question.
A more severe issue was generally encountered at the lowers motivation question. The
majority of respondents had marked fewer than three factors that lowered their
motivation. By this point respondents understood the task, and were aware that threefactors is what the question was looking for. However, without three factors to refer to,
respondents looked to the questionnaire for guidance on what to do next. It was clear
that up towas not clear or explicit enough to prompt respondents to move onto the next
question once none, one or two factors had been ranked, where less than three factors
were available. Respondents reacted to this by:
Closing the questionnaire, and not moving onto question 4 until prompted to
do so by the interviewer
Marking however many factors they had selected at question 2 and moving to
question 4. Although this was what the question intended them to do, this
made these respondents feel that the question and their response was
incomplete.
In either event, respondents looked to the question and failed to immediately register the
significance ofup toin the question.
7/28/2019 RAG - Measuring Employee Motivation
43/70
JUNE 2011 Draft v6 REVISED Report 43
Recommendation 21. Reiterate the consistency check with an instruction that these
most important factors should be selected from factors selected at
question 2 (for respondents who do not read the question first).
Recommendation 22. Provide an instruction that permits leaving the question
unfinished, and give clear sign posting telling respondents whatto do next
8.3 The answer frameworks
A preferred version was presented, followed by an alternative version with a different
answer framework. The order in which each approach was presented was rotated across
interviews. The questions below illustrate the raise motivation questions. This was
followed by an identical question asking about factors that lowermotivation.
CHART 7 TESTED VERSION 2
3a) Of the factors that you have identified that currently raise your motivation to
do a good job; please rank up to three in order of importance
Please rank only those factors that you indicatedraiseyour motivation
Select one factor only in each column
Most
Important
1
Secondmost
important
2
Third most
important
3
a. My Work Life Balance
...................................
w. Other Specify______________
Respondents expressed no strong preference for either of the two answer frameworks.
On balance, version 2 was considered a better approach as it appeared more familiar
and closer to questionnaires or forms that they had seen before. Of the two, version 1
was found to have the greatest risk of response error. Version 1 gave an impression of
being incomplete, due to the replication of a singular response category from Q2 (i.e.
raises motivation for question 3a and lowers motivation for question 3b) and the area of
blank space next to the answer column. This encouraged one respondent to duplicate
responses from question 2 before realising the illogic of that. The response categories at
version 2 were thought to prevent that, and clearly stated that something different was
being asked at this question prompting respondents to read the question and instruction
with greater care.
7/28/2019 RAG - Measuring Employee Motivation
44/70
JUNE 2011 Draft v6 REVISED Report 44
Recommendation 23. Version 2 is the recommended approach as this minimises
potential response error, and allows answers to be recorded in a
way consistent with the rest of the survey.
Recommendation 24. Reformat the other specify factor to encourage consideration of
those issues when choosing the most important.
CHART 8 RECOMMENDED VERSION
We would now like you to choose the three most important issues that ra ise your
motivation, and place them in order of importance.
3 a) Thinking of the issues that you have selected that raiseyour motivation to do a goodjob, please select the issues that affect your motivation the most in order of
importance to you.
You can select up to three issues
Please select from the issues that you s elected that ra ise your m ot ivat ion in quest ion 2 I f you cannot select three issues, please answer as best you c an, then go to qu est ion 3b
Tick one box only in each column
The most
Important issue
1
The second
most important
issue
2
The third
most important
issue
3
a. My Work Life Balance
-----------------------------------------------------------------------------------------------------------------
w. Any other issues that raise your motivation
7/28/2019 RAG - Measuring Employee Motivation
45/70
JUNE 2011 Draft v6 REVISED Report 45
9. QUESTION 4: ASPECTS OF PAY AND BENEFITS
The final questions asked respondents to select the aspects of their pay and benefits that
raised (4a) and lowered (4b) their motivation to do a good job.
On the whole the question structure was familiar, and so there were fewer issues with
completing the question at this point. The main issues encountered were;
understanding of the aspects of pay and reward listed;
mapping their own pay package to the aspects listed;
understanding how to answer where less than three aspects apply.
For this question, a preferred answer framework was presented to respondents, with two
alternative approaches tested after the questions were completed. The order in which the
answer frameworks were presented was rotated for each interview. Illustrated below are
the tested versions of the questions for raising motivation. Versions 1 and 2 were
followed by identical questions, asking for aspects that lower motivation.
CHART 9 TESTED VERSION 1
Answ er quest ion 4 only i f youidentified material rewardas a factor that either raises or
lowers your cu rrent level of mot ivat ion in quest ion 2
Below is a list of aspects that make up pay and benefits
4a) Please rank in order of importance up to threeaspects of pay and benefits that currentlyraiseyour motivation to do a good job,
Please insert 1 for most important aspect, 2 for the second, 3 for the third.
Raises my motivation
1. Basic salary level
9. other (please specify) e.g. season ticket loan, childcare vouchers,
subsidised canteen____________________________________
7/28/2019 RAG - Measuring Employee Motivation
46/70
JUNE 2011 Draft v6 REVISED Report 46
CHART 10 TESTED VERSION 2
Answer quest ion 4 only i f youidentified material reward as a factor that either
raises or lowers your c urrent level of motivat ion in quest ion 2.
Below is a list of aspects that make up pay and benefits
4a) Please rank in order of importance up to three aspects of pay and benefits
that currently raiseyour motivation to do a good job.
Select one aspect only in each column
Most
Important
1
Second
most
important
2
Third most
important
3
1. Basic salary level
10. None of the above
CHART 11 TESTED VERSION 3
4 Please rank in order of importance up to threeaspects of material reward that
currently raiseyour motivation to do a good job, and up to three aspects ofmotivation that currently loweryour motivation to do a good job.
Please insert 1 for most important aspect, 2 for the second, 3 for the third.
Raises my
motivation
Lowers my
motivation
1. Basic salary level
10. None of the above
9.1 Routing
Question 4 versions 1 and 2 were tested with a routing introduction, instructing that only
those who answered that pay and benefits either raised or lowered their motivation at
question 2 should proceed. However, those that gave a neutral option at Q2 felt that theyshould also be answering the Q4, since there may have been mixed considerations in
7/28/2019 RAG - Measuring Employee Motivation
47/70
JUNE 2011 Draft v6 REVISED Report 47
their response at question 2 that polarised their response to neither raises nor lowers.
This was proven to be the case as respondents who answered neutrally at question 2
were able to select answers that both raise and lower at the pay and benefits question.
Subsequent interviews were tested with the routing instructions removed, and
respondents responding neutrally at question 2 still felt the pay and benefits question tobe relevant to them.
Recommendation 25. Route all respondents through the pay and benefits question
9.2 Answer framework
Of the three versions of the answer framework, version 3 was considered to be the most
difficult to understand. Respondents attributed this to a long, wordy question and an
involved task of:
Sorting through aspects to check which apply
Deciding which aspects raised and which aspects lower
Deciding which are the most important before recording an answer
This was also combined with a certain level of frustration that across the four question
types this would introduce a fourth style of answer framework to interpret. There was a
clear preference to use the answer framework that was used at question 3. Therefore,
respondents generally found version 2 to be the preferred approach.
9.3 Understanding of task
The question was well understood by this point, being described as having to choose
three again. However, once again, there were issues registering up to, with
respondents focus on picking out three aspects. When probed, respondents revealed
that they felt the question was actually asking for three, and felt almost compelled to
provide three answers even though they understood up toto be allowing fewer.
However, it was very difficult for some respondents to identify three aspects, and so
there was a potential for their answers to reflect how an aspect could potentiallyaffectmotivation. For example ifbasic salary levelwas to fall, then it could become something
that lowers motivation, but at present it may have no effect.
Consequently, there is a risk that respondents may select the factors for both raising and
lowering motivation. This was attributed to this being the final question of the set, and a
lack of a clear instruction to end if less than three aspects apply. During the interviews,
suggesting that respondents consider how they felt about each aspect at the current time
clarified the intention of the question and enabled respondents to move through the
factors more easily.
7/28/2019 RAG - Measuring Employee Motivation
48/70
JUNE 2011 Draft v6 REVISED Report 48
Recommendation 26. Provide an instruction that permits leaving the question
unfinished, and give clear sign posting telling respondents what
to do next.
Recommendation 27. Include an instruction explicitly guiding respondents to focus on
how they feel at the moment. The instruction should also
acknowledge that a bestanswer will enable respondents to
decide between the positive/neutral/negative responses, and
discourage ticking more than one response.
9.4 Understanding aspects of pay and benefits
A key issue with the question was the relationship between the list of aspects and
respondents understanding of their pay and benefits. When probed at question 2, as to
what was included in material reward/pay and benefits, the top of mind aspects werebasic pay, pension package and allowances. On further probing, season ticket loans,
subsidised travel and canteens and performance related pay were common additions.
This indicated that the problem was not understanding the aspects, but recalling their
own pay and benefits package, and mapping that to the appropriate aspects in the
question. An example of this is to consider subsidised canteens. Since it was listed as an
example, this prompted a respondent to include it in their answer. On further probing it
was revealed that they would have been unlikely to include subsidised canteens if it had
not been stated within the response option. This indicated that respondents were notrecognising any logic behind how the other pay aspects were named, but using the
examples within the heading as a prompt to direct their response.
Since the aspects themselves were reasonably well understood, few changes have been
suggested to the list of aspects.
Tested Aspects Revised Aspects
Basic salary level Basic salary level
Latest pay settlement / review Latest pay settlement / review
Performance-related pay elements, e.g.bonuses, profit sharing
Performance-related pay, for example:bonuses, profit sharing
Access to pay increments/ structuredpay progression
Access to pay increments/ structured payprogression
Other pay elements, e.g. overtime pay,shift pay premiums, unsocial hourspayments
Other pay elements, that is paymentsreceived on top of your basic pay, forexample: allowances, overtime pay, shift
premiums, unsocial hours pay
7/28/2019 RAG - Measuring Employee Motivation
49/70
JUNE 2011 Draft v6 REVISED Report 49
Pension benefits being built up Your pension package
Other non-pay benefits, e.g. healthinsurance, car/car allowance
Other benefits, that is benefits that are paidfor by your employer, for example: healthinsurance, use of a car, subsidised canteen
Other (please specify) e.g. season ticketloan, childcare vouchers, subsidisedcanteen
If there are any other aspects of your payand benefits that raise your motivationplease specify these below
One significant addition to the list of aspects is an option to select none of the above.
Respondents who felt unable to answer either of the