Dattner Consulting, LLC Personnel Selection How to assess and develop individuals and teams http://www.dattnerconsulting.com/presentations/indteamassessdev.pdf pp. 2 - 22 Why employment interviews are often unreliable and poor predictors of job performance, and how to improve them http://www.dattnerconsulting.com/presentations/interviews.pdf pp. 23 - 45 A handbook for training interviewers to design and conduct interviews that more accurately predict candidates' job performance http://www.dattnerconsulting.com/presentations/interviewsmanual.pdf pp. 46 – 67 Psychometric tests that organizations use to select employees http://www.dattnerconsulting.com/presentations/selecthandbook.pdf pp. 68 – 87 How to ensure the legality of pre-employment cognitive ability and personality tests http://www.dattnerconsulting.com/presentations/legal.pdf pp. 88 – 89 Sample report based on psychometric assessments and structured interview- candidate recommended http://www.dattnerconsulting.com/posassess pp. 90 – 93 Sample report based on psychometric assessments and structured interview- candidate not recommended http://www.dattnerconsulting.com/negassess pp. 94 - 96 “Measuring hiring managers” http://www.dattnerconsulting.com/hrmagazinejune06 pp. 97 – 101 The Brian Lehrer Show on WNYC Radio: The Cult of Personality Testing http://www.dattnerconsulting.com/lehrer.mp3
Transcript
Dattner Consulting, LLC
Personnel Selection How to assess and develop individuals and teams http://www.dattnerconsulting.com/presentations/indteamassessdev.pdf
pp. 2 - 22
Why employment interviews are often unreliable and poor predictors of job performance, and how to improve them http://www.dattnerconsulting.com/presentations/interviews.pdf
pp. 23 - 45
A handbook for training interviewers to design and conduct interviews that more accurately predict candidates' job performance http://www.dattnerconsulting.com/presentations/interviewsmanual.pdf
pp. 46 – 67
Psychometric tests that organizations use to select employees http://www.dattnerconsulting.com/presentations/selecthandbook.pdf
pp. 68 – 87
How to ensure the legality of pre-employment cognitive ability and personality tests http://www.dattnerconsulting.com/presentations/legal.pdf
pp. 88 – 89
Sample report based on psychometric assessments and structured interview- candidate recommended http://www.dattnerconsulting.com/posassess
pp. 90 – 93
Sample report based on psychometric assessments and structured interview- candidate not recommended http://www.dattnerconsulting.com/negassess
HANDBOOK OVERVIEW PART I: SELECTION INTERVIEW OVERVIEW
Introduction to Interviews
Multiple Phases of the Selection Interview
Types of Interviews
Selection Interview Questions
• Credentials and Technical Information
• Experiences
• Opinions
• Behavior Descriptions
PART II: ENHANCING THE INTERVIEW
Preparing for the Interview
Developing Structure
Conducting the Interview
Planning Timing and Logistics
Questioning Effectively
Listening Actively
Using Nonverbal Cues
Eliminating Errors and Biases
Legal Considerations
Closing the Interview
I n t e r v i e w e r T r a i n i n g H a n d b o o k
PART I: INTRODUCTION TO INTERVIEWS
The employment interview has historically been and remains the single most common and important personnel selection tool, despite overwhelming evidence that interviews have low reliability and validity.
Although the judgments that interviewers make are often highly subjective and not often
predictive of a candidate’s job performance, research has shown that there are two ways to enhance the accuracy of interviews:
Create a standard, structured interview format for all applicants Train interviewers to be consistent in the way they interview and evaluate candidates
Unstructured Interview Candidates are informally interviewed, and different questions are asked of each applicant. Behavior Description Interview Candidates are asked about what they did in prior job situations that are similar to situations they may encounter on the job for which they are being considered. The interviewer may also ask discretionary probing questions for details about the relevant situations, the interviewee’s behavior in the situation, and the outcome. The interviewee’s responses are then scored with “behaviorally anchored” rating scales developed by job experts. Situational Interview Candidates are interviewed about what they would do in various job-related situations. The job-related situations are usually identified using the “critical incidents” job analysis technique. The interviews are then scored using a scoring guide developed by job experts. Comprehensive Structured Interview Candidates are asked questions about what they did and/or would do in various job-relevant situations in order to demonstrate their job knowledge, and ability to fulfill worker requirements. This kind of interview assesses how likely it is that the candidate will perform well in all aspects of the job. Interview Panel The job candidate gives oral responses to job-related questions asked by a panel of interviewers. Each member of the panel then rates each interviewee on such dimensions as work history, motivation, creative thinking, and presentation. This technique may not be feasible for jobs in which there are a large number of applicants that must be interviewed.
Traditional or Behavior Description Hiring Methods?
A recent review of published research (over 150 studies) on interviews found one-on-one unstructured interviews averaged only 19% predictive accuracy.
Unstructured panel interviews only rose to 35% accuracy. Research-based behavioral interviews, however, achieved an accuracy of up to 80%- a four-
fold improvement.
The best predictor of future performance is past performance in similar situations.
The more recent the past behavior, the greater its predictive power. The more longstanding the behavior, the greater its predictive power.
I n t e r v i e w e r T r a i n i n g H a n d b o o k
The purpose of the selection interview is to gather information that is specifically relevant to the position for which interviewers are hiring.
Since interviewers usually get exactly what they ask from their candidates, they need to be
careful about how their questions are structured in order to elicit the most useful information.
There are four types of information gathered in an interview:
Credentials and Technical Information
Experiences
Opinions
Behavior Descriptions
CREDENTIALS AND TECHNICAL INFORMATION QUESTIONS CREDENTIALS questions are an opportunity to obtain information about a person’s education, employment history, and past achievements, etc., which is also verifiable through other sources. For example:
1. What degrees do you hold? 2. What was your major? 3. Do you have a driver’s license? 4. What size budget did you manage? 5. How long did you work at your last job?
Advantages These questions can provide good clues about what the person knows and can do and are useful in the screening process. Disadvantages They do not provide information about what a person will choose to do in a given situation. NOTE: Many questions related to biographical information are illegal. (i.e.: Where were you born? What is your marital status?) See “Legal Considerations” section at the end of this handbook.
I n t e r v i e w e r T r a i n i n g H a n d b o o k
TECHNICAL questions determine if a candidate has the technical knowledge necessary to do a job. For example:
1. How do you write a Microsoft Access query? 2. How do you conduct a job analysis? 3. What costs do you consider while calculating a return on investment (ROI)?
Advantages Technical questions can provide clues to whether an applicant actually has the knowledge that his/her credentials indicate. Disadvantages Although this information is valuable, it may be more economical to collect it in a work sample test. The result of asking technical questions is that one can get a fairly good idea of whether or not the person can do the job.
Make a note of the credentials or technical type of question(s) you usually ask in an interview. Is there a more effective way of getting this information from the candidate? If so, how?
EXPERIENCE QUESTIONS EXPERIENCE QUESTIONS provide information about what the applicant has done in the past. These are general “work experience” questions and are the most commonly asked questions in interviews. For example:
1. What were your duties in your last job? 2. Tell me about your responsibilities in your last position. 3. Describe a typical day in your company. 4. What do you do when a customer complains in an unfriendly way?
Advantages Work experience is important and these questions provide an overview of what the applicant has done in the past. Such questions can be helpful in probing for more specific information about performance. Disadvantages By themselves, these questions do not give any information about how well the applicant performed the described task or activity. Even if the candidate did perform the task or activity well in the last situation, it may have involved different behaviors than those required for the new job.
Make a note of the experience type question(s) you usually ask in an interview.
Is there a more effective way of getting this information from the candidate? If so, how?
I n t e r v i e w e r T r a i n i n g H a n d b o o k
OPINION QUESTIONS OPINION QUESTIONS yield information that describes what the applicant thinks about a given topic. Opinion questions elicit self-evaluations, self-reports of strengths/weaknesses, likes/dislikes, and goals. For example:
1. Describe your strengths and weaknesses. 2. Why are you applying for a job here? 3. What do you like best about your job? 4. What do you think is the most important part of your job? 5. How do you feel about working overtime?
Advantages Opinion questions reveal areas where the interviewer can seek more specific behavioral information. They provide information about what the applicant thinks is important and allow for further probing relevant to the job opening. Disadvantages Interviewers may end up making assumptions about the candidate’s behavior as a result of the answers to opinion questions. They do not provide any real evidence about what the candidate currently does, or would do in the specific position for which he/she is being interviewed.
Make a note of the opinion-type question(s) you usually ask in an interview. Is there a more effective way of getting this information from the candidate? If so, how?
BEHAVIOR DESCRIPTION QUESTIONS BEHAVIOR DESCRIPTION QUESTIONS request detailed accounts of specific events from the applicant’s past. For Example:
1. Please describe a specific technical challenge that challenged your technical skills. 2. Tell me about the most difficult client you faced last year. How did you prepare? How did
you respond to the client’s concerns? 3. Think about a time when you noticed the early warning signs of a problem which would have
been costly if not detected. When did you first notice the problem? What was the first thing you did to correct it?
I n t e r v i e w e r T r a i n i n g H a n d b o o k
Advantages The information obtained from these questions allows the interviewer more objectively to assess applicant accomplishments and competencies, yielding more accurate hiring decisions. Disadvantages Behavior Description interviews are time consuming. Interviewers also require more skill and effort to ask and score these types of questions well.
Make a note of the Behavior Description type question(s) you usually ask in an interview. Is there a more effective way of getting this information from the candidate? If so, how?
PART II: ENHANCING THE INTERVIEW As an interviewer, you are given the very important task of helping your organization select new members. The more accurate you are in your predictions of who will be a successful employee, the more successful your organization will be. Best Practices:
Preparing for the Interview
Developing Structure
Conducting the Interview
Planning Timing and Logistics
Questioning Effectively
Listening Actively
Using Nonverbal Cues
Eliminating Errors and Biases
Legal Considerations
Closing the Interview
I n t e r v i e w e r T r a i n i n g H a n d b o o k
PREPARING FOR THE INTERVIEW
ADVANCED PREPARATION is crucial to any interviewing process. Without it, your interview could be disorganized and unfocused. The following steps are suggested:
Clearly define the position and competencies necessary for successful performance in the position.
Prepare an interview strategy. Decide in advance how you are going to keep a record of the interview i.e. note taking or tape
recording.
POSITION DESCRIPTION AND ASSESSMENT CRITERIA DESCRIBING THE ROLE AND THE SELECTION CRITERIA must happen in the earliest stages of the interview process:
Level of performance expected Knowledge and skills Attitudes and feelings Special qualifications for to the position
The above are criteria that are intended to ensure that interviews will be valid and fair for all candidates. If used consistently, criteria provide your organization with a uniform yardstick against which each interviewee can be objectively and fairly measured. Principal role of Job Analysis is:
To assess job content so that knowledge, skills, abilities, and other requisite employee specifications can be identified
To develop selection measures such as tests, interviews, and work samples to asses knowledge, skills and abilities
Job Analysis Employee Specifications Selection Instrument Development
One way to establish criteria is study the job specifications and determine the abilities, skills
and competencies, and personal attributes necessary for performing all specified job functions Another way is to talk to job incumbents or other employees who know what it takes to do the
job well
RECORDING ANSWERS ACCURATE RECORDS of candidate responses are a critical input when interviewers synthesize the data to make a hiring decision. There are two effective alternatives: Audio Cassette Recording Pros:
Provides a clear record of responses that can be reviewed later Allows the interviewer to listen more effectively during the interview Provides objective evidence of an applicant’s responses should the interviewer ever have to
testify in a court case Cons:
May cause initial discomfort for the applicant in some cases Can increases the time required during assessment if the tape needs to be reviewed in order
for the interviewer to rate responses Best Practices:
Alert the candidate to the recorder and obtain his/her permission to tape the session. Inform the candidate of how the recording will be used and who may listen to it.
Note Taking Pros:
Focuses assessment of applicant answers on critical points in an efficient way Provides a record of questions asked and answered during the interview
Cons: Increases the possibility of missing key details Taking detailed notes may slow down the interview
Best Practices: Take notes throughout the interview or not at all in order to avoid shaping responses. Do not to take notes on just things that you view to be negative or positive. Take notes that are descriptive and not evaluative and avoid jumping to conclusions. Give stories a label and write down key words to facilitate subsequent recall. Note the frequency and recency of behaviors when possible.
I n t e r v i e w e r T r a i n i n g H a n d b o o k
DEVELOPING STRUCTURE STRUCTURE enables interviewers to work easily and comprehensively through a candidate’s credentials and history. For example:
1. Greeting and small talk 2. Transition Statement 3. Overview
4. Educational Background
5. Work History
6. Job-Related Outside Activities
7. Self-Assessment
8. Firm and Job Information/
Q & A
9. Further Action/Close
Establish rapport Direct the conversation to the interview Let the candidate know how you plan to conduct the interview. Verify degrees, coursework, etc., given on the resume and/ or application form. Explore all work experience. Collect data chronologically Ask questions about work-related activities such as clubs, hobbies, and volunteer work. Adhere to EEO guidelines. Ask for candidate’s assessment of his or her strengths and weaknesses Give relevant information about the job and the firm. Answer candidate’s questions. Sell the job as appropriate. Outline next steps in your firm’s decision-making process. Thank candidate and close.
The structure will vary depending on whether the interview is a one-on-one or a panel interview. The following should be kept in mind regarding panel interviewing:
Panel interviewing saves time, builds team consensus and buy-in, and sends a message to candidates regarding the importance of team collaboration in the workplace.
A decision is based on the opinions of several people and therefore less subjective. Make sure that the participation of panel members is equal and that no single member
dominates the interview. The roles of team interviewing should be based on the skills, knowledge, and abilities of
individual team members. Identify who will be the lead interviewer; who will moderate the structure, flow, and time of
each interview. Before the panel interview:
Identify members of the interview team and schedule a team meeting (this activity is typically performed by the lead interviewer):
o Draft a distribute a memo to team members regarding the meeting o Attach a copy of the position description o Attach a copy of the candidate’s resume
Lead or participate in the interview team meeting o As a group determine which person will handle specific areas and probe questions
during the interview o Draw up a schedule of interviewing times
I n t e r v i e w e r T r a i n i n g H a n d b o o k
The lead interviewer needs to send a reminder to the interview team several days before the
interview o Attach a list of areas to probe o Attach an interview appraisal form
Review the position description, competencies, and candidate’s resume Develop a list of probing questions related to your company’s areas to probe
o Make sure the questions do not violate EEO guidelines Gather the materials that will be needed during the interview, including:
o Note taking form o Interview appraisal form o Business card
NOTE: It is important to not overwhelm a candidate with multiple interviewers firing questions without some sense of sequence or coordination between panelists.
CONDUCTING THE INTERVIEW CONDUCT in the first few moments of an interview is vitally important. This is when initial impressions (of both interviewer and candidate) are formed. Initial Contact People make very quick judgments about one another when they first meet. Ideally, you should give applicants as similar an experience as possible of being greeted and brought to the room for the interview. This will enable more accurate comparison of candidates than would be possible if different interviews begin differently. A few words of small talk, for example, about directions or travel to the interview location, will set the candidate at ease.
CULTURAL DIFFERENCES Be aware of cultural differences when greeting interviewees: in some cultures, men greet each other with an embrace; in others, men and women alike simply bow. The handshake, in various forms, is the most universally accepted and generally the safest option.
THE IMPORTANCE OF ESTABLISHING RAPPORT Create a positive rapport with the candidate by standing up to greet him/ her. Lean toward the candidate and make appropriate eye contact. This encourages candidates to relax.
I n t e r v i e w e r T r a i n i n g H a n d b o o k
Using a Structuring Statement A structuring statement is a clear statement that lets an applicant know what will happen in the interview and puts him/her at ease. A good structuring statement should include:
A brief background about your experience with the company The position, title, and department for which the candidate is being interviewed The amount of time required for the interview and what you want to accomplish in that time Mention of your note-taking or tape-recording process An explanation of the nature of the questions that will be asked
Give an example of a structuring statement that you use while interviewing candidates.
PLANNING TIMING AND LOGISTICS TIMING AND LOGISTICS require preparation and sets the tone for the ensuing interview. The following diagrams illustrate some seating arrangements for the interview process. Key: interviewer candidate Face-to-face suggests formality Side-by-side implies cooperation Meeting at a workspace creates relaxed atmosphere Formal Interview Informal Interview Interview in your office
Make alternative dates and times available for holding interviews in case candidates are unable to attend on suggested days.
Schedule interviews with a generous amount of time between them. Interviews should be conducted in quiet places, free from interruption. Try to avoid sitting behind a desk.
What space and set-up do you use and what impact does it have on the candidate?
QUESTIONING EFFECTIVELY QUESTIONS—the variety and precision—are vital in determining the success of the information exchange between you and each candidate. There are seven different types of questions: Open-Ended
Solicits broad, opinion-based, direct responses Allows respondent to take initiative in answering as they see appropriate Provides valuable insights regarding the candidate’s priorities, style, thought processes, etc. Example: “How have your previous jobs prepared you for greater responsibilities?”
Closed-Ended
Seeks verification of information by narrowing the choice of answers to either yes or no You can recognize closed-ended questions by the first words of the sentence:
o Are you/ Were you…? o Do you/ Did you…? o Can you/ Could you…?
Example: “Have you performed this type of work before?” Use a closed-ended question when you’ve targeted a point of information: “Did you intend to
complete the requirements for the master’s program?” “How many employees did you supervise?”
Probing
Seeks more information on previous statements Considers whether candidates have answered a question to your satisfaction. If not, continue
probing Example: “Could you please say more about your team leadership role?”
Paraphrasing:
Mirrors the content and feelings of what the candidate is saying in your own words Example: “Let me see if I understood what you were saying about…”
I n t e r v i e w e r T r a i n i n g H a n d b o o k
Asks for specific information on a defined topic. Directs the candidate’s answers to a narrow focus for evaluating knowledge, skills, and
abilities. Example: “What do you find challenging about supervising a design team?”
Problem-solving:
Describes a hypothetical or real scenario to check for common understanding and knowledge from respondents.
Asks for an explanation of how a situation would be handled. Example: “How would you handle an irate customer?”
Behavioral Description and Follow-up Probes
Behaviorally based interview questions should focus on a candidate’s past behavior, and should include coverage of the situation, what the candidate did, and what the result was.
A behavior description interview is a new experience for most selection candidates The candidate may be unprepared because they are not used to talking so descriptively about
themselves and their personal accomplishments Interviewers also need to be sensitive to the varied cultural backgrounds of the candidates
which may impact the extent to which candidates provide detailed descriptions to the interviewer
Thinking of experiences takes time and energy Example:
“Please describe a situation where you had to use influence to achieve a goal. Why was influence necessary? What was your strategy for using influence? What obstacles did you encounter? How effective were you in influencing?”
LISTENING ACTIVELY ACTIVE LISTENING is one of the most essential abilities of an interviewer. Consider:
Summarize what a candidate is saying to show that you are listening. Keep checking with the candidate that you have understood everything you have been told. Be curious. It helps you listen properly. Ensure you understand a candidate’s question before answering. Jot down questions as they occur to you. Listen to the emotions behind the words. Correct any bad listening habits that you have. Keep a check on your body language.
I n t e r v i e w e r T r a i n i n g H a n d b o o k
Develop listening skills to facilitate effective interviews: Physical Listening Signals
Look at the candidate Sit upright; vary your posture Smile occasionally Use gestures Be yourself
Conversational Listening Dialogue Listening dialogue techniques have 3 benefits:
Keep your active listening level high Gives the interview a conversational tone Encourages the candidates to provide more and better information.
Incorporate conversational listening techniques:
Comment occasionally Echo significant words or phrases Rephrase for understanding, clarification, or summarization
Commenting: an added statement of your own to what the candidate has just said. Keep the comment brief and relevant to the same subject
Interviewer: Why did you choose this as your career path? Candidate: I got interested in this subject when I was in college Comment: I also got interested in this field while in college Candidate: Yes- I feel fortunate to have discovered my interest when I was still in school and could take relevant courses.
The benefit of the interviewer’s comment is threefold:
The questioning remains conversational The interviewer demonstrates active listening Candidate presents in greater detail his/ her career orientation
Echoing: a listening dialogue technique by which you repeat verbatim a key word or phrase the candidate said.
Interviewer: Why are you leaving your current job? Candidate: I can’t see much of a growth potential. My boss is sort of standing in the way. Echo back: Standing in the way? Candidate: Yes. He seems to keep all the credit for himself so I don’t get much visibility.
By echoing back a selected word/ phrase, you encourage the candidate to contribute additional information. This serves as a very subtle probe and also shows that you are listening actively.
I n t e r v i e w e r T r a i n i n g H a n d b o o k
Rephrasing: a restatement of the candidate’s thought using different words.
Interviewer: Why are you changing careers? Candidate: I love teaching but the pay isn’t enough. Rephrase: So you feel intrinsically rewarded but not adequately extrinsically rewarded. Candidate: No matter how hard I work, my paycheck stays the same. I want to be rewarded for my extra efforts
By recapturing the candidate’s thoughts in other words, the interviewer demonstrates that he/she not only heard the response but also understood it.
USING NONVERBAL CUES NONVERBAL COMMUNICATION shapes the tone and direction of the interview. In particular, interviewers need to make sure that they do not send negative nonverbal signals during the interview. These might include:
Looking away from the candidate Looking at your watch often Watching people walking by A bored facial expression Slumping or restlessly shifting in your chair
However, interviewers also need to be on guard that they do not send too many positive nonverbal signals, as these may shape candidates’ responses. Appropriate nonverbal cues include:
Eye contact with the candidate A friendly but not overly relaxed posture An alert facial expression
You should use nonverbal cues to indicate general attention and interest, but not to punish or reward any specific thing that a candidate says. Lastly, different cultures have different norms of body language and personal space. Where some people are open and tactile even with strangers, others feel discomfort if you sit or stand too closely.
I n t e r v i e w e r T r a i n i n g H a n d b o o k
ELIMINATING ERRORS AND BIASES ERRORS AND BIASES corrupt interview validity. As interviewers, you need to be aware of common errors and biases in interview evaluations.
Leniency Error: all candidate’s are rated superior Central Tendency Error: all candidates are rated as average Stringency Error: all candidates are rated as poor Contrast Effect: the impression of the previous candidate influences the evaluation of the
current candidate First Impression Error: evaluations are based on the first impression of the candidate “Similar-to-me-Error”: evaluations are favorable because candidate is similar in some way(s)
to the interviewer Guarding against error and bias:
Treat each area of inquiry in an interview independently and make no assumptions about
performance Review the objective rating standards for each question before the interview Review the final rating profile of an applicant; if you see a large number of high or low
ratings, or large number of ratings in the middle (for example, too many “3”s), and limited variance, you may want to consider whether errors and biases are the reason for this pattern.
I n t e r v i e w e r T r a i n i n g H a n d b o o k
LEGAL CONSIDERATIONS THE LAW describes appropriate and inappropriate interview protocol. In general, one should only ask job-relevant questions and questions which do not probe about irrelevant information about an applicant’s personal life, personal history or membership in any demographic group or protected class. Equal Employment Opportunity Commission (EEOC) and Americans with Disabilities Act (ADA) legal and illegal questions:
Age o Illegal: How old are you? When did you graduate from high school? o Legal: None
Race or national origin
o Illegal: What is your race? Where were you born? o Legal: None
Citizenship
o Illegal: Are you an American citizen? o Legal: Can you document your right to work in the United States?
Religion
o Illegal: What is your religion? Do you attend church/synagogue/mosque? o Legal: None
Marital status
o Illegal: Are you married? Have you ever been married? What was your maiden name? What kind of work does your husband/wife do?
o Legal: None
Family status o Illegal: Where do you live? Who do you live with? Do you have children? How
many? How old are they? Who takes care of them? Are you pregnant? Do you plan to have children?
o Legal: Are there any factors, which would prevent you from meeting your work obligations or getting to work on time?
Disabilities and health
o Illegal: Do you have any disabilities or health problems? o Legal: Can you fulfill the essential functions of the job?
Arrest record
o Illegal: Have you ever been arrested? o Legal: Have you ever been convicted of a (relevant) crime?
I n t e r v i e w e r T r a i n i n g H a n d b o o k
CLOSE interviews with a clear statement to the candidate about next steps. Your closing statement should include:
Thanking the applicant for his or her time What next steps will be – e.g.: further interviews, a meeting of the hiring committee, etc. The time frame before next steps When the applicant will next hear from the organization
Please write a brief closing statement for discussion.
I n t e r v i e w e r T r a i n i n g H a n d b o o k
TESTING AND ASSESSMENT OVERVIEW COGNITIVE ABILITIES TESTS PERSONALITY AND TEMPERAMENT TESTS SALES ABILITIES TESTS
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
2
EMPLOYEE SELECTION: TESTING AND ASSESSMENT
TESTING AND ASSESSMENT OVERVIEW
STEPS IN THE DEVELOPMENT OF A SELECTION PROGRAM MAJOR CONSIDERATIONS WHEN TESTING
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
3
TESTING AND ASSESSMENT OVERVIEW: STEPS IN THE DEVELOPMENT OF A SELECTION PROGRAM
Job Analysis
Identification of Relevant Job Performance Dimensions
Identification of Knowledge, Skills, Abilities (KSAs) Necessary for the Job
Development of Assessment Devices to Measure KSAs
Validation of Assessment Devices
Use of Assessment Devices in Processing Applicants
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
4
TESTING AND ASSESSMENT OVERVIEW: MAJOR CONSIDERATIONS WHEN TESTING Appropriateness
Validity Reliability Legality
Usefulness
Job Responsibilities Industry Level Multiple Measurements Report Options and Quality
Logistics
Scoring/Results Turnaround Time Ease of Use Administration Options Cost
User Experience
Face Validity Time to Complete
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
5
EMPLOYEE SELECTION: TESTING AND ASSESSMENT
COGNITIVE ABILITIES TESTS
WATSON-GLASER CRITICAL THINKING WONDERLIC PERSONNEL TEST MULTIPLE APTITUDE BATTERY-II
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
6
COGNITIVE ABILITIES TESTS: WATSON-GLASER CRITICAL THINKING (WGCTA) Source: The Psychological Corporation http://www.pantesting.com/products/PsychCorp/WGCTA.asp Purpose: Predict an employee’s career path based on critical thinking skills Attributes/Abilities Assessed:
Inference Recognition of assumptions Deduction Interpretation Evaluation of arguments
Logistics:
80 items Approximately 60 minutes to complete Paper/pencil administration; hand-scoring
Sample Item: Mr. Brown, who lives in the town of Salem, was brought before the Salem municipal court for the sixth time in the past month on a charge of keeping his pool hall open after 1 a.m. He again admitted his guilt and was fined the maximum, $500, as in each earlier instance. On some nights it was to Mr. Brown’s advantage to keep his pool hall open after 1 a.m., even at the risk of paying a $500 fine.
COGNITIVE ABILITIES TESTS: WONDERLIC PERSONNEL TEST (WPT) Source: Wonderlic http://www.wonderlic.com/ Purpose: Measures cognitive ability as an accurate predictor of employment success Attributes/Abilities Assessed:
Candidate’s ability to: learn a specific job solve problems understand instructions apply knowledge to new situations benefit from specific job training be satisfied with a particular job
Logistics:
50 items Exactly 12 minutes to complete Paper/pencil administration; hand-scoring
Sample Items: Q1: Two people caught 36 fish; X caught 8 times as many as Y. How many did Y catch? Q2: Are the meanings of the following two sentences: 1. similar, 2. contradictory, 3. neither similar or contradictory? It is always well to moor your ship with two anchors. Don’t put all your eggs in one basket. Output: Score which is compared against norms
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
8
COGNITIVE ABILITIES TESTS: MULTIPLE APTITUDE BATTERY-II (MAB-II) Source: Sigma Assessment Systems http://www.sigmahr.com Purpose: Assess candidate’s aptitudes and intelligence Attributes/Abilities Assessed:
Verbal IQ Performance IQ Full Scale IQ
Logistics:
Subtests, which can be individually or group administered 7 minutes to complete each subtest; 100 minutes to complete full battery Paper/pencil administration; Mail-in scoring
Sample Items: Q1: In throwing three dice together, what are the chances of obtaining three 5’s?
a. 1 in 6 b. 1 in 15 c. 1 in 18 d. 1 in 36 e. 1 in 216
Q2: A synonym of insipid is:
a. treacherous b. enduring c. poisonous d. fearless e. dull
Output: Scales with interpretive instructions: “…indicating a significantly higher degree of aptitude relating to knowledge of diverse facts, suggesting a wide range of interests and an excellent long term memory.”
Vigilance Abstractedness Privateness Apprehension Openness to Change Self-Reliance Perfectionism Tension
Logistics:
185 items Approximately 35-50 minutes to complete Online administration and scoring
Sample Items: Q1: I get new ideas about all sorts of things, too many to put into practice.
True – ? – False Q2: I let little things upset me more than they should. True – ? – False Output: Basic Interpretive Report “At times, Mr. X may show the self-discipline and conscientiousness needed to meet his responsibilities. At other times, he may be less restrained, following his own wishes.” “Typically Mr. X tends to take things in stride and adapt to circumstances.”
300 items Approximately 30-60 minutes to complete Paper/pencil administration; mail-in scoring
Sample Items: Q1: You give little thought to your failures after they are past. Y – ? – N Q2: You like to sell things (that is, to act as a salesperson). Y – ? – N Q3: You often crave excitement. Y – ? – N Q4: You speak out in meetings to oppose those who you feel sure are wrong.
Y – ? – N Output: Interpretive Report “He usually avoids being surrounded by other people, and he tends not to initiate conversations with strangers.” “He does not take things too seriously and is inclined to act impulsively.”
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
12
PERSONALITY AND TEMPERAMENT TESTS: NEO PERSONALITY INVENTORY, REVISED (NEO PI-R) Source: Psychological Assessment Resources http://www.parinc.com Purpose: Provide a general description of normal personality Attributes/Abilities Assessed:
Conscientiousness Agreeableness Neuroticism Openness to experience Extraversion
Logistics:
243 items Approximately 45 minutes to complete Paper/pencil administration; mail-in scoring
Sample Items: Q1: I am not a worrier. Strongly Agree – Agree – Neutral – Disagree – Strongly Disagree Q2: If I don’t like people, I let them know it. Strongly Agree – Agree – Neutral – Disagree – Strongly Disagree Q3: Sometimes I’m not as dependable or reliable as I should be. Strongly Agree – Agree – Neutral – Disagree – Strongly Disagree Output: Scores and Narrative Report “He is forceful and dominant, preferring to be a group leader rather than a follower.” “This individual is rational, prudent, practical, resourceful, and well-prepared.”
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
13
HOGAN PERSONALITY INVENTORY Source: Hogan Assessment Systems 800-756-0632
http://www.hoganassessments.com/HPI.aspx Purpose: Can be used either for selecting or developing high-caliber employees Attributes/Abilities Assessed: 7 dimensions that influence occupational success; scales divided into percentile scores (high=above 65th percentile, average=between 35th and 65th percentile, low=below 35th percentile):
206 True/False items 15-20 minutes to complete Paper and pencil or online
Certifications: User must have a BA or advanced degree in psychology or a related field, or have received accreditation through Hogan Systems; workshops are $1,100 and are usually held in Atlanta, GA Output: PDF file, various report options available Cost: $150 onetime setup fee, plus $40 per report
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
14
EMPLOYEE SCREENING QUESTIONNAIRE (ESQ) Source: Sigma Assessment Systems http://www.sigmahr.com/ Purpose: Predict positive and counterproductive work behaviors Attributes/Abilities Assessed:
Org commitment and job satisfaction
Customer service Productivity Accuracy Promotability Employee theft Absenteeism OTJ alcohol and drug abuse
Lateness Sabotage Driving delinquency Security, confidentiality, and safety violations
Loafing Unauthorized giveaways to friends and family
Logistics:
27 items Approximately 15 minutes to complete Online administration and scoring
Sample Item: Of the four statements below, select the Most and Least:
I get along with people at parties quite well. Changes in routine bother me. I often make people angry by teasing them. If someone gave me too much change, I would point it out.
Output: ESQ Narrative Report “Will not approach clients to seek out ways to be helpful.” “Demonstrates care and attention when performing the job.”
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
15
EMOTIONAL QUOTIENT INVENTORY (EQ-I)
Source: Multi-Health Systems http://www.mhs.com/ Purpose: To assess the emotional intelligence of job candidates Attributes/Abilities Assessed:
Interpersonal relationships Social responsibility Empathy
Adaptability
Problem solving Reality testing Flexibility
General Mood
Happiness Optimism
Logistics: 133 items Approximately 30 minutes to complete Paper and pencil or online administration and scoring
Certifications: “B-level” certification from MHS Output: Individual Summary Reports provide the overall EQ score, scores for each of the 15 areas measured, and scores for validity indices.
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
16
EMPLOYEE SELECTION: TESTING AND ASSESSMENT
SALES ABILITIES TESTS
SIGMA SURVEY FOR SALES PROFESSIONALS CALIPER FIRST STEP FOR SALES WALDEN SALES SKILLS TEST
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
17
SALES ABILITIES TESTS: SIGMA SURVEY FOR SALES PROFESSIONALS (3SP) Source: Sigma Assessment Systems http://www.sigmahr.com/
Purpose: Selection and placement for a wide range of sales positions
Attributes Assessed:
Technical Orientation
Creativity Thoroughness Risk Taking Open Mindedness First Impression Interpersonal
Relations Sensitivity Social Astuteness Communication
Motivation Self Discipline Flexibility Independence Self-Esteem Emotional Control
Dependability Ambition Assuming
Responsibility Vision Organizational
Spokesperson Short-term
Planning Strategic Planning Productivity
Logistics:
352 items Approximately 35-40 minutes to complete Online administration and scoring
Sample Items: Q1: “I am careful to plan for my distant goals.”
Strongly Disagree – Agree – Neutral – Disagree – Strongly Disagree Q2: “If someone criticizes the way I work, I point out their deficiencies.”
Strongly Disagree – Agree – Neutral – Disagree – Strongly Disagree Output: 3SP Report “Warm helpful people, like Mr. X, are quite ready to assist customers with creative solutions.” “His self-confidence makes him particular effective at promoting ideas, products, or services.”
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
18
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
19
SALES ABILITIES TESTS: CALIPER FIRST STEP FOR SALES Source: Caliper http://www.calipercorp.com
Purpose: Assess and eliminate candidates who do not have the qualities needed to succeed in sales
75 items Approximately 20 minutes to complete Online administration and scoring
Sample Items: Q1: Select the term that is Most and Least descriptive of you:
Solitary Assertive Impulsive Consistent
Q2: Complete the analogy: bruise – fall; bump - ? Wound Lump Knock Limp
Q3: I cannot stand being distracted once I’m involved in something. Strongly Disagree – Agree – Neutral – Disagree – Strongly Disagree Output: Narrative Report “In a sales situation, her motivation to persuade will often fuel her efforts to convince her clients to make a commitment.” “…she should naturally seek out opportunities to be with prospects and clients.”
DATTNER CONSULTING, LLC WWW.DATTNERCONSULTING.COM
20
SALES ABILITIES TESTS: WALDEN SALES SKILLS TEST Source: Walden Personnel Testing and Consulting http://www.waldentesting.com/ Purpose: Assess the knowledge, skills and abilities required to succeed in sales positions, including Selecting Sales Representatives, District Sales Managers, and Sale Clerks Attributes Assessed:
Knowledge of general sales principles Knowledge of sales terms Relevant vocabulary skills Understanding of issues that can affect the sales process Ability to deal with several sales situations Basic mathematical and calculation skills Logic and attention to detail
Logistics:
6 items Exactly 65 minutes to complete Online administration and scoring
Sample Items: Q1: Below are seven key steps (a to g) in the selling cycle. They are in random order. On the lines provided, place the digits 1 to 7 to indicate the best logical order to execute these steps for a successful sale.
a. addressing concerns b. presentation of the product c. getting referrals d. closing the sale e. prospecting f. qualification g. original contact
Output: Candidate Score and Hiring Recommendation “With an overall score of 81%, Ms. Logan strongly demonstrates the skills needed to succeed in a sales position.”
Dat
tner
Con
sulti
ng, L
LC
ww
w.d
attn
erco
nsul
ting.
com
Lega
l gui
delin
es fo
r per
sona
lity
test
sLe
gal g
uide
lines
for p
erso
nalit
y te
sts
To m
axim
ize
test
ing
effe
ctiv
enes
s an
d m
inim
ize
lega
l ris
ks, t
he A
mer
ican
Bus
ines
s La
w J
ourn
al
prov
ides
the
follo
win
g te
n gu
idel
ines
for t
he u
se o
f per
sona
lity
test
s fo
r per
sonn
el d
ecis
ions
in
empl
oym
ent s
ettin
gs:
1.1.O
nly
train
ed p
rofe
ssio
nals
sho
uld
adm
inis
ter a
nd e
valu
ate
pers
onO
nly
train
ed p
rofe
ssio
nals
sho
uld
adm
inis
ter a
nd e
valu
ate
pers
onal
ity
ality
te
sts.
Thi
s w
ill in
crea
se th
e pr
obab
ility
that
test
resu
lts a
re
test
s. T
his
will
incr
ease
the
prob
abili
ty th
at te
st re
sults
are
inte
rpre
ted
inte
rpre
ted
accu
rate
ly.
accu
rate
ly.
2.2.O
nly
use
test
s th
at a
re d
evel
oped
by
expe
rts in
the
test
ing
fiel
Onl
y us
e te
sts
that
are
dev
elop
ed b
y ex
perts
in th
e te
stin
g fie
l d a
nd a
re
d an
d ar
e co
nsis
tent
with
pub
lishe
d pr
ofes
sion
al s
tand
ards
. co
nsis
tent
with
pub
lishe
d pr
ofes
sion
al s
tand
ards
. 3.3.
Test
dat
a, in
clud
ing
spec
ific
answ
ers
and
raw
sco
res,
sho
uld
be
Test
dat
a, in
clud
ing
spec
ific
answ
ers
and
raw
sco
res,
sho
uld
be
conf
iden
tially
mai
ntai
ned.
Rec
omm
enda
tions
and
con
clus
ions
resu
lco
nfid
entia
lly m
aint
aine
d. R
ecom
men
datio
ns a
nd c
oncl
usio
ns re
sul ti
ng
ting
from
the
use
of p
erso
nalit
y te
sts
shou
ld o
nly
be d
isse
min
ated
tofro
m th
e us
e of
per
sona
lity
test
s sh
ould
onl
y be
dis
sem
inat
ed to
auth
oriz
ed
auth
oriz
ed
indi
vidu
als
for j
obin
divi
dual
s fo
r job
-- rel
ated
pur
pose
s.
rela
ted
purp
oses
. 4.4.
Em
ploy
ers
shou
ld a
void
put
ting
labe
ls o
n an
indi
vidu
al th
at m
ay
Em
ploy
ers
shou
ld a
void
put
ting
labe
ls o
n an
indi
vidu
al th
at m
ay in
hibi
t in
hibi
t fu
ture
job
oppo
rtuni
ties
or a
ffect
the
pers
onfu
ture
job
oppo
rtuni
ties
or a
ffect
the
pers
on’’ s
per
sona
l life
.s
pers
onal
life
.5.5.
Em
ploy
ers
shou
ld u
se th
e m
ost u
pdat
ed v
ersi
on o
f an
avai
labl
e te
Em
ploy
ers
shou
ld u
se th
e m
ost u
pdat
ed v
ersi
on o
f an
avai
labl
e te
st. F
or
st. F
or
exam
ple,
the
MM
PI
exam
ple,
the
MM
PI -- 2
elim
inat
es m
any
offe
nsiv
e te
st q
uest
ions
that
app
ear
2 el
imin
ates
man
y of
fens
ive
test
que
stio
ns th
at a
ppea
r on
the
orig
inal
MM
PI.
Use
of u
pdat
ed te
sts
may
hel
p to
elim
inat
eon
the
orig
inal
MM
PI.
Use
of u
pdat
ed te
sts
may
hel
p to
elim
inat
epo
tent
ial
pote
ntia
l in
vasi
on o
f priv
acy
clai
ms.
inva
sion
of p
rivac
y cl
aim
s.
(Kim
berli
R. B
lack
: “P
erso
nalit
y S
cree
ning
in E
mpl
oym
ent”
Am
eric
an B
usin
ess
Law
Jou
rnal
, Mar
ch 1
994;
32,
1)
Con
tinue
d→
Dat
tner
Con
sulti
ng, L
LC
ww
w.d
attn
erco
nsul
ting.
com
Lega
l gui
delin
es fo
r per
sona
lity
test
s Le
gal g
uide
lines
for p
erso
nalit
y te
sts
(con
tinue
d)(c
ontin
ued)
6.6.E
mpl
oyer
s sh
ould
use
test
s in
con
junc
tion
with
oth
er s
cree
ning
E
mpl
oyer
s sh
ould
use
test
s in
con
junc
tion
with
oth
er s
cree
ning
te
chni
ques
. Thi
s w
ill in
crea
se th
e lik
elih
ood
of m
akin
g an
acc
urte
chni
ques
. Thi
s w
ill in
crea
se th
e lik
elih
ood
of m
akin
g an
acc
urat
e at
e as
sess
men
t of a
n in
divi
dual
asse
ssm
ent o
f an
indi
vidu
al’’ s
abi
lity
to d
o w
ell i
n a
job.
s ab
ility
to d
o w
ell i
n a
job.
7.7.E
mpl
oyer
s sh
ould
obt
ain
a jo
b ap
plic
ant
Em
ploy
ers
shou
ld o
btai
n a
job
appl
ican
t ’’ s o
r em
ploy
ees
or e
mpl
oyee
’’ s w
ritte
n co
nsen
t s
writ
ten
cons
ent
befo
re a
dmin
iste
ring
an e
xam
.be
fore
adm
inis
terin
g an
exa
m.
8.8.O
nly
use
test
s th
at h
ave
been
sho
wn
to b
e re
late
d to
the
job
in
Onl
y us
e te
sts
that
hav
e be
en s
how
n to
be
rela
ted
to th
e jo
b in
que
stio
n.
ques
tion.
9.9.
Em
ploy
ers
shou
ld s
elec
t a te
st th
at is
com
mon
ly u
sed
for t
he p
arE
mpl
oyer
s sh
ould
sel
ect a
test
that
is c
omm
only
use
d fo
r the
par
ticul
ar jo
b tic
ular
job
in q
uest
ion.
For
inst
ance
, pro
ject
ive
mea
sure
s ar
e pr
edom
inan
tlyin
que
stio
n. F
or in
stan
ce, p
roje
ctiv
e m
easu
res
are
pred
omin
antly
used
in
used
in
the
clin
ical
set
ting
and
are
not r
eadi
ly a
dapt
able
to th
e em
ploy
the
clin
ical
set
ting
and
are
not r
eadi
ly a
dapt
able
to th
e em
ploy
men
t con
text
m
ent c
onte
xt
10.
10.
Em
ploy
ers
shou
ld c
onsi
der t
he n
eeds
of d
isab
led
indi
vidu
als
whe
nE
mpl
oyer
s sh
ould
con
side
r the
nee
ds o
f dis
able
d in
divi
dual
s w
hen
cond
uctin
g pe
rson
ality
test
s. T
he A
mer
ican
s w
ith D
isab
ilitie
s A
cco
nduc
ting
pers
onal
ity te
sts.
The
Am
eric
ans
with
Dis
abili
ties
Ac t
(AD
A)
t (A
DA
) pr
ohib
its th
e us
e of
test
s th
at s
cree
n ou
t ind
ivid
uals
with
dis
apr
ohib
its th
e us
e of
test
s th
at s
cree
n ou
t ind
ivid
uals
with
dis
a bilit
ies.
bi
litie
s.
(Kim
berli
R. B
lack
: “P
erso
nalit
y S
cree
ning
in E
mpl
oym
ent”
Am
eric
an B
usin
ess
Law
Jou
rnal
, Mar
ch 1
994;
32,
1)
DATTNER CONSULTING, LLC 6 Washington Place, #161G New York, NY 10003 Phone: (212) 501-8945 Fax: (212) 658-9285
JANE DOE - ASSESSMENT
OVERVIEW Ms. Doe is a calm and confident executive who projects a strong leadership presence. In her view, she is well positioned and well prepared to be the president and CEO of Acme, Inc. There were a few times during the interview when she said things like “when I start the job” instead of “if I start the job.” However, she did stop herself after these statements and smiled, saying “I know, I don’t have the job yet.” Given the circumstances under which she left her last job, and the fact that she has had over two years off, it is not surprising that she is somewhat defensive when asked about her career progression. Ms. Doe also stated that she has been given feedback in the past about being defensive, and that she has endeavored to improve in this area. A positive interpretation of her defensiveness is that it is a reaction to occasions where Ms. Doe hasn’t been able to achieve the level of success or performance that she expects of herself, and these high standards are an important motivator for her.
INTELLECTUAL EFFECTIVENESS Ms. Doe clearly has the intellectual capacity for succeeding in a senior managerial or executive level role. She is in the 99th percentile on the Watson-Glaser Critical Thinking Appraisal and in the 91st percentile on the Wonderlic Personnel Test. The personality assessments, however, indicate that Ms. Doe has only a moderate level of intellectual curiosity. Ms. Doe is verbally expressive and articulate, but not always concise. She seemed to view the interview as more of an exercise in narrating her past job experiences than as an opportunity to answer specific questions. However, she was able to subtly weave in her qualifications for the role throughout the interview.
INTERPERSONAL SKILLS Ms. Doe is very friendly, likable and charismatic. The assessments indicate that others are likely to seek her out for sympathy and support, and during the interview it was clear that she prioritizes being empathic and supportive. At the same time, she is also likely to be candid, and even blunt. For example, when she was asked during the interview why Large Corporation, Inc. had sold the company she was working for after such a short period of time, she boldly answered “a better question to ask is why they bought it in the first place.” Some colleagues are likely to appreciate this openness and no-nonsense approach, but others may prefer more tact and diplomacy. Another example of her candor was when she was asked about how the Acme role would challenge her, she answered that she didn’t want to sound arrogant, but that she feels that her prior career experience has fully prepared to be successful.
DATTNER CONSULTING, LLC Jane Doe Assessment June 15, 2005 Page 2 of 4
STRATEGY AND TACTICS FOR GROWTH
Although the assessments describe Ms. Doe as valuing “both the new and the familiar” they also indicate that she will “not seek out novelty for its own sake.” It may be worth considering whether the demands of the role for which she is being considered will necessitate a higher degree of creativity than Ms. Doe is likely to demonstrate. In describing what strategies would be necessary to grow Acme’s business, she spoke mainly in general terms, saying “we would need to do” various things without going into much detail about how. She was not sure if it would be necessary to write a new business plan or to work from an existing one, but said she would be happy to do either. When asked if she had ever written a business plan she replied that at various jobs “we” had written business plans, but it seemed that she had never been responsible for writing a business plan by herself. It may be worthwhile to ask her for further details about what her approach for growing the business would be, and/or her analysis of the pros and cons of Acme’s current strategy in terms of both top line and bottom line growth. The assessments indicate that Ms. Doe’s preference might be to prioritize the quality of work over the quantity of work, so if she is offered and accepts the role it may make sense to clearly communicate to her senior management’s view of how inevitable tradeoffs between quantity and quality of products can be optimized.
WORK ETHIC
The assessments indicate that Ms. Doe is likely to be fast-paced, forceful and energetic, as well as highly determined and persistent. She cares deeply about professional success, and described willingly working long hours and weekends in order to get the job done. The assessments confirm that Ms. Doe “tends to set aside her own wishes in order to meet responsibilities.” She described how important academic success was to her as a student, and how her professional ambitions are a continuation of her academic ambitions. Ms. Doe also described how important it is to her to be able to recruit employees who share her work ethic.
ATTENTION TO DETAIL AND FOLLOW-UP
The assessments reflect Ms. Doe’s tendency to be “moderately well organized and fairly reliable” with “an average amount of self-discipline.” On a more positive note, she is also described as “rational, prudent, practical, resourceful and well prepared.” She likes getting deep into details herself so that she doesn’t have to rely on others to filter information for her and mentioned learning how to use databases so that she could do her own queries and analyses. It would likely be helpful to ensure that if she is hired she recruits some staff members who are more detail-oriented than she is.
DATTNER CONSULTING, LLC Jane Doe Assessment June 15, 2005 Page 3 of 4
MOTIVATING EMPLOYEES
Ms. Doe’s personality assessments indicate that she is able to balance being firm with being sympathetic, and that her social skills are solid. Therefore, she will probably be a likable boss without being a particularly inspirational leader. Her friendliness and open-mindedness do provide benefits in this area, in that Ms. Doe is likely to find out what employees are motivated by rather than making untested assumptions about their motivations. Ms. Doe also seems likely to be able to be tough when necessary, and described how she has been quite capable of terminating underperformers after giving them sufficient feedback and opportunities to improve. As with teaching where Ms. Doe “did much better with the gifted kids than the challenged kids” she is likely to feel more comfortable managing a strong team for growth than turning around a troubled team. One thing that is very important to Ms. Doe is making sure that employees always know what is expected of them so that “there are no surprises.” She empathizes with employees and described how she had been frustrated and upset in situations where expectations had not been clearly communicated to her. Overall, she is likely to be successful in motivating the team that she builds and leads.
RECRUITING AND TEAM BUILDING
According to the assessments, Ms. Doe is emotionally well-adjusted, and has a high level of confidence and self-esteem. She also is likely to take a leadership role in groups, and to generate some enthusiasm. These attributes are likely to help her in recruiting and team building. She described how much she enjoys the opportunity to build a team, and to fill an organization with people who will bring a sense of urgency and a level of performance similar to her own. She is particularly proud of having built some high-performing sales teams and her relatively high emotional intelligence will boost the chances that she will hire the right people.
ORGANIZATIONAL CULTURE BUILDING
Ms. Doe has a positive and happy disposition, and is therefore likely to set a positive emotional tone for the organization that she leads. She is also likely to model and build a culture of candor and accountability in which employees will feel encouraged to speak up when they see opportunities for processes and systems to be improved. She described the importance of culture and would likely pay careful attention to the development of a new organizational culture if she is offered the job.
DATTNER CONSULTING, LLC Jane Doe Assessment June 15, 2005 Page 4 of 4
COMPETENCIES AND DEVELOPMENT NEEDS Probable Competencies
• A high level of analytic, math and verbal intelligence • Strong motivation to succeed in growing a business • Work ethic, dedication, and passion for education and publishing • Friendliness, calm, confidence and charisma • Candor and willingness to speak up, even about sensitive issues
Probable Developmental Needs
• Developing a clear strategy and tactics for Acme’s growth • Demonstrating diplomacy and demonstrating tact when necessary • Being too defensive at times instead of soliciting and learning from performance feedback
SUMMARY AND RECOMMENDATIONS In conclusion, Ms. Doe has the cognitive abilities and personality attributes that would likely enable her to be successful in the role for which she is being considered. She has a high level of intelligence, the right degree and kind of motivation, and a personality profile that is likely to be a good fit with the role and the organizational culture. Therefore, she is recommended for the position of President and CEO of Acme Publishing.
The above summary is based upon the results of the NEO-PIR, the Wonderlic Personnel Test, the Watson Glaser and the 16PF tests, and an interview which was conducted on June 15, 2005. These tests of cognitive ability and personality are based on professionally conducted research and modern scale construction techniques. The results should be interpreted as probabilities since neither job performance nor human behavior can be predicted with one hundred percent accuracy. This report is intended to be used in conjunction with other job relevant information gathered during the screening process. Ultimately, a final decision for any job candidate should be based on a combination of the results detailed above and other sources, such as candidate education, work experience, references, presentations, and face-to-face job interviews.
Report prepared by: Ben Dattner, Ph.D. [email protected] 1-212-501-8945
DATTNER CONSULTING, LLC 6 Washington Place, #161G New York, NY 10003 Phone: (212) 501-8945 Fax: (212) 658-9285
JOHN DOE - ASSESSMENT
WORK APPROACH
The assessments indicate that Mr. Doe is an energetic, dominant, bold leader who is highly independent-minded and forms his own opinions. Friendly to strangers and likely to make a good first impression, Mr. Doe is likely to be a good conversationalist. He is conscientious, neat, punctual and well organized, and is likely to strive for excellence in any endeavor he undertakes. He is willing to challenge the status quo, and to experiment with new and innovative ways of doing things, although he can also respect the value of tradition. He likes to control others and his environment, and can be persistent until he gets his way. Socially fearless and uninhibited, he is comfortable in group situations, is willing to share his true thoughts and feelings with others, and is likely to have a high need for influence and attention. The assessments state that Mr. Doe is likely to “enjoy the limelight”. He is generally confident and optimistic, and willing to take risks, although he may have self-doubt on occasion and may have blocked off awareness of some of his negative attributes. He had a difficult time in the interview articulating his weaknesses and describing any negative or developmental feedback that he had received during his career. On the positive side, he is self-critical, admits mistakes and is open to coaching and feedback about how he can improve his performance. However, when stressed or under pressure, he may become “threatened or alarmed” the assessments predict, adding that “he may require a lot of reassurance about his performance” which might become burdensome for his boss. Despite his insecurities, he is likely to be a resilient leader and to be energetically persistent in the face of obstacles and setbacks. At times, though, he may come across as “overly demanding or pushy” and the assessments caution, he may “intimidate colleagues.” In the interview, he mentioned that he used to motivate people “more with sticks than with carrots” but that he has learned that he “needs to go a little lighter on his team”. He is able to take unpopular stands and to give others candid feedback, even if it is negative, but may have some lack of awareness about others. His average level of emotional intelligence can make it difficult for him to understand others’ true motivations and concerns. The assessments caution that Mr. Doe “may be unaware of the effects or appropriateness” of his actions which means that cultural sensitivity is likely to be an area of weakness for him. He is likely to take an objective and unsentimental approach to work and to management, but is able to be empathic at times, and can be supportive to staff when necessary. At times he can be considerate to others, and responsive to requests for assistance. However, he may have a tendency to take credit too often which can be demotivating to others.
INTELLECTUAL EFFECTIVENESS
Mr. Doe tested in the 85th Percentile on the Wonderlic Personnel Test, which measures basic math and verbal skills and is like an SAT test. This score is acceptable, although at the
DATTNER CONSULTING, LLC Mr. John Doe Assessment May 12, 2006 Page 2 of 3 lower end of acceptability for someone applying for a position as senior as vice president. Mr. Doe scored in the 15th percentile on the Watson-Glaser Critical Thinking Test, which is like the LSAT or the Analytic section of the GRE. This very low score, especially relative to other candidates for positions at the level of the job for which Mr. Doe is being considered, raise serious concerns about his ability to be effective in the role. Part of the reason that he did not test well could be due to his personality, which the assessments describe as “impulsive” and “careless about rules and procedures”. Some of the mistakes that Mr. Doe made seem very basic, and like he wouldn’t have made them if he had paid more attention or taken the assessments more seriously. In the interview, Mr. Doe mentioned that he was concerned that he might not have done particularly well on the analytic assessment since “analytics are not one of my greatest strengths”. It would be helpful to ensure that reference checks and any additional interviews focus specifically on ascertaining whether Mr. Doe has the basic intelligence and problem solving ability to be successful in the vice president role. There is some indication that he would might do better in divergent (creative) thinking tasks than in convergent (analytic, problem solving) tasks.
INTERPERSONAL RELATIONSHIPS
On the positive side of interpersonal relationships, Mr. Doe enjoys interacting with other people and prefers working collaboratively. He can also be candid with other people, and they will likely know where they stand with him at any given moment. However, the assessments raise some serious concerns about his ability to effectively lead and collaborate. According to the tests, Mr. Doe can be “self-critical, tense and moody” and as a result, may at times actually “yell at his staff”, “become abrupt and even irritable” and be “blunt and undiplomatic.” He is not an agreeable person, and the assessments warn that he can be “brusque or even rude in dealings with others.” He also is likely to be “impatient with others’ mistakes” and “unconcerned about annoying others” and “likely to express hostile feelings directly with little hesitation”. Mr. Doe is likely to risk alienating colleagues and staff because he puts “his priorities ahead of others” and acts “hastily and impetuously” and “before others understand his reasons”. The adjectives used to describe Mr. Doe in the assessments are extremely negative, and include “stubborn, critical, selfish, callous, competitive and antagonistic”. While he is unlikely to be well-liked by others, he may be respected for his critical independence. The assessments also mention that he is manipulative and “willing at times to flatter or trick people into doing what he wants” and prone to getting overly involved in office politics. Given the above, Mr. Doe would likely face substantial challenges in building loyal, cohesive teams and in setting a positive example as a leader.
DATTNER CONSULTING, LLC Mr. John Doe Assessment May 12, 2006 Page 3 of 3
COMPETENCIES AND DEVELOPMENTAL NEEDS Probable Competencies • Making a good first impression that draws upon his extraversion and sociability • Boldness and willingness to take risks, take unpopular stands and confidently lead others • Creativity, receptivity to new ideas, and willingness to change course if necessary
Probable Developmental Needs • Difficulty in successfully solving complex problems that require analytic ability • Challenges in leveraging emotional intelligence to understand and motivate others • Coming across as rude, hostile, brusque, harsh, stubborn, manipulative and selfish • Managing his own work and those of others in a consistent and disciplined manner
SUMMARY AND RECOMMENDATIONS Although Mr. Doe has good experience and some of the personality attributes that could contribute to effective performance, his low cognitive ability test scores and problematic personality attributes raise serious concerns about his candidacy. Given a consistent and negative set of assessment results, the risks and costs of hiring Mr. Doe are likely to outweigh the potential benefits of hiring him, and therefore, he is not recommended for the position.
The above summary is based upon the results of the Wonderlic Personnel Test, the Watson Glaser Critical Thinking Appraisal, the NEO PI-R, the Hogan Personality Inventory, and the 16PF tests (none of which were proctored or timed), and an interview which was conducted in New York City on May 12, 2006. These tests of cognitive ability and personality are based on professionally conducted research and modern scale construction techniques. The results should be interpreted as probabilities since neither job performance nor human behavior can be predicted with one hundred percent accuracy. This report is intended to be used in conjunction with, not as a substitute for, other job relevant information gathered during the screening process. Ultimately, a final decision for any job candidate should be based on a combination of the results detailed above and other sources, such as candidate education, work experience, comprehensive reference checks, presentations or work samples, and face-to-face job interviews.
Report prepared by Ben Dattner, Ph.D.; 1-917-533-7987 or [email protected]
ffim*w#mm
I I By Roberl J. Grossman I
. .. Meas,unngHiring Managers
early halfthe store managers at the Wallis Cos.' 40convenience stores/gas stations in Missouf start-ed as entry-level workers who were hired by thepreceding store ma.nager. That's a healthy successrate for store managers who iilentified and
groomeil their successors.
BUSingSSgS Can Does wallis cos. have greathinng maragers? Or, is it
reap rewards j"str""r.vr
success rates.
by measuring Unfortunately, itis haxdto tell because Wallis, likemany companies, doesn'tevaluate the hidng perform-ance of its managers. Whowere those malagers who
hiring managers'
picked the applicants who went on to succeed? How did theymake their hiring decisions? Could the criteria they applied beadopted by other hiring ma.nagers throughout the company?
Ttrese are good questions that most HR professionalsdont ask. Because of that, experts say, they are missing outon a golden oppodunity to tap and duplicate a valuableresource in their organizations best practices of successfulh i r i n o m o n q o c r c
The managers who made the good hires would be relative-ly easy to identift: At Wallis-a major Midwest gasoline dis-tributor based in Cub4 Mo,-convenience store managers dotheir own hiring.
"Sometimes you fail to see opportunities that should beobvious," admits Rachel Andreasson, SPH\ vice president
June 2006 HR tqag.dna 9l
of organizational sewices. "This would appeax to be a meas-ure that would help us with our business strategy. If we caniilentify six managers who have developed good-quality em-ployees a-nd retain them longer than average, we may be ableto haxness their strengths and apply them throughout the or-sanization."
Expandthe foqrs?Until recently, HR's metrics in recruitment and talent selec-tion have been concentrated on efficiency-measuring cost-per-hire and time-to-fill data, Now, erperts maintain thatefrciency is only pa"rt of a winning formula; quality must bethe other part.
'When you make fan unsuccessfu] hirel, especially in man-agerial, sales or other jobs with customer contact, the down-side risks a"re high; you can lose millions in rwenues," saysScott Erker, senior vice president of selection solutions atDevelopment Dimensions Internationa.l, a global HR consult-ing firm in Pitisburgh.
"Assume you've hired loo new sales representatives andperiodically group their performance in three categoriesi
he o.plains. "Ten Aperformers quicklyaJld then consistentlyexceed their goals, 20B performers regu-larly reach theirs, and70 C performers axeclose but not quitethere. Ifjust 10 oftheC's had been an A orB, the impact on rev-enue would be sub-startiali'
Sure, you carl up-grade most employees to some degee wit}r training and coach-ing, but even the best txainers axe not miracle workers. "Ifyou
need a 7-footer to play center on your basketball team and yourecruit a promising player who is 6-feet-2, the finest hainingavailable wont get )ou where you want to be," says Jim DelRosaxio, vice president of talent acquisition at Veritude, asta.mng services company in Boston,
Fortunately-or unfortunately-most of the performer'sfate is sea,led at the hiring stage. Shrilies show that 65 percentof the time, the hiring process will be the cleciding factor indetermining whether you will end up with a gooil performer,Erker sals.
AGommltmentGapVirnrally all top executives say itt important to hire the bestpeople. But Del Rosario says they admit that tracking the out-comes ofhiring decisions is a lower priority than other aspectsof their business. When executive teams monitor the cost-efrectiveness of HR's hiring practices, they ilont ask HR formetrics or other formal feedback about hiring managers andtheir skills in talent selection. Arrd while company leaders holdline ma.nagers strictly accountable for achieving maxketinga.nd sales taxgets, they give managers a fiee pass on tleir ta.lentselection and development track record.
Pete Rarnstad, executive vice president of strategy andfinance at Personnel Decisions Intemational, a globd con-sulting firm in Minneapolis that specia.lizes in talent mar-agement, says managers' hiring choices should be subject tothe same level of scrutiny as their peformance in generatingrevenue or managing money. "You need to ileploy the sarnerigor you apply to accounting and marketing to hiring deci-sions," he says. "That means using outcome-based metricsthat really matter."
Without measuring managers' hiring performance,Ramstad says, "you don't have a reliable way to know whetherit's going well or not."
Moreover, he points ouq if things arent going well, you
94 HR ttlagazlne June 2006
The need to identifvmanagers w[o can pickwrnnrnq emploveesyvi | | i nJelqify as com petitionfor talent heats up.
'"r'on't knorv why-or horv to fui thc problem, rvhich could becauscd by hcatcd conpclition 1bl labor, a poor enploymentbrand, uncornpctiLivc salary and benelits, ol a poor hiringdecision.
''You hire somconc ."l,ho tulns otlt to be a poor' pedblmer.
Why did it happcn?" llanstad asks. "Did 1'ou have a rveak
applicant group to choose tiom? Or did 1'ou have a stroDligroup and mahc a poor sclccLion? IfJ,ou u,ant to tix a probleul,it's impofiant to be ablc to seg..menl rvl-ry it happened. Youcould nccd cithcr a selccLion or recruitlnelt inten'ention."
Ben Daltner ol Dattner Consulting in Nerv York siws: "Not
ibllowing- Lhe lrail back to the l-ririnpl nanager doesn't makescnsc. Itt like lunling a lnuLua] ltnd a[cl lrot being able tolcarn rvho picked the best stocks:'
Furlhcf, Lire leecl to idellit nanagels rvho can pick u'in-ning employees i'ili only interlsiI] a^s the labol fbrce g|ors at r|slorver antl slorver- r'ate and competition for talent heats up.Cdnpanies that i.lentily sLrch nanagers can pleLy to theirstrenllths, use them as lole models and builcl training pro-glans around then.
Implementation IssresWith any rnetric, the wayiiis defined and irnplemented will aflfect its usefulness in rneusuting business-related pertbrmance.Ancl *hile the oeed exists for hiring managel Dret cs, thereappalently is no idcal solution cuncntly in usc.
ets should do to measure the effectir''eness of hiling man-agers-and l.hat potential stunbling bkrcks they might
encountel oD the $'ay:
Start with retention rnetrics. Ai the r-ery leir"st, Drea-sr. 'ing
early iurnover is a useful indicator ofhiting success, sirl's JackPhillips, chairman of the ROI Institute ir Bimiogham, Ala.,lvhich offerc consulting and trahing iD measurernent. "liack
tbe percentage ofpeople hired by each rnanager u'ho lcave inthc 6rst 90 days," hc suggcsts. "lfyou makc an implopcl dccision, they usually leave early on. lfyou had 15 perccnt to 20pcrccnt oftbe people leaving in the first 90 days, that \l'ouldsay you havc an ineffcctivc managcll'
Pl.rillips, wlro is also the aulhol ofP?o-cing l he Vthtt oJ IIR:I'Iotr and Why to Measun RO1 (SIIRM, 2oo5), sa),s nlore-soplisticated rnetrics rnay turl1 out to be tine-collsullliigbusyrvork. "You make a loL of rvolk lbr youlselfifyou look fbr'more-conplicated ways to neasure."
Retentio[ metrics should be vour- fbuntlaticxr rreiLsures,agrees Jay Conger', resealch chail in leadership stuciies atClaremolt McI{en[a College i[ Claremont, Calif:, and authorof Buikling Ltadr:rs: Hoa Sutccsst'itl Cornptrrties Dcxr:lop tltr:Ncilt GeneratiorL (Jossey-Bass, 1999). But CoDger su:lgests thiltfurther cligging may be necessan to discover wh;Lt is causingtLD'no\tr'.
"You'll knorv 1'our avereLge tenule for people unde' ar urtrr'keting director is thlee ye.Lrs, but ih.rt some directols are ios-ing people earlier What's going on?" Conger asks. "ls itbecause ihe director is a craz-y micromanager'? Do people feel
they can't rnove up ihe ladder? Or do you have a managcr whohas a higher-ihan-avemge retention ratc but rvho historicallvhiles a lot of C plaJers rvben you're looking 1br molc As andB's?"
'lb gct to thc bottom ofthcsc qucslions, add a longiludinalrnalker or ficld that links thc hiring malagcr to Lhc irirc, su!,..gests Warrcn Cinnick, dircctor ol'leadership ald successiolplanning serl'iccs at PWC Saratoga, a Chicago bascd consultancy that providcs bcnchmarkin[i mctlics. CinDicli lcconmcnds doing,.. it for each manager by 1'early cohorl. For e-tam-ple, look at everyone hired in 2ooo ancl see rvhere Lhey iue il12006. Foul netrics should do it:
lune 2006 HR Magazine 95
. What percentage left voluntarily or involuntarily?
. How mrmy have consistently scored in the iop bands ofyour perfbnnance evaluation systeln?. What is the evidence that people have beei promoted upthe ladder over iime?. How many people hired at the entry level have moved intoleadership positions?
Cinnick says this kind of assessment zeros in on managerswho foster a mentodng culture. "In a norrnal talent review,these managers may not come up as 'high potentials,'but theymay be good coaches that carr be insuumental in helping oth-ers move aIead."
Focus on keyjobs a:rd desired outcornes. John Boudreau,
professor a-nd research director at the Center for EffectiveOrgarizations at the University of Southern California, LosAlgeles, says it's not necessary or cost-effective to track everyhiring decision. He suggests the following strates/:. Identifu pfuotal jobs, Divide your iobs into three cate-golies: where high performance is pivota.l, where it has mod-est impact and where it matters less. Focus on thejobs wherethe manageN' decisions will have the biggest inpact on yourbottom line. Generally, these jobs will be high-level manage-rial slots. However, there are other positions, often withcustomer-service implications, where the manager's judg-ment may bejust as critical.. Segment jfor aariability. Sepaxate out positions in which,even after upfront HR vetting, managerial discretion is es-sential to the selection process. These ale thejobs for whichmanagers' decisions will be critical and thai should betracked.. Detennine @hat to mea.sure. Limit metrics to the role themaragerial position plays in your business strategy. For exam-ple, you may want to measure how effective a maJrager is at se-lecting people who can hit the ground running. Or you maywant to measure who excels at developing ta-lent over time, orwho selects the most A or B performe$.. Measure ooer time. Track in rolling time frames of thleeto live years to deiermine howthe managers' selections farethroughout their tenure. Compare results among man-agers.
Use self-a,ssessments. Allow managers to self-assess theirperformance using data such as sales goals, employee promo-tion and performance reviews.
Ramstad says asking managers to self-assess emphasizes tothem the value you place on talent selection. He recommendsnanagers complete a self-assessment immediately after a can-didate has been hired, grading the qualiqi of the talent pool,how good they were at picking people oui of the applicantpool, and how good they were at larding ihe talent. Givingthen a task they must do at the time ofiheir decision encour-ages them to focus on the long-term aspects of their choices.(See'A Self-Assessment Tool" on page 94.)
And why not also assess HR's qua.litaiive acuity at pickingwinners? "Have HR recruiters predict success on a five-pointscale and then see who is more accurate over time*HR or thehiring manager," Dattner says. Altematively, Dattner advisesthat hiring managels and HR managers put this practice intoplace informally to give themselves feedback, even ifthey don'tshare it.
Gorurtering Manager ReluctanceHR executives may be wary of selling outcome met cs to theirhiring managers who are not enthusiastic about the prospectof having their taleni-selection acumen become a subject for
96 Hf, Magazim June 2006
their annual lcl'ieu's. Thc answer; Phillips says, is to assruenanagers that the data will nol bc uscd for this purpose."These kinds of metlics should be used for coaching ol coun-seling, not to remove or penalize the manager', unless the pirt-
tern continres o\'er time."Others say if your organization values talent selection,
eventrrally you'll have Lo incorporatc it in pcrformance evalua-tions. Ho\\.? Ease them in alld build acceptance lbr thc met-lics. "Use then lbr two years fbr development, where you don'tmake a fonnal a^ssessneitl' Conlier says. "Then Lmnsilion to
zrn accountabilif mocle. You'll be ir-sking then to speld rrrore
tirne on coerching and developnent."
Notes oI GautionA concern cited by sone epefts is $'hat might be called a
moral hazard: If a nanager'.s or,n perlbrnance levieu, in-
cludcs a measurc ofsuccess ofhis "progeny," he'.s more likell togive them good lcricrvs. Thc halo elTect mil] canl o\''er to pro-
notion recommcndations as well.To counter such "gaminp;l meLrics should bc built with
checks and l-ralances. FIow? Conller recommends including nteam-per'fbrnrance met c, reqrlililrEi lnanag..eIs to dcmonstlzrte
that subordinates' outstiuding pedbrmancc was coDslstent1\,ith their. tealn.s acconplishlnellts.
In addition, a ligorous revierv by IIR of peribnlancc
appraisals lookingfol concrete erianples to suppoft marragers'
ratings should take out sone ofthe subjectivity ofthe process.
Limit metrics to the role theman?ge[ial position plays iny0u r Dusrness strategy.
Of coulse, sorne experts queslion whcther it's pmctical or'necessary to track hi ig-outcone meldcs. Sarah George, sen-ior vicc president of recmitirrg at Wachovia Colp. in Charlotte,N.C., says 4t percent ofihe positions filled by tl.re banking andlinancial scrvices provider liut yeal were lilled rith intcmalcandidatcs. She says her managels make thousands of hiringdecisions annually, many at entry-level supervisor
"We know how our rnanagers do lrom experiencei' Gcorgesays. "Maty of us can tell you, 'Yes, l'e know malagers X\Z;they hile great people.' Individual per-formance probablvwouldn't be lvolth tracking. By the time you got your ansu.er',somethingwould have changcd and the dataq,ouldn'tbe accu-rate. Colpolate legend is probablyiust as effective."
George rneLy be right, but ConSer isnt sure. "'l'hey may dis-cover their beliefs are mlthology," he cautions, recommendingtracking data instead.
The Gosts of Action vs, InactionEvel u'ith the benefits of identifuing successflil hiring lnan-iLgers and sharing best practices, there are obstacles to theprocess. Some erecutivcs cite lack of technicarl ciLpal-riliry, say-ing they don't have sullicicnt automated systems to track thedata. Othels say they know it! a good idear but other p oritiescome first. Still others claim it's too costly for the retum it \\.illdeliver.
RamstiLd disagrees: "It'.s nol about noney, but cultulalchange. ln fact, its really too costly llot Lo do it]'
Thcrc's also a residue ofdoubt among erec[Livcs about theintegriq/ of HR mehics, Del Rosario says, ol1 tl]e basis of\\'hathc sals he is told by those with u.hom ire rvorks and u'hom hehas as clicnts. "Why add more reports?' he says. 'A lot of I{Rrepods go unread nolv because executives don'L see rv\ thcyneed thc data."
Finally, top management may be rvara about con{ioltingpeople rvho are not good at selection. "What nucks it up isliceless fear o1'conllict," Conger says. "When cliticized, thernaniLger looks to piss the blamc along to HR for not doing zrIigolousjob on the flont end. Fingcrs start pointing; emotioisflare. Still, alter you see what eflcctiveness met cs c.rn do toheJp your business, obviously it's rvorth the troublel E!
lloBurl J. GxossMAN, ,\ coNTRTBUTTNG EDrroR or ,{11 .\{acAZrNn, rsA LA\!ryIiR ANI) A PI{OITFJSSOR OF Xfr\N,{.GEMtrNT STUDIES AT NIL{RISI'CoLLrctr rN Poucrrir.FrPstD, N.Y
