EEI’s SO/PD II Employment Test Battery

Midwest Energy AssociationElectric Operations Technical & Leadership Summit

May 13 – 15, 2015

Background

EEI’s System Operator/Power Dispatching employment test battery has been in use since the 1980’s

Over time, a number of changes have occurred in how the jobs are performed and staffed:

New technologies

Increased job demands competing for employees’ attention

Impact of increased regulatory compliance requirements (FERC, SERC, NERC, etc.)

Changing applicant pools and increased hiring projections

In 2010, plans were initiated to develop a computer-based test to address the above changes

Participating Companies

Benefits

Capability to measure job candidates’ multitasking ability and the impact of stress on job performance;

Inclusion of recent methodologies that are likely to have less adverse impact against protected classes of job applicants;

Computer-based test battery that better reflects the job environment than a paper-based test battery and facilitates test administration and scoring; and

Inclusion of Gas Controller jobs for combination electric and gas companies.

Intended Jobs

The new test battery will be used for hiring in three SO/PD functional areas

Distribution – monitoring and troubleshooting the distribution system including the dispatching of field crews

Transmission – monitoring, regulating, and troubleshooting the transmission system including substations

Generation/Balancing Authority – monitoring and regulating the energy supply to meet load and interconnect sales commitments

It will also be validated for use in hiring Natural Gas Controllers

SO/PD II Project Phases

Three main phases:

Job Analysis – included a review of job documents, development of a task list, job observation, and survey to identify the Knowledge, Skills, and Abilities (KSAs) critical to the job

Test Development – based on the information collected in the job analysis phase, three new tests were developed and two were chosen from EEI’s existing library

Validation – evaluate the relationship between scores on the test and performance on the job

Phase 1: Job Analysis

Detailed analysis of three SO/PD functional areas as well as Natural Gas Controller

Reviewed hundreds of job descriptions

Conducted job observations at multiple sites

Conducted interviews with job incumbents

Developed a comprehensive task list with input from over 30 SO/PD & NGC supervisors, managers, trainers, and incumbents

Administered industry-wide Job Analysis Questionnaire to over 1,400 incumbents and supervisors to determine which tasks are critical to success and what characteristics are required to perform them

Phase 2: Test Development

EEI and contractor evaluated what types of selection tests would be most effective at measuring the identified KSAOs

Two new tests were developed and two existing EEI tests were selected for inclusion

Analytic Thinking Skills Test (ATS)

The test contains three sections: Argument, Problem Solving, and Logic-based Reasoning.

Argument – consists of passages that present an argument and some evidence supporting the argument followed by a question that asks the test taker to draw a conclusion based on the evidence.

Problem-Solving – consists of passages followed by a set of facts that the test taker must use to determine the answer.

Logic-based Reasoning – consists of passages that present some facts about a specific topic followed by several statements that represent inferences one might draw. The test taker must decide if the statement is True, False, or Indeterminable.

Analytic Thinking Skills Test (ATS)

Example Argument Question

B is the correct answer

Our IT department has noticed an increase in the number of computer viruses. Please be careful about opening e-mails from anyone you do not know.

The argument above depends on which of the following assumptions?

A. Computer viruses being transmitted through e-mails are particularly damaging.

B. E-mails sent from unknown parties are more likely to contain computer viruses.

C. Employees are more likely to receive e-mails from unknown parties than from people they know.

D. The company’s virus protection software is not effective against viruses transmitted through e-mails.

E. Computer viruses are only transmitted through e-mails.

Analytic Thinking Skills Test (ATS)

Example Problem-Solving Question

B is the correct Answer

A project must be completed by performing tasks in order. There are five tasks: Task1, Task2, Task3, Task4, Task5.

• Task1 and Task4 must be completed consecutively, but only after completing Task5.• Task3 cannot be completed before Task1.• Task4 depends on the completion of Task2.

Based on the above information, which of the following is a correct ordering of the project?

A. Task2, Task5, Task1, Task3, Task4B. Task5, Task2, Task4, Task1, Task3C. Task1, Task2, Task3, Task4, Task5D. Task2, Task1, Task4, Task5, Task3E. Task3, Task5, Task1, Task4, Task2

Analytic Thinking Skills Test (ATS)

Example Logic-Based Reasoning Question

“Indeterminable” is the correct answer

Explosives are substances or devices capable of producing a volume of rapidly expanding gases that exert a sudden pressure on their surroundings. Chemical explosives are the most commonly used, although there are mechanical and nuclear explosives. All mechanical explosives are devices in which a physical reaction is produced, such as that caused by overloading a container with compressed air. While nuclear explosives are by far the most powerful, all nuclear explosives have been restricted to military weapons.

From the information above, it can be validly concluded that:

All mechanical explosives have been restricted to military weapons. (T/F/I)

Multitasking Simulation (MTS)

Test takers must complete four separate tasks, each presented in a separate quadrant of the screen simultaneously

Memorization task – candidates must memorize a series of short alphanumeric codes and then recognize them when prompted.

Basic math skills task – candidates solve addition problems.

Visual monitoring task – candidates must monitor a needle on a gauge as it moves from the center to the left or right and stop it before it reaches the bottom of the gauge.

Listening task – candidates periodically hear low and high tones and must click a button to identify the type of tone.

Multitasking Simulation (MTS)

Existing EEI Tests

The results of the job analysis were reviewed to determine any additional target areas for assessment

Basic math and verbal ability were identified as needed at entry

Existing tests in EEI’s library were reviewed and Math Usage and Reading Comprehension were identified for inclusion in the test battery

Phase 3: Validation Study

About 800 incumbents took the new test battery and their job performance was rated by their supervisors

Data were analyzed to evaluate the effectiveness of the test battery in predicting job performance

The relationship between scores on the new test battery and job performance were found to be statistically significant indicating that the SO/PD II test battery is an effective predictor of job success.

Implementation

The SO/PD II test battery was rolled out on April 30th, 2015 for companies to begin using in their hiring process

Practice tests are available to companies by subscription through EEI

For more information about the test or to discuss using it in your company, contact Amanda Allen at EEI (aallen@eei.orgor 202-508-5153).