Working Smarter: Job Performance Metrics For the Smart Grid

SANS SCADA Conference Presentation

• Transforming the electric power infrastructure

• Lack of a viable cybersecurity workforce

• Our Nation’s epic journey

• Grid modernization efforts require:

- Advanced, continually maturing cybersecurity capabilities

North American Power Grid: Vast and Ever Growing Cyber-Attack Surface

Smart Grid Cybersecurity Panel

• NBISE – National Board of Security Examiners • The National Board • Job Performance Panels

- Operational Security Testing (OST)

- Advanced Threat Response (ATR)

- Smart Grid Cybersecurity (SGC)

We identify and measure the proficiency, the performance,

and the potential of the cyber security workforce.

Purpose: The project contributes to the Department of

Energy’s efforts to develop a competency model and explore

assessment methods focused on the job responsibilities and

unique skill set of Smart Grid cybersecurity specialists.

Who: Those primarily responsible for operational security

functions for day-to-day operations, but not engineering and

architecture, in smart grid environments.

Outcome: A measurement model for assessing

knowledge, skills, and abilities in the areas of technical and

operational skills.

Smart Grid Cybersecurity Panel Mission

• Managing Partner – UtiliSec

• Specializes in Smart Grid security architecture design and penetration testing.

• Led the Smart Grid Security Architecture group in the creation of NIST Interagency Report 7628

• Currently plays key roles in the Advanced Security Acceleration Project for the Smart Grid (ASAP-SG), National Electric Sector Cybersecurity Organization Resources (NESCOR), and Smart Grid Interoperability Panel (SGIP).

• Taught courses in hacking techniques, forensics, networking, and intrusion detection for multiple universities, corporations, and security conferences. Currently an instructor for the SANS Institute. In addition to electric power industry conferences,

• Frequently presents at top security conferences such as Black Hat, DEFCON, OWASP, and AusCERT.

• Co-leads prominent open source projects including the Samurai Web Testing Framework, Middler, Yokoso!, and Laudanum.

• MBA in International Technology

• CISSP and SANS GIAC certified Incident Handler (GCIH), Intrusion Analyst (GCIA), and Web Application Penetration Tester (GWAPT).

Justin Searle, SGC Panel Chair

Scott King, SGC Panel Vice Chair

• Manager, Security Operations – Sempra Energy

• Since 2001 information security field supporting the Department of Defense and shore based Navy and Marine Corps.

• Multiple roles within the security community supporting federal government, DoD, state government, commercial companies, and most recently critical infrastructure.

• For the past three years, Mr. King has worked for the Sempra Energy Utilities family of companies as the Security Operations Manager.

• He and his team have enterprise operational security responsibility for the corporate IT infrastructure as well as the gas and electric controls networks.

• Certifications

(CISSP, HISP, IAM, IEM)

Subject Matter Experts and Advisory Group

Panel Member Representation

Service

Government

Industry

Research

Vendor

SGC Panel Productivity

Master Vignettes

82

109

165

13

Job Responsibilities

26 Job Roles

Operational or Incident

Response Scenarios

Process Steps

Job Tasks 546

Target Cybersecurity Job Roles

Three job roles became the focus, based on Workforce Development and Job Performance Model process:

Security Operations Specialist

Intrusion Analyst

Incident Response Specialist

SMART GRID CYBERSECURITY

JOB PERFORMANCE MODEL

JOB ANALYSIS QUESTIONNAIRE

How can you get involved?

www.sgcjaq.nbisesites.org

Example JAQ Survey Questions

Job Analysis Questionnaire - Logistics

Participation will be limited to 15 minutes Survey participants will have the option to take an additional 15-minute portion of the survey.

The survey can be taken with any browser or web-capable device.

Target date for release: February 1, 2012

Thank You and Questions?

www.sgcjaq.nbisesites.org

To participate in the Smart Grid Cybersecurity

Job Analysis Questionnaire, please visit:

Backup

What is a Vignette?

A collection of: • a critical incident title or description • when the incident occurs (frequency and/or action

sequence) • what happens during the incident (problem or situation) • who is involved (entities or roles) • where the incident might happen, now or in the future

(systems or setting)

Further definition of a vignette might include: • why it is important (severity or priority of response) • how the critical incident is addressed (method or tools that

might be used)

Elicitation Tools and Methods

Incident Response Vignette Identification & Elaboration

Elicitation Tools and Methods

Vignette Master Categories

• Data Leakage/THEFT [9075]

• Network Attacks [9076]

• Substation/SCADA Attacks [9077]

• AMI Attacks [9078]

• Client Side Attacks [9079]

• Phishing Incidents [9080]

• Network Separation and attack paths [9081]

• Incident Response Process & Log Management [9082]

• Encryption Attacks [9083]

• Security Testing [9084]

• Threat & Vulnerability Management [9085]

• Access Control Maintenance [9086]

• Risk management, compliance and audit [9087]

Elicitation Tools and Methods

Three key roles based on SGC Panel elicitation exercises

The SGC panel selected three job roles based on dozens of candidate roles:

Elicitation Tools and Methods

SGC Job Responsibilities & Required Job Tasks

Elicitation Tools and Methods

(Example responsibility with its tasks)

Responsibilities & Task Mapping

Elicitation Tools and Methods

Task Identification & Voting

Elicitation Tools and Methods

In Summary

• Support the development of this Job Classification Report and job roles

• Ready for broader industry contribution

• Establish the context for an accelerated training

program

• Advance a measurement model

• Validate the performance model

• Create foundation for assessing performance during

and after training

Results, Innovations, Future Plans

• Context elicitation through vignettes and responsibilities

• Competencies defined at novice, apprentice, expert and

master levels for multiple roles (organizational language)

• Creating profiles for technical and operational skills

• Reusable libraries to assess multiple roles

• Validation and extension of job performance model Standards based on:

- Validated curricula

- Assessment

- Simulation libraries

How is the NBISE Approaching Competency Different?

SGC Panel - The difference

Put KSA diag here

• The Job Analysis Questionnaire (JAQ), is broadly disseminated within the Electric Sector community.

• The JAQ seeks to gauge both the relative importance of given tasks and

the frequency with which they are conducted.

• In parallel with survey data analysis, NBISE staff and researchers work

with experts to augment the growing competency model report with critical incident analysis.

• Intensive, in-depth interviews are geared toward documenting

characteristic and challenging events that embody the most important aspects of the job.

• Further work is done with these “critical incidents” and “situational

judgment scenarios” to determine how the actions taken by a novice, apprentice, journeyman, and expert-level practitioner are differentiated.

The SGC Job Analysis Questionnaire encourages broad workforce input

SGC JAQ Survey Initiative

Competency Modeling Process

SGC Panel - Phase 1 Process