Front-End Analysis Methods for Noncommissioned Officer Education

System (NCOES)

Final Program Brief

Gus Finerson Dynamics Research Corporation

Jennifer K. PhillipsCognitive Performance Group

Marisa L. MillerU.S. Army Research Institute

27 Jun 2012

2

• Requirement

• Program Overview

• Process

• Products/Outcomes

• Questions

Agenda

3

Statement of the Need

• Training approaches are differentially suited to developing certain types of skills.

– The Army needs to rigorously and systematically optimize its use of institutional, unit, and self-study training by improving its training site selection process.

• Newly identified operational training requirements have been addressed in institutional training.– There are costs associated with continuously modifying

institutional POIs to account for rapid changes in the COE. – There is a need to distinguish between enduring tasks and

non-enduring tasks, as a potential discriminator between ideal training sites.

4

Program Description

• Front-End Analysis Methods for NCOES project conducted by the Army Research Institute (ARI) in support of the Institute for NCO Professional Development (INCOPD)

– Contractor: Dynamics Research Corporation

– Subcontractor: Cognitive Performance Group

• Period of Performance: August 2011 – July 2012

• Primary Objective: Design an analysis tool that yields optimal

alignment of tasks to institutional training, operational unit training

or self-development/dL

Program Overview

5

Des

ign Identify best

practices and critical site selection factors for placing training

Dev

elop

Describe front-end analysis (FEA) method and instantiate in Site Selection Tool for CTSSBs

Ass

ess Incorporate

user feedback to increase effectiveness of Site Selection Tool

Goal: Improve effectiveness and efficiency of CTSSB process to recommend placement of training.

Phase 1: Design

Objectives• Determine what makes a task appropriate for institutional instruction,

unit training, or self-study• Identify factors for site selection considered by “experts”• Define enduring vs non-enduring tasks, and identify their distinguishing

characteristics

Activities• Literature Review• Data Collection

– Observed CTSSBs• Ft. Benning (11A) and Ft. Bliss (WLC)• Interviewed CTSSB members (4)

– Interviewed SMEs (31)• NCOA Commandants, course managers, instructors, training

developers, etc.• Ft. Benning, Ft. Bliss, Ft. Huachuca, U.S. Coast Guard

6

Phase 2: Develop

• Objectives – Provide a structured and systematic process for considering the

most critical factors for site selection, as identified by SMEs– Assist CTTSB members who have strong operational and

garrison experience to bring to bear, but are novices at site selection

– Maintain the role of human judgment and group discussion, but increase the efficiency and rigor of the process

• Activities– Iteratively developed a logic chain for applying discriminating

factors to site selection recommendation– Identified perceived user requirements for integrating the tool into

CTSSB process– Embedded logic chain in Excel-based Site Selection Tool (SST)

7

Phase 3: Assess

• Objectives – Ensure the logic chain is sound– Maximize usability of the tool within the CTSSB process to ensure

value added to the users• Activities

– Feedback via telephonic demonstrations with 5 SMEs– User Testing at Ft Benning 19A CTSSB– Testing of paper-based SST at Ft Belvoir 35G CTSSB– User Testing at Ft Huachuca 35T CTSSB– Iterations of SST at each juncture

8

Phase 1 Findings

• Concept of enduring vs. non-enduring task is theoretically sound...– Often equated with Doctrine vs. TTP, or general vs. theater-specific– May also relate to systems and equipment that change

9

• … but in practice, task endurability is not considered as a factor for site selection– Some MOSs making efforts to remove specific systems or equipment

from task descriptions in favor of function/outcome– Institution allows for broad and rapid delivery of key training, and

therefore may be appropriate even if the task is non-enduring– Existing tasks can become relevant to new MOSs– Leadership mandates can render the point moot

• Concept of emerging tasks – those that become relevant as adversaries, technologies, and the global environment evolve– Sensitive site exploitation and forensic analysis– Operating against and within networks, both adversary and civilian

• Key leader engagements / tactical questioning • CoIST tasks

Site Selection Factors(SME n=32)

10

• Nature of the Knowledge– Hands-on/Motor skills (17)– Basic facts and procedures (15)– Task complexity (7)– Conceptual and critical thinking (11)– Detail (3)– Segmentability (5)– Difficulty to retain (8)

• Enduring Army Standards– Universality (11)– Core task/Foundational (12)– Need for standardization (9)– Need for control (7)– Leadership (7)– Doctrinal (18)– Task Criticality (8)

• Instructional Affordances– Peer learning benefit (7)– Question & answer (5)– Practical application (17)– Residence of expertise (7)– Need for observation-based assessment (6)

• Task Performance– Safety/Risk (6)– Doctrinal Basis (13)– Frequency of performance (4)– Current force proficiency (3)– Unit-specificity/MOS-specificity (14)

• Site Affordances– Need for equipment (12)– Time available to train (8)– Time required to train (5)– Annual training or qualification (3)– Integration ability (4)– Training updates (5)

Feasible via Self Study?

High = Institution

Institution or Unit?

Resource availability?

Discriminating Factors

11

Goal: Identify a manageable set of factors that will most often produce a sound site selection recommendation.

Factor Short Question Long Question

Safety; Risk Safety concern?DURING TRAINING, what is the risk of injury to personnel or damage to equipment when training this task?

Need for Equipment; Hands-on Motor Skills

Equipment and hands-on training required?

Is actual equipment – a weapon, vehicle, or system – and hands-on training, instead of an equipment substitute, required to train this task?

Task Complexity Difficult to learn? How complex or difficult to learn is this task?

Need for Observation-Based Assessment

Assess performance by watching face-to-face?

Does a trainer need to watch performance face-to-face to assess it?

Training Updates Updates needed more than yearly?

Will the training change more than once a year?

Residence of Expertise SMEs available or accessible at unit?

Are SMEs readily available or accessible to train this task at the unit?

Annual training or qualification

Annual qual./cert. at unit? Is recurring qualification or certification conducted at unit?

Universality; Need for Standardization

Universal task, uniform training required?

Is the task universal and requiring uniformity of training across the MOS?

Need for Control Reliable and controlled assessment?

Is a reliable and controlled assessment highly important?

Peer Learning Benefit Peer-to-peer learning critical and required?

Is sharing experiences with peers from other units critical and required to maximize learning?

Leadership Criticality to leadership? How critical is the task to the leadership role?

Access to Equipment Safety/equipment available at unit?

Do most units have the safety measures and equipment readily available to perform training?

12

Logic Tree

Guiding Principles for Development

• Ease of use for administrators and users• Integrates conceptually with critical task identification process• Enables CTSSB members to indicate disagreement or special

circumstances• Compiles data across CTSSB members• Flags tasks for discussion when consensus is low• Enables real-time additions or modifications to the task list• Can be flexibly applied across range of CTSSB circumstances

13

Site Selection ToolOverview

• User Group 1: Administrators– E.g., Task analysis branch – He/she initiates the process and receives the final results

• User Group 2: CTSSB Members– Soldiers from across the force/MOS

• Excel spreadsheet with incorporated responsive logic

• For each critical task, logic tree of 3-12 questions about nature of the

task and its training

• As function of logic branching, SST recommends training site:

– Institutional, Unit, Self-development/dL

14

How SST Supports CTSSB

15

Critical Task Selection Process

Critical Task Selection Process

Site Selection via

SST

Site Selection via

SST

CTLCTL

CTL Site Recommendations

CTL Site Recommendations

Introductory Briefing

Introductory Briefing

CTSSB

CTL Site Recs Compiled via SST

and Discussed

CTL Site Recs Compiled via SST

and Discussed

Master CTL Site Recommendations

Master CTL Site Recommendations

Post-CTSSB

Consider Institution Time and Resource ConstraintsConsider Institution Time

and Resource ConstraintsJustified

RecommendationsJustified

Recommendations

Board Admin Activities

Board Admin Activities

CTSSB Member Activities

CTSSB Member Activities

ProductsProducts

Key

Sample User SST

16

Task ListShort Version of

Questions

Red Caret for Long Version of

Questions

SST Recommendation

User Opinion, If Different from SST

Task Number Greyed Out Cells Reset ButtonsQuestion Definitions

Demonstration

17

User SST

Sample Administrator SST

18

Set % Consensus Button

Votes and Consensus on User Opinion

 

All User Comments

Task ListVotes and Consensus on SST Recommendation

Tasks for Group Discussion Highlighted Amber

Individual User ResponsesTask Number

Outcomes of User Assessment

• Three rounds of SST functionality modifications based on user recommendations

• Board members considered a broader range of site selection factors when using the SST– SST resulted in change to site recommendation in 3 cases out of 11

possible 19A LT tasks• Differences in board member opinions were apparent as a result of the

SST, in contrast to the current board process• Board members appreciated the thought priming that results from the

SST• Consensus thresholds must vary, since CTSSB circumstances vary• CTSSB administrators had more favorable reactions to the SST than

the board members; the research team relied more heavily on their comments and suggestions for SST modifications than board members

19

Final SST Functionality

• Issue: Some tasks are outliers, and human judgment trumps SST recommendation– Board member inputs personal site recommendation when he/she

disagrees with SST• Issue: Institutional course, but not the course under review

– “Institution-Functional” is among the board member opinion choices• Issue: Different CTSSBs require different agreement thresholds

– Administrator sets consensus threshold at 50%, 60%, 70%, or 80%. Single highlighting threshold active (amber)

• Issue: Board member may be have limited exposure to task and wish to abstain from a vote– Consensus calculated based on number of respondents vice total board

members; therefore, board member can leave row blank when unfamiliar with the task

20

Potential Future Modifications

• Ability for Administrator to customize the “Institution” recommendation as the specific course under review– E.g., SST Recommendation produced would be “Armor ALC” instead of

“Institution”• SST Recommendations populate only after board member has

answered all questions for all tasks, and “Submits” responses – Pro: Defeats against gaming the system– Con: Adds time requirement; user must re-view each task to select

personal opinion• Ability for Board Members to answer all 12 questions for every task

– Pro: Sense that important questions are always asked and answered– Con: Adds time requirement

• Web-based set of CTSSB resources, including task criticality ratings and links to full task descriptions and other relevant resources– Caution: Retain face-to-face group discussion

21

Products

22

Summary

• SST can increase the effectiveness of site recommendations– Board members assisted in applying more rigor and thought to their

knowledge of the task and subsequent recommendations– Enables board members to have equal votes in the site selection portion

of the CTSSB• SST can increase the efficiency of the site selection portion of the

CTSSB– High consensus tasks do not require discussion– Group discussion time can be focused on those tasks on which board

members disagreed– Encourages less assertive members to voice divergent views; prevents

board from inappropriately concluding consensus• SST is available upon request

– In the future, will be accessible via Army Training Network

23

24

• Marisa L. Miller, PhD (ARI) Marisa.Miller@us.army.mil

Marisa.L.Miller6.civ@mail.mil (706) 545-2450

• Gary Rauchfuss, PhD (INCOPD)

Gary.Rauchfuss@us.army.mil(757) 501-5439

• Phillip Finerson, CSM (R) (DRC)

Pfinerson@drc.com

(270) 268-9900

• Jennifer Phillips (CPG)

jenni@cognitiveperformancegroup.com(440) 937-8434

Points of Contact