Interim Findings and Recommendation from the Whole House Process Evaluation – SCE Contractor Training Assessment

May 1, 2012

Presented to: Staff from SCE and PG&E

Prepared by: ASW EngineeringSBW Consulting, Inc.

Part Two of Today’s Presentations

SCE Contractor Training Assessment In-depth examination of SCE’s training

designs, methods and materials. Do they prepare contractor’s to succeed?

Context Rapid change in training underway, so

assessment looked at a moving target. Focused on SCE but there are wider

implications. Recommendations from the overall process

evaluation need to be considered before training is changed.

2May 1, 2012

Training Big Picture

Contractors must attend training to qualify Performance gaps – a training problem?

3May 1, 2012

Assessment Objectives

Effectiveness and appropriateness Do the training objectives address EUC

tasks?

4May 1, 2012

Assessment Methods

Reviewed program materials (program design, training and reporting)

Interviewed key personnel (RHA, ICF) Mapped training objectives to program

requirements Assessed discrepancies

5May 1, 2012

Assessment Criteria

How well the training and materials: Supported behavior change Included motivation to action Applied adult learning principles

6May 1, 2012

Adult Learning Principles and Practices

Learner orientation, buy in, engagement Learner success engineering Practice, application, and interactivity Learning objectives

7May 1, 2012

Effect of Teaching Styles and Average Retention Rates

8May 1, 2012

Delivery Mode and Average Retention Rates

9May 1, 2012

Practice and application are crucial to training effectiveness

Relating Learning Levels to Performance Outcomes

10May 1, 2012 Bloom’s taxonomy

11

Required Training

Participation Workshop Basic Path Training Advanced Path Training Online Learning Center Mentoring

May 1, 2012 11

Participation Workshop

2-hour session required for at least one person from each contractor – webinar or classroom

Orients contractors to goals, scope of work, marketing

Describes contractor qualifications, benefits

12May 1, 2012

Participation Workshop Findings

Good pitch of program Lacks details on:

Job processing expectations (Job Reporting, EnergyPro, documentation)

Level of rigor for QA/QC

13May 1, 2012

Participation Workshop Recommendations

More content on: EnergyPro requirements Exposure to job reporting

documentation and process Job reporting may change based on other

recommendations Refer to:

“Adopt Common Statewide Job Reporting” (Whole House Process Evaluation – PG&E_SCE Interim Report Workbook, Caption Tab, ID=13)

14May 1, 2012

Basic Path Training

Mandatory for Basic Path program Exception if BPI Building Analyst on staff

15May 1, 2012

Basic Path Training Findings

California Building Performance Contractor’s Association (CBPCA) 3-day course a good match-up between training content and program measures

Lacks: Formal objectives Adult learning principles

16May 1, 2012

Basic Path Recommendations

Incorporate: Worksheets More hands-on practice Ways to demonstrate proficiency (e.g.,

asbestos identification and insulation measurement)

Formal learning objectives Adult learning principles

17May 1, 2012

Advanced Path Training

EUC requires BPI Building Analyst certification

Certification can be obtained with no formal training

Certification does not guarantee EUC skill set

Dozens of third-party trainings offered

18May 1, 2012

Third Party BPI Training

Quality and depth of training ranges widely

0 to 12 days Classroom, online, self-trained Insufficient practice for EUC

fieldwork (performance gaps) Loose alignment with some

measures (air sealing, weatherization)

19May 1, 2012

Advanced Path Training Findings

CBPCA 12-day course provides: Solid foundation for Advanced Path work Strong combustion safety curriculum Some hands-on practice with diagnostic tools

Lacks: Formal training objectives Adult learning principles Descriptive text for photos

20May 1, 2012

Advanced Path Recommendations

For CBPCA 12-day course: Add training objectives Apply adult learning principles Add descriptive text to photos

Supplement BPI training with: More hands-on practice with field testing procedures,

protocols Practice of program-specific desk work (forms, energy

modeling) Remedial training for specific performance gaps

(asbestos, air sealing, etc.)

21May 1, 2012

Concerning HVAC Quality Installation (QI Program) and EUC

For minor HVAC repair work, BPI training is adequate

For major HVAC repair or replacement, then a permit is required and contractor must meet Title 24 quality standards

HVAC QI training addresses Title 24 code

22May 1, 2012

Online Learning Center – A New Resource

On-demand training on: JRT Administration Energy Modeling Job Processing Combustion Safety Testing

Mandatory participation; must pass exam

23May 1, 2012

Online Learning Center Findings

Well designed training Addresses some performance gap issues Incorporates adult learning features Useful for BPI certified contractors who did

not attend CBPCA 12-day course

24May 1, 2012

Online Learning Center Recommendations

Record exam scores for multiple employees

Expand topics: BPI and program standards Field inspection / testing Asbestos identification Insulation and weatherization

25May 1, 2012

26

Mandatory Mentoring

Required of one person from each contractor for both program levels

Three topics: Field Work Job Reporting/Job Processing Energy Pro

May 1, 2012

Mentoring Findings

Not formal training per se Mentoring includes:

Observation of contractor performance Evaluation and scoring Feedback

27May 1, 2012

Mentoring Recommendations

Standardize mentoring and evaluation protocols

QC firm should provide input Provide remedial training adapted to

individual needs, based on QA/QC findings Remote mentoring for Energy Pro and Job

Reporting

28May 1, 2012

29

Top 10 Performance Gaps

Performance Gaps Inaccurate insulation R-value Inability to identify asbestos Failure to achieve minimum air-sealing Inaccurate wall, window, and floor measurements Improper combustion safety testing (CO readings) Failure to identify combustion appliance hazardous

conditions (e.g., gas leaks) Failure to complete Basic Path measures for

Advanced projects

May 1, 2012

Top 10 Performance Gaps

Performance Gaps Inaccurate insulation R-value Inability to identify asbestos Failure to achieve minimum air-sealing Inaccurate wall, window, and floor measurements Improper combustion safety testing (CO readings) Failure to identify combustion appliance hazardous

conditions (e.g., gas leaks) Failure to complete Basic Path measures for Advanced

projects

30May 1, 2012

Top 10 Performance Gaps (continued)

Performance Gaps (continued) Information does not agree across documents (JRT,

Energy Pro, forms) Inaccurate duct leakage tests

Inaccurate assessment of insulation coverage and quality

Refer to: “Performance Gaps Observed in SCE Whole House

Training” (Whole House Process Evaluation Interim Report Workbook, Caption Tab, ID=74)

31May 1, 2012

Proposed Inspector-Trainer Role

Establish an internal inspector-trainer within each contractor’s staff—will be responsible for: Attending all required training Providing in-house training Ensuring quality of work

Suggestions: Collaborative effort between QA/QC (RHA) and mentors

(ICF) for training inspector-trainers Develop and provide train-the-trainer materials for

inspector-trainers

32May 1, 2012

Proposed Marketing and Sales Training

Finding Little marketing and sales content in current

training materials Recommendation

Suggest formal marketing and sales training focused on contractor sales force

Classroom opportunity to practice EUC program sales skills

Opportunity to apply adult learning principles

33May 1, 2012

From Interim Report - Ongoing Safety Training Findings

Safety issues are noted frequently and may be missed by contractors

34May 1, 2012

Safety Issue SCE/SCG (n=16) PG&E (n=51)Possible asbestos in ducts - noted by inspector 7 NAPossible asbestos in ducts - noted by contractor 3 2CAZ failure in Test-in 4 12CAZ failure in Test-out 6 3CAZ failure not corrected after Test-in failure 2 0Mechanical ventilation recommended or required - noted in pre-application

1 4

Mechanical ventilation recommended or required - noted in post-application

7 7

From Interim Report - Ongoing Safety Training (continued)

Recommendations Provide ongoing safety training to contractors. Based on

feedback from inspectors and QC Issues to emphasize, include:

Low house air leakage Possible asbestos in ducts CAZ testing Correct installation of insulation

Refer to: “Ongoing safety training” (Whole House Process Evaluation – PG&E_SCE

Interim Report Workbook, Recommendations Tab, ID=17) Relevant To

PG&E and SCE, BIG and ICFI / RHA

35May 1, 2012