California Public Utilities Commission
Energy Efficiency Program Evaluation of the Group B Sectors
Research Workplan
WE&T Sector Area
December 3, 2019
opiniondynamics.com Page ii
With Subcontractors
opiniondynamics.com Page iii
Table of Contents
1. Workforce Education & Training (WE&T) .......................................................................................................... 1
1.1 Area Staffing Plan .................................................................................................................................... 3
1.2 Deliverable 26 (WE&T-1): WE&T and Installation Improvement Evaluation Study ............................. 4
1.3 Deliverable 27 (WE&T-2): Partnerships with Training Institutions Impact Evaluation ...................... 15
1.4 Deliverable 28 (WE&T-3): WE&T Career Connections Process Evaluation ........................................ 23
1.5 Deliverable 29 (WE&T-4): Career and Workforce Readiness Process Evaluation ............................ 28
1.6 Deliverable 30 (WE&T-5): Knowledge, Skills, and Abilities Market Studies ...................................... 32
Workforce Education & Training (WE&T)
opiniondynamics.com Page 1
1. Workforce Education & Training (WE&T)
The California Long Term Energy Efficiency Strategic Plan (CLTEESP) sets forth the WE&T vision, “By 2020,
California’s workforce is trained and fully engaged to provide the human capital necessary to achieve
California’s economic energy efficiency and demand-side management potential.”1 This document outlines
the work scopes for the CPUC Group B WE&T research and evaluation studies, which together examine
multiple facets of the Workforce Education and Training landscape. Through these deliverables, we will:
◼ Assess the causal link between IOU workforce training programs and improved installation practices
(Deliverable 26);
◼ Seek to understand the functioning and impacts of partnerships with WE&T training and job
placement organizations (Deliverable 27);
◼ Conduct a process evaluation of the WE&T Career Connections Program (Deliverable 28);
◼ Conduct a process evaluation of the Career & Workforce Readiness Programs (Deliverable 29); and,
◼ Identify knowledge, skills, and abilities (KSA) needed for implementers, contractors, technicians, or
contractor laborers on energy efficiency concepts (Deliverable 30).
Conceptual Models
This research will be guided by two complementary conceptual frameworks: Bloom’s Taxonomy for the
Cognitive Domain and Kirkpatrick’s
Model for training evaluation. Bloom’s
Taxonomy is useful for defining
learning objectives and learning
outcomes while Kirkpatrick’s model
is broader in focus and addresses
the overall assessment of training
programs.
Bloom’s Cognitive Taxonomy is a
core element in the education and
training community. The full
taxonomy addresses three
domains: Cognitive (thinking),
Affective (feeling), and Psychomotor
(physical movement.) However, the
Cognitive Domain is the most
comprehensive. The original
taxonomy, established by Bloom in 1956, defined the levels in the cognitive domain as Knowledge,
Comprehension, Application, Analysis, Synthesis, and Evaluation. These levels were refined in the 1990s to
reflect the levels shown in Error! Reference source not found..
1 California Public Utilities Commission. “California Long Term Energy Efficiency Strategic Plan” Sept. 2008, updated Jan. 2011.
Accessed May 8, 2017. http://www.leginfo.ca.gov/pub/15-16/bill/sen/sb_0301-0350/sb_350_bill_20150911_enrolled.pdf
Figure 1: Bloom's Updated Cognitive Taxonomy
Workforce Education & Training (WE&T)
opiniondynamics.com Page 2
The Kirkpatrick Model is the gold standard framework in adult training circles for assessing training programs
and is noted in The California Evaluation Framework discussion of Information and Education programs.2
Where applicable, we will utilize this framework to guide our approach to these five WE&T studies. As
illustrated in Figure 2, Kirkpatrick’s Framework consists of four levels. The first level is Reaction. Reaction
measures how participants feel
about the learning experience. The
value of Level 1 is that a good
training experience improves
knowledge transfer. Level 2 is
Learning. Learning assesses the
degree participants change
attitudes, increase knowledge, or
enhance skills as a result of the
learning experience. The value of
Level 2 is to demonstrate that
learning occurs as a result of the
training. The third level is Behavior.
Behavior measures the degree to
which participants apply what they
have learned outside of the learning
environment. This level seeks to
demonstrate whether trainees take
the information they learn and apply
it. Finally, Level 4 is Results. Results refer to the degree targeted outcomes are achieved system wide. For
results, we seek to measure the program’s overall impacts and tangible results, such as energy savings, job
creation, job placement, improved quality, and increased productivity. The value of measuring Level 4 is to
inform the return on training investment that a program, entity, or organization realizes from the training
endeavor.
Given the breadth of the work conducted under the WE&T program implementation budget and the limitations
in evaluation funding, one of the key areas of consideration is how to prioritize any evaluation efforts--
specifically when the focus is on assessing energy impacts in training interventions that have a large breadth
and depth of types, topics, pre- and post-knowledge level of attendee population, large variability in decision-
making ability of the attendee population, participation population type and so forth. With this in mind, Opinion
Dynamics has worked to balance level of rigor, timeframes, potential data availability and scope among the
five WE&T studies. However, we are flexible and adaptable to changes in the work scopes based on what we
learn each year compiling Deliverable 06: Gaps and Emerging Issues Report and through conversations with
the Energy Division Project Manager for WE&T and the Project Coordination Group (PCG).
2 Note that Bloom’s Cognitive Domain Taxonomy and Kirkpatrick’s Model are complementary. The first is more focused on learning
objective and training design and the latter is a broader framework centered on evaluation.
Figure 2: Kirkpatrick’s Model
Workforce Education & Training (WE&T)
opiniondynamics.com Page 3
1.1 Area Staffing Plan
Dr. Ellen Steiner, Vice President at Opinion Dynamics, will serve as the overall area lead for the WE&T
deliverables. Dr. Steiner is a passionate researcher who embraces analytic challenges and is committed to
ensuring the useful application of results and insights to inform systems thinking and strategy development.
Dr. Steiner has 19 years of experience in training and development. She served as a professor in the
Department of Education at the University of Denver teaching graduate level research and statistic courses,
as well as educational assessment courses focusing on the K-12 population. She also advised over 100
doctoral dissertations. Dr. Steiner parlayed this experience into work at Sun Microsystems where she designed
the company’s corporate-wide competency management system--conducting pre-inventory data reviews, job-
task analyses, and verification processes to define knowledge, skills and abilities for a wide range of job
families. She also created supporting learning paths to develop the organization’s current and future skill
base. Dr. Steiner has evaluated numerous training programs – not just in energy, but also for Sun
Microsystems, and the University of Denver. She holds a Certificate in Measuring and Evaluation of Training
Programs from the American Society for Training and Development. In addition to her adult education and
training expertise, Dr. Steiner’s specialties include qualitative research, program evaluation, policy analysis,
ZNE, HVAC, customer engagement, market research, and mixed-method research designs.
Dr. Steiner holds a Ph.D. in Quantitative Research Methods from the University of Denver, an M.A. in Higher
Education Administration from the University of Denver, and a B.S. in Human Development and Family Studies
from the University of Delaware. Dr. Steiner’s background in social sciences and her applied business
experience support her effective communication and application of results and insights to inform cutting edge
program design, program evaluation, customer experience initiatives, and strategy development.
Deliverable Timelines
In Table 1, we identify the timelines for Deliverables 26-30.
Table 1. Consolidated Timelines for WE&T Studies
Task Q4
2018
Q1
2019
Q2
2019
Q3
2019
Q4
2019
Q1
2020
Q2
2020
Q3
2020
Q4
2020
Q1
2021
Q2
2021
Q3
2021
Q4
2021
Final Workplans x x
Deliverable 26
(WE&T-1): WE&T
and Installation
Improvement
Evaluation Study
x x x x x x x x x x x x
Deliverable 27
(WE&T-2):
Partnerships with
Training Institutions
Impact Evaluation
x x x x x x x x x x x
Deliverable 28
(WE&T-3): WE&T
Career Connections
Process Evaluation
x x x x x
Deliverable 29
(WE&T-4): Career x x x x x
Workforce Education & Training (WE&T)
opiniondynamics.com Page 4
Task Q4
2018
Q1
2019
Q2
2019
Q3
2019
Q4
2019
Q1
2020
Q2
2020
Q3
2020
Q4
2020
Q1
2021
Q2
2021
Q3
2021
Q4
2021
and Workforce
Readiness Process
Evaluation
Deliverable 30
(WE&T-5):
Knowledge, Skills,
and Abilities Market
Studies
x x x x x
1.2 Deliverable 26 (WE&T-1): WE&T and Installation Improvement
Evaluation Study
Past research3 has demonstrated that market actors who participate in education and training programs make
changes to their practices that result in energy savings. Over half (59%) of market actors who took Energy
Center courses between 2006 and 2009 made changes that became standard practice. Two of five market
actors (43%) said they made changes that resulted in self-described “measurable” energy savings.
The same study conducted a savings analysis of 29 market actors who took courses in lighting, building
envelope, and HVAC. The case studies estimated that these 29 market actors made changes based on what
they learned that resulted in approximately 10,000 MWh and a large majority of the market actors (87%)
shared what they learned with others. Thus, there is likely a multiplier effect where the course impacts extend
beyond the course taker and makes the potential for savings quite large.
This study is especially important to truly understand the causal links between installer training and installer
practice. For example, quality HVAC installation is a highly technical activity in which improper execution of the
necessary steps can lead to incorrect diagnoses of problems as well as solutions that can decrease efficiency;
and thus, it is essential installers are learning how to install HVAC units to maximize energy savings. However,
there are challenges attributing savings to market actors. Some of these challenges are at least partially
mitigated through the focus on installer practices in this particular study; thus limiting--to some extent--the
number of potential outcomes due to training. However, there are also challenges specific to market actors
such as:
◼ Attribution: Number and type of classes, level of knowledge of market actors, multiple classes by
market actors, multiplier effects create attribution issues
◼ Market actors likely apply what they learn numerous times across a variety of projects. Some
market actors may be able to describe a “typical project” that will accurately reflect most of their
work and could thus be extrapolated to all of their work in a given year. Other market actors work
on a variety projects and there is no typical project. As a result, it is difficult to conduct a survey
that is not overly burdensome yet also captures the full range of savings achieved.
3 Opinion Dynamics, Wirtshafter Associates, Inc., Jai Mitchell Analytics, Summit Blue Consulting, “Indirect Impact Evaluation of the
Statewide Energy Efficiency Education and Training Program.” Retrieved May 5, 2018.
http://www.calmac.org/publications/06%2D08%5FStatewide%5FEducation%5Fand%5FTraining%5FImpact%5FEval%5FVol%5FI%5F
FINAL%2Epdf
Workforce Education & Training (WE&T)
opiniondynamics.com Page 5
◼ Some market actors will work in areas that are not part of an IOU territory. It could be difficult to
ensure that savings is only estimated for those projects that took place in IOU territory.
◼ Market actors work on a number of projects and many may have received support from another
IOU program. It is challenging to avoid double counting of savings in these cases.
◼ Market actors are often one member within a team in new construction or retrofit practices, thus
influences over ultimate energy saving action may have had multiple sources.
◼ Past research has shown that market actors change their marketing practices as a result of what
they learn from the Energy Centers. The courses provide information they can use to sell customers
on more energy efficient options that the market actor may be aware of before taking the course.
It is challenging to estimate the additional installations and savings resulting from improved
marketing.
◼ These challenges are then compounded among market actors in that market actors can touch
multiple buildings that may vary in equipment installed, size of building, and usage patterns.
Moreover, education and training can not only affect the type of equipment installed, but also the
practices of the market actors.
◼ Data collection: Given the multitude of reach of market actors, and the potentially multiple influencing
factor over the construction and/or retrofit, data collection for this population is more complex, and
even in the eventuality that it can be collected, the actual energy savings estimate will require
assumptions on multiplier effects and attribution of savings.
Despite these challenges, there is an expectation that market actors in fact provide outsized savings,
especially when compared to end-users, given their potential reach. In addition, in the state of California, 94
PA energy efficiency programs in 2017 involved a contractor that performs retrofit installation and/or
maintenance work on existing buildings. Lighting and HVAC equipment are the most common equipment types
included in these programs--with 51% of programs involving one equipment type, the other, or both. Other
common measures include water heaters (15%), weatherization (14%), and refrigeration (12%). In addition,
most programs involve installation only (70%) or both installation and maintenance work (21%).4 Few
programs include maintenance only. Thus, installation is a high impact area for many of California’s key energy
efficiency programs making assessing energy savings for training of installers essential.
WE&T Program Administrators continue to develop classes that are geared toward market actors because of
the belief that they have a wider reach, and that they can have a trickle-down--and multiplier effect--across the
teams they work with as well as the sheer number of programs market actors touch. Thus, despite its
challenges, it is important to ascertain causal links between installer training and installer practice.
1.2.1 Study Objectives
The study objectives include:
◼ Assess the effectiveness of WE&T programs specifically among installation professionals;
◼ Assess the impact of each training workshop/session provided for relevant professions;
4 Opinion Dynamics, “Responsible Contractor Policy for EE Programs: Market Intelligence Study.” Accessed December 31, 2017.
http://www.calmac.org/publications/Responsible%5FContractor%5FPolicy%5FStudy%5FReport%5FFINAL%2Epdf
Workforce Education & Training (WE&T)
opiniondynamics.com Page 6
◼ Identify and assess the connection between training and improved installation practices;
◼ Provide recommendations on how to improve WE&T programs; and,
◼ Collect data to assess whether and to what extent WE&T programs drive energy savings.
Due to the broad scope of WE&T programs, and the wide range of energy applications and diverse set of trade
and professional job opportunities to which they apply, we will focus our investigation on residential and small
commercial HVAC (including heat pump water heaters) WE&T, exploring past data from the Residential Quality
Installation program to guide our assessment.
1.2.2 Overview of Evaluation Methodology
To assess the effectiveness of installer training, the Opinion Dynamics Team needs to determine the key
knowledge, skills and abilities (KSAs) that are needed to effectively conduct energy efficient installations.
Since the focus of Deliverable 30 is defining KSA’s for HVAC installation, we will focus this evaluation on HVAC
installation related courses. The overall objective of this study is to assess if there is a causal link between
IOU workforce training programs and improved installation practices, we have devoted our research resources
to focusing on examining this link. We will prepare a white paper investigating how NMEC-based analyses can
be employed to provide quantitative indicators of changes in installation practices as a result of training efforts
consistent with Kirkpatrick’s Level 4 criteria.
We will utilize the Kirkpatrick Model and Bloom’s Cognitive Taxonomy discussed above as frameworks of this
study. The Opinion Dynamics Team will also look for synergies between this study and Deliverable 27 as we
know several of the installer learning interventions are delivered through partnerships. The table below
identifies the methods we will utilize to meet the study objectives.
Table 2. WE&T Deliverable 26 Study Methodology by Research Objectives
Study Objective
Materials
&
Database
Review
Energy
Division
and
WE&T
Staff
Interviews
NMEC
Assessment
Exit
Survey
Installer
Surveys
Instructional
Design
Assessment
Assess
Learning
&
Behavior
Conduct
Impact
Evaluation
Assess the
effectiveness of
WE&T programs
specifically
among
installation
professionals
✓ ✓ ✓ ✓ ✓ ✓
Assess the impact
of each training
workshop/session
provided
✓ ✓ ✓ ✓ ✓ ✓ ✓
Identify and
assess the
connection
between training
✓ ✓ ✓ ✓ ✓ ✓
Workforce Education & Training (WE&T)
opiniondynamics.com Page 7
Study Objective
Materials
&
Database
Review
Energy
Division
and
WE&T
Staff
Interviews
NMEC
Assessment
Exit
Survey
Installer
Surveys
Instructional
Design
Assessment
Assess
Learning
&
Behavior
Conduct
Impact
Evaluation
and improved
installation
practices
Provide
recommendations
on how to
improve WE&T
programs
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Collect data to
assess whether
and to what
extent WE&T
programs drive
energy savings.
✓ ✓ ✓ ✓ ✓ ✓
1.2.3 Evaluation Methodology: Detailed Efforts and Deliverables
Task 1. Materials and Database Review
Opinion Dynamics will complete a review of program materials, such as course catalogs, syllabi and other
available course materials, implementation plans, learning intervention types, exit survey data from courses,
and the course databases, program administrator business plans, past evaluations, key overarching studies
such as the Needs Assessment, and relevant policies, such as SB-350. We will utilize much of the information
obtained in Deliverable 30 to maximize research investment. We will submit an initial request for relevant
program materials, budgets and participant data available. Upon initial review of these data, we will submit a
second data request for specific information tied to courses focused on HVAC installers. These documents
may also include updated indictors, metrics, goals, participant tracking databases to date, and all available
curriculum plans.
Reviewing these materials will allow the team to:
◼ Develop a base understanding of the current program design and implementation including any sub-
components and how they address installers;
◼ Understand what data is available at the learning intervention level;
◼ Catalog the learning interventions targeted at installers in HVAC;
◼ Identify key program management and implementation staff we want to interview in Task 2 and
develop the questions that we want to pose to them; and,
◼ Ascertain what data is captured from exit surveys and how they meet Kirkpatrick’s Level 1
objectives.
Deliverables: Two Data requests
Workforce Education & Training (WE&T)
opiniondynamics.com Page 8
Task 2: Energy Division and WE&T Program Staff Interviews
The Opinion Dynamics Team will complete interviews with key individuals from the Energy Division, the PAs,
and stakeholders to develop a picture of how WE&T learning interventions are addressing the needs of HVAC
installers. The budget currently anticipates that we will conduct up to six interviews in 2019 and six interviews
in 2020.
The interviews with the Energy Division will be held in person or via telephone. All key Commission staff will be
invited to attend. Our key research team will conduct the Energy Division interviews.
PA staff will include the Center Directors and other key utility staff as determined to be relevant. These
interviews will take place via telephone. Through these interviews, we will:
◼ Discuss what changes are planned as a result of the business and implementation plans;
◼ Understand origin of metrics, indictors and goals, and the data to support them;
◼ Explore the effectiveness of various program components;
◼ Understand partnerships and the tracking of partnerships related to installation (if this data is not
already collected for Deliverable 27); and,
◼ Explore how adult learning principles are used/applied for each learning intervention targeted at
installers.
The outcomes of the interviews will consist of a complete and up-to-date set of current and future learning
interventions targeted at the installers of interest, a list of partners involved with the target audience, and
understanding of updated goals, indicators and metrics.
Deliverables: Draft and final interview guides
Task 3. Assessing the Use of NMEC Methodology to Evaluate WE&T Programs
In this task, Opinion Dynamics and Tierra Resource Consultants will develop a whitepaper that explores how
the overarching NMEC methodology may be applied to the evaluation of WE&T programs. In this whitepaper,
we will explore:
◼ What WE&T non-resource program performance metrics might be assessed through NMEC
techniques, including new metrics that will advance the use of NMEC?
◼ How program and meter data be can matched to participants in a WE&T non-resource activity such
that an NMEC analysis would provide meaningful results including quantitative metrics on changes in
installation practices?
◼ What is state of NMEC development and how might its evolving application relate to the WE&T sector?
◼ What is the most effective ways to develop baselines in a complex environment?
◼ Can NMEC methods be used to supplement data resources and provide new tools to specifically
assess the effectiveness of WE&T programs targeted at end users and market actors?
Workforce Education & Training (WE&T)
opiniondynamics.com Page 9
◼ Can NMEC methods be used to provide quantitative Kirkpatrick Level 4 performance metrics indicating
real changes in installation practices leading to energy savings?
Deliverables: Draft and final whitepaper
Task 4: Review and Refine Learning Intervention Exit Survey
The Opinion Dynamics Team will collaborate with the PA’s to make revisions to ensure those surveys covers
all aspects of Kirkpatrick’s level 1 evaluation. We will make sure the exit survey includes rating scale questions
and open-ended questions. Examples of such questions include:
◼ On a scale from 0 representing Strongly Disagree and 10 representing Strongly Agree, I was given
adequate opportunity to practice what I have learned.
◼ On a scale from 0 representing Not at All Confident and 10 representing Extremely Confident, how
confident are you that you will be able to apply what you have learned back on the job?
◼ What assistance or resources will you need to successfully apply what you learned on the job?
We will ask the utilities to use the same refined exit survey in all learning interventions targeted at installers
in 2020 and part of 2021. We will analyze this data to include in our final report
Deliverables: Refined exit survey, analysis of exit survey data
Task 5: Conduct Installer Surveys
Building upon any existing exit survey data, we will conduct surveys of HVAC installers who have participated
in a WE&T learning intervention in 2019. The objective of this survey is to address Kirkpatrick’s Levels 1-
Reaction, Level 2-Learning and Level 3-Behavior. Since we will likely have exit survey data that covers much
of Level 1, we will only include missing Level 1 components. We will focus on understanding:
◼ Reaction—Often called a “smile sheet”, we will assess satisfaction with all elements of a course.
◼ Learning—We will assess development of knowledge and skills as well as motivations to participate
and how they will use the learning experience to further their careers.
◼ Behavior—We will determine how participants have applied the knowledge and skills they learned in
the work place as well as identify any challenges they have encountered.
We will also focus on understanding the number of WE&T learning interventions installers have participated
in as well as key demographics. Finally, we will collect data that could be used to assess whether and to what
extent WE&T programs drive energy savings.
In order to determine the sample frame, we will utilize the program tracking database. In the Indirect Impact
Evaluation of the Statewide Energy Efficiency Education and Training Program Study,5 Opinion Dynamics et al.
made the recommendation that all Centers create a “common registration form that is used across all Centers
including participant type, profession, years in profession, and existing knowledge.” Unfortunately, this
5 Opinion Dynamics, Wirtshafter Associates, Inc., Jai Mitchell Analytics, Summit Blue Consulting, “Indirect Impact Evaluation of the
Statewide Energy Efficiency Education and Training Program.” Retrieved May 5, 2018.
http://www.calmac.org/publications/06%2D08%5FStatewide%5FEducation%5Fand%5FTraining%5FImpact%5FEval%5FVol%5FI%5F
FINAL%2Epdf
Workforce Education & Training (WE&T)
opiniondynamics.com Page 10
recommendation was not acted upon, so missing data will likely be an issue. For budgeting purposes, we have
planned to conduct an online survey with up to 200 installers. We will plan to refine our numbers based on
the data gathered in Tasks 1 and 2. We will offer a $75 gas card as a token of appreciation. From past
research, we know that gas cards are particularly motivating for this population. We have also budgeted to do
phone-call follow-ups to increase our response rate. We will conduct these surveys in Year 2.
Deliverables: Sample plan; draft and final survey
Task 6: Instructional Design Assessment
Using data from Task 1 and the survey in Task 5, Opinion Dynamics will perform an instructional design
assessment of up to 8 HVAC installer targeted WE&T learning interventions. We will gather data as described
below to address the following questions for each learning intervention.
◼ Do the WE&T learning interventions meet the needs of installers?
◼ Do the learning interventions target an appropriate range of learning levels and performance
outcomes (e.g. from basic learning levels, such as Knowledge and Comprehension, to more
sophisticated levels, such as Application, Analysis, Synthesis, and Evaluation6)?
◼ Do the objectives, content, and activities of the learning interventions target the knowledge, skills
and abilities defined in Deliverable 30?
◼ Do course materials and related supporting materials (e.g. checklists, job aids, and references)
help ensure an effective transfer from the classroom to the job? To the end-user?
◼ Are the learning interventions using effective learning strategies for the target audience?
◼ Do the materials reflect strategies and tactics that support adult learning principles (e.g. obtain
learner buy-in, build on what learners know, engage the learners, set up learners for success, let
learners apply what they have learned, etc.)?
◼ Do the materials reflect strategies and tactics consistent with best practices in adult learning (e.g.
lesson plan, content decisions, learner centricity, interactive activities, practice opportunities,
facilitation, feedback, assessments)?
◼ Are instructors delivering the training as it is designed — and if not, how do deliveries differ from
the apparent design intent?
◼ Do instructors employ adult learning principles and best practices effectively?
◼ How can WE&T programs improve offerings for installers?
◼ How can course focus, design, and materials better reflect the needs of the target audiences?
◼ How can the courses better employ learning strategies appropriate to class participants?
◼ How can instructors’ deliveries be refined to better incorporate the key adult learning principles
and practices?
6 Based on Bloom’s Updated Cognitive Taxonomy
Workforce Education & Training (WE&T)
opiniondynamics.com Page 11
In order to assess these learning interventions, we will conduct the following two sub-tasks for the up to 8
selected courses: (1) refine evaluation yardsticks; and, (2) evaluate course materials. These are described
below.
Refine Evaluation Yardsticks
The Opinion Dynamics Team will refine the evaluation yardsticks that we developed in past process
evaluations for California WE&T that focus on adult learning principles and practices, as well as program goals
and objectives. We will incorporate the HVAC KSA’s developed in Deliverable 30 and update the program goals
and objectives targeting installers. Whenever appropriate, we will use the same criteria used in the previous
process evaluations, so we can effectively compare results from those efforts to findings from this study. At
this time, we anticipate that it will address how well the training:
◼ Employs adult learning best practices;
◼ Supports installer-specific needs; and,
◼ Develops job-relevant knowledge, skills and abilities (helps prepare participants for job advancement
or job placement).
Our team will conduct a “yardstick review” of relevant available HVAC installer course materials for learning
interventions targeted at installers. We anticipate that the available materials will vary among the courses,
and may include:
◼ Lesson plans
◼ Instructor guides
◼ Presentation materials
◼ Participant workbooks
◼ References, job aids, and other participant support materials
◼ Materials for demonstrations or hands-on practice
◼ “Homework” assignments, if the course spans multiple days or is tightly integrated with other courses
◼ Quizzes, tests, polls, etc.
Based on our work thus far, we will coordinate with the Energy Centers Staff and the Energy Division Project
Manager for WE&T to select the two installer learning interventions that are most tied to impacting installation
practices and contributing to potential energy savings. In particular, we will leverage information on trainings
that we glean from our review of course materials (Task 1), installer surveys (Task 5) and the instructional
design assessment (Task 6) to develop questions that will establish a baseline of knowledge and installation
behaviors. We will build on these data in subsequent tasks to assess knowledge and behavior change that
may contribute to additional energy savings. This will include an in-person audit and instructor interviews
described below.
In-person Audit of Learning Interventions
In-person audits by the study team will allow us to assess whether:
Workforce Education & Training (WE&T)
opiniondynamics.com Page 12
◼ Learning intervention implementation is consistent with the design and intent of the course materials;
◼ Instructors are successfully using adult learning principles and practices in structuring and delivering
learning interventions; and
◼ How course materials support additional energy savings.
Conduct Instructor Interviews
Opinion Dynamics will conduct on-site instructor interviews as part of the in-person audit of learning
interventions, depending upon the instructor’s availability and willingness to participate in the interview. We
anticipate that on-site interviews may include discussions with 14 different instructors as there are some
courses that include multiple instructors.
These interviews will consist primarily of open-ended questions to allow for expansion on topics, and will focus
on:
◼ The instructor’s previous experience with and training in adult learning principles and practices;
◼ Ways in which instructors tailor the use of the materials to better meet the needs of the participants;
◼ Instructors’ assessments of how training may improve energy efficiency projects and lead to additional
energy savings;
◼ Special considerations that may affect findings from the in-person audit (e.g. whether this was a
“typical” delivery of the class or whether there were special circumstances); and,
◼ Instructors’ suggestions for refining the course design and materials and enhancing processes related
to design and delivery of courses within the program.
We will develop an interview guide to support these discussions to help ensure all relevant questions are
addressed during the conversation.
Deliverables: Evaluation yardsticks, draft and final interview guides, course-specific data from review of
materials, course-specific data from in-person audits.
Task 7: Assess Learning
In this task, we will assess Kirkpatrick’s Level 2 – Learning for the two selected installer focused HVAC learning
interventions. At this level, we will measure the degree to which participants change attitudes, increase
knowledge, and enhance skills as a result of the learning experience. For these two learning interventions, we
will develop pre- and post- tests administered via web survey to ascertain knowledge and skills gain and
application based on the learning objectives of the learning intervention. We will also explore the possibility of
using simulations (such as the HVAC Job Ready simulations that Interplay Learning is developing with the
California Community Colleges) to gain insight on ability to apply the knowledge and skills in real-world settings.
We will develop effective questions to address the relevant levels of Bloom’s Taxonomy7 that align with the
learning objectives. We will ask the Energy Centers to use these pre- and post-tests in all occurrences of the
two learning interventions in Year 2.
7 Bloom’s Taxonomy is a classification system used to define and distinguish different levels of human cognition including Knowledge,
Comprehension, Application, Analysis, Synthesis, and Evaluation.
Workforce Education & Training (WE&T)
opiniondynamics.com Page 13
Deliverables: Draft and final pre- and post-tests
Task 8: Assess Behavior
Using the same two installer focused learning interventions prioritized in Task 7, we will assess Kirkpatrick’s
Level 3 – Behavior. In this level, we measure the degree to which participants apply what they have learned
outside of the learning environment. While we will have the self-report data regarding application of learning
in the installer survey, we think it is essential to actually assess the application of the learning through direct
observation. To do this, we will recruit participants to conduct ride-alongs with to observe them in the field. We
will rely on our trained consulting staff to recruit participants for this effort, potentially using the utilities and
instructors to add credibility to our recruitment process. We will explain to participants that we are interested
in refining the curriculum for the class they participated in to reflect more real-world experience as opposed
to assessing their installation knowledge, skills, and abilities. We expect each session to last one day and will
work to schedule the day when the installer is working on installation activities (as opposed to invoicing or
other duties).
During each ride-along, Eric Shum or Tom Hines of Tierra Resource Consultants, will carefully observe the
installer using a rubric developed to align with the KSAs developed for Deliverable 30 and the learning
objectives of the learning intervention.8 During the down-time between appointments, Mr. Hines or Mr. Shum
will debrief with the participant about the last customer appointment to dive deeper into the intricacies of the
decision-making process as well as the installer’s motivations, and challenges. We will also use this
opportunity to ask about the installer’s definition of installation success and his or her experience with the
learning intervention. We have budgeted to conduct ride-alongs with up to 20 installers.
There are often differences between what people say and what they do in a real situation. This dual-pronged
method, which combines observation and interviewing, allows us an opportunity to identify these gaps. Mr.
Shum or Mr. Hines will take notes and obtain photographic and audio documentation when possible. We will
offer installers a $500 incentive in appreciation of their time.
Deliverables: Draft and final observation protocols
Task 9: Conduct Indirect Impact Evaluation
The California Protocols, as well as the International Performance Measurement and Verification Protocol
(IPMVP), outline methods for measuring indirect energy savings impacts. Applying these methods to education
and training efforts, however, poses several challenges as discussed earlier. One of these challenges is the
potential difference in training audiences. WE&T training programs often serve end-users and market actors.
End users are defined as individuals who are in a position to undertake direct behaviors that lead to energy
savings actions. For example, a facility manager is in a position to take energy savings actions in one or more
of the premises they manage. In contrast, market actors are defined as individuals that may influence energy
savings in multiple premises over a long period of time. Examples of such individuals include, architects,
designers, and contractors who are in a position to share the lessons learned through the WE&T experience
to others who did not participate in the intervention. For this study, we will focus on end-users of who are in a
position to take direct energy savings actions.
To conduct the indirect impact evaluation, we will use the data collected in earlier tasks to inform engineering
analysis to determine gross energy impacts. This is the process of asking participants detailed questions
8 A rubric is an evaluation tool which outlines performance criteria and definitions of multiple levels of achievement for each
performance criteria.
Workforce Education & Training (WE&T)
opiniondynamics.com Page 14
regarding actions taken after program participation, then using engineering analysis to estimate savings due
to these changes for each of the selected focus areas.
From our experience conducting impact evaluations on WE&T programs, we are well aware of the challenge
to collect sufficient information through a survey effort. One of the primary challenges to this approach is the
difficulty of collecting sufficient information regarding energy savings actions taken (e.g. replacement and
changes to O&M behaviors) within the short time frame of a telephone or Internet survey. However, we have
iterated survey content and question wording over time to maximize the reliability and validity of the data
collected through self-report surveys.
Since WE&T programs are often designed to channel attendees into other resource programs, we will also
measure cross-program attribution to ensure that gross savings from WE&T interventions are not also being
counted toward gross savings from other IOU EE programs or vice versa. We will assess cross-program
attribution of program savings through responses to the participant survey. Responses from this survey will be
used to identify whether any of the reported energy savings actions also utilized a rebate or incentive from an
IOU energy efficiency program. If so, the Opinion Dynamics Team will deduct the estimated energy savings
value from the total gross savings for the participant, using the equation below.
Gross Savings = Sum of Energy Savings Actions Estimates – Energy Savings Rebated through IOU EE program
The data gathered will inform gross impacts estimates, as findings will serve as inputs into engineering
algorithms. To support the engineering analysis, we will draw upon existing data sources such as the
Commercial Buildings Energy Consumption Survey (CBECS), the California Commercial End-Use Survey (CEUS),
and the Database of Energy Efficiency Resources (DEER) to determine savings for specific measures, as well
as to gather information on baseline load intensities among other uses.
Task 10. Reporting
The Opinion Dynamics Team will work with the Commission to develop draft and final reports in accordance
with CPUC specifications.
Deliverable: Draft and Final Reports
1.2.4 Study Timeline and Milestones
In Table 3, we identify the timeline for the study.
Table 3. Deliverable 26 Timeline
Task Q4
2018
Q1
2019
Q2
2019
Q3
2019
Q4
2019
Q1
2020
Q2
2020
Q3
2020
Q4
2020
Q1
2021
Q2
2021
Q3
2021
Task 1: Materials and
Database Review x x x x X x x x
Task 2: Energy Division
and WE&T Program
Staff Interviews
x x
Task 3: Assessing the
Use of NMEC
Methodology to Evaluate
WE&T Programs
x x x
Workforce Education & Training (WE&T)
opiniondynamics.com Page 15
Task Q4
2018
Q1
2019
Q2
2019
Q3
2019
Q4
2019
Q1
2020
Q2
2020
Q3
2020
Q4
2020
Q1
2021
Q2
2021
Q3
2021
Task 4: Review and
Refine Learning
Intervention Exit Survey
x x
Task 5: Conduct
Installer Surveys x x
Task 6: Instructional
Design Assessment x x x x
Task 7: Assess Learning x x
Task 8: Assess Behavior x x
Task 9: Conduct Basic
Indirect Impact
Evaluation
x x
Task 10: Reporting x x
In Table 4, we identify the milestones and total budget for the study.
Table 4. Deliverable 26 Milestones and Budget
Milestone Deliverable Budget Schedule
Milestone 1: Landscape
Analysis and NMEC (Tasks 1-3) NMEC Whitepaper
December
2019
Milestone 2: Course
Assessments (Tasks 4-6)
Course-specific data from in-
person audits July 2020
Milestone 3: Assessing
Learning and Behavior (Tasks
7 & 8)
Pre-tests, Post-tests, and
Observation Protocol
December
2020
Milestone 4: Impact Evaluation
and Reporting (Tasks 9 & 10) Final Report
October
2021
Total $525,000 October
2021
1.3 Deliverable 27 (WE&T-2): Partnerships with Training Institutions
Impact Evaluation
Program administrators are one of many organizations statewide that have interest in developing California’s
current and future energy workforce. The WE&T landscape is complex and diverse, spanning a number of high-
potential markets, delivery channels, occupations, and jobs. As the CLTEESP states, “An effective,
comprehensive WE&T program for a new energy efficient economy requires collaborative efforts by many
entities. It is not the core mission of utilities to effectuate the level of change needed to create a
comprehensive WE&T program, nor can ratepayers fully fund the effort.” The CLTEESP identifies government
agencies, educational institutions, community-based and non-profit organizations, and industry and labor
organizations as key partners in identifying and implementing effective workforce strategies to sustain and
grow a robust green energy economy.
Workforce Education & Training (WE&T)
opiniondynamics.com Page 16
In the business plans, program administrators continue to focus on partnerships with third-party entities as a
key cross-cutting strategy for meeting the state’s ambitious energy goals. However, interveners, such as the
Consortium of Energy Efficiency, posit that “Insufficient alignment between the WE&T programs and the
State’s core occupational training institutions, such as community colleges, apprenticeship programs, CSUs,
UCs, CBOs and workforce development boards, has been a major barrier to achieving the State WE&T goals.”
A review of WE&T annual reports also suggests that there is a wide variety of partnerships, but it is unclear
how these partnerships work together to achieve overall portfolio goals.
Thus, in order to understand the impacts of partnerships, we will need to define what a partnership is, and
what makes a partnership effective, before we can even begin to understand the impact outcomes such as
energy savings, job creation, or job placement.
1.3.1 Study Objectives
The research objectives for the partnerships with training institutions impact evaluation are to:
◼ Define the term partnership;
◼ Identify partnership types;
◼ Map existing and planned partnerships to partnership types;
◼ Determine the evaluability of partnerships;
◼ Understand the functioning and impact of partnerships with WE&T training and job placement
organizations;
◼ Characterize how partnerships are being implemented;
◼ Determine and measure partnership effectiveness indicators (such as joint ownership and
accountability for results, and effective communication and collaboration strategies); and,
◼ Assess the impacts (such as job placement and gross energy savings) of two WE&T partnerships as
case studies.
1.3.2 Overview of Evaluation Methodology
Table 5 below identifies the methods we will utilize to meet the study objectives.
Table 5. WE&T Deliverable 27 Study Methodology by Research Objectives
Study Objective
Data
Request and
Materials
Review
Program
Staff
Interviews
Define
Partnerships and
Catalog
Partnership
Types
Survey
Current
Partners
Job
Placement
Surveys and
Interviews
Conduct Impact
Evaluation
Define the term
partnership ✓ ✓ ✓
Identify partnership
types ✓ ✓ ✓ ✓
Workforce Education & Training (WE&T)
opiniondynamics.com Page 17
Study Objective
Data
Request and
Materials
Review
Program
Staff
Interviews
Define
Partnerships and
Catalog
Partnership
Types
Survey
Current
Partners
Job
Placement
Surveys and
Interviews
Conduct Impact
Evaluation
Map existing and
planned partnerships
to partnership types
✓ ✓ ✓ ✓
Determine
evaluability of
partnerships
✓ ✓ ✓ ✓
Understand the
functioning and
impact of
partnerships with
WE&T training and
job placement
organizations
✓ ✓ ✓ ✓ ✓
Characterize how
partnerships are
being implemented
✓ ✓ ✓
Determine and
measure partnership
effectiveness
indicators
✓ ✓ ✓ ✓ ✓
Assess the impacts
(such as gross energy
savings, job
placement and job
creation) of WE&T
partnerships
✓ ✓
1.3.3 Evaluation Methodology: Detailed Efforts and Deliverables
Task 1. Initial Data Requests, Program Material Review, and Literature Review
Opinion Dynamics will complete a review of program materials such as business plans, implementation plans,
past WE&T evaluations, program theory and logic models (PTLM), and relevant policies such as SB-350. We
will also conduct a comprehensive literature review of how partnerships are defined, what constitutes a
partnership, what are the characteristics of a good partnership and how partnerships have been evaluated.
We will conduct a systematic search of the available literature on partnerships, including conference
proceedings, regional report databases, Google Scholar, and academic databases. We will use a set of key
words to search each database in a consistent manner. At the outset of this effort, we will define the criteria
for inclusion in the literature review, which will ensure that the literature review process is efficient, focused,
and relevant. Opinion Dynamics will assess the results of keyword searches with respect to relevance based
on abstracts or executive summaries. Papers or reports deemed applicable to the topic at hand will then be
selected for a full text review.
We will submit an initial request for relevant program materials, budgets and partnership data as available.
These documents may also include updated program performance metrics as well as associated goals,
Workforce Education & Training (WE&T)
opiniondynamics.com Page 18
participant tracking databases to date, and all available partnership plans, contracts, memorandums of
understanding, etc. Reviewing these materials will allow the team to:
◼ Understand current program design and implementation including current and planned partnerships
and how they fit into the program and market landscapes;
◼ Understand what data is currently tracked related to each of the WE&T sub-programs and how data
is shared among the PAs; and,
◼ Identify key program management and implementation staff we want to interview in Task 2 and
develop the questions that we want to pose to them.
We will leverage background information collection and review across all studies in this proposal to minimize
PA burden and maximize the efficient use of evaluation resources. Opinion Dynamics will conduct a thematic
analysis of each of the identified data sources above using NVIVO—a powerful qualitative analysis tool—to
uncover evidence-based insights and findings. Using NVIVO, we will mine the data fully and ensure that none
of the data collection information is lost as we progress through the study. As necessary, we will submit a
secondary data request for additional material identified as relevant to this evaluation through Task 2.
Deliverables: Data request
Task 2: Conduct Program Staff Interviews
We will conduct semi-structured interviews with PA WE&T staff to understand the role of partnerships in each
of the WE&T programs and subprograms. The objective of speaking with WE&T staff is to understand: (1)
partnership definitions, (2) roles and responsibilities related to partnerships, (3) current and planned
partnerships by program and sub-program, (4) partnership goals, (5) the partnership development process,
(6) partnership tracking data, (7) partnership best practices and lessons learns, (8) partnership theories of
change, and (9) partnership evolution given planned modifications to the portfolio. We will also use these
interviews to understand how programs are meeting regulatory guidelines of expanding/initiating partnerships
with entities that do job placement and requiring placement experience for any new partners in WE&T
programs and new solicitations.9 We will also explore the nuances of PA funding related to job placement.
We anticipate these interviews happening via telephone and lasting approximately 60-90 minutes each. Our
team plans to conduct 16 of these interviews total—8 in Year 1 and 8 in Year 2—to understand how things
have evolved and to inform our partnership program data tracking reports in both years.
Deliverables: Draft and final interview guides, second data request (if needed)
Task 3. Update PTLMs, Define Partnerships and Catalog Partnership Types
A key to ensuring that this study is successful is to answer the question: “What is a partnership?” For WE&T,
A17-01-013 Rev 1 indicates that a final metric for “Expanding WE&T Reach via Collaborations” is the “number
of partnerships by sector (complete partnership defined by curriculum developed jointly + agreement). While
that is one definition of a partnership, based on our prior work, we hypothesize that there are many types of
partnerships and corresponding outcome paths that would influence how to define an impact evaluation so
that it does not waste valuable time and resources on a premature or inappropriately designed evaluation.
From our experience, we hypothesize there are different types of partnerships based on funding structures,
9 A17-01-013 Rev 1 Proposed Decision
Workforce Education & Training (WE&T)
opiniondynamics.com Page 19
agreement types, contractual relationships, and if partners provide oversight to subcontractors or deliver
services directly to the end-user.
Since partnerships are a strategy as opposed to a program, we will also catalog how these partnerships are
implemented and theories of change related to this strategy. We hypothesize there are multiple outcome
pathways for partnerships such as the co-development of curriculum, train the trainer agreements, or the
lending of energy saving tools. Theories of change link outcomes and activities to explain the how and why
the desired outcomes will occur. In contrast, logic models describe program activities and link them to outputs
and outcomes. The business plans propose changes to the structure of their WE&T portfolios10 and thus we
will update our past program theory logic models (PTLMs) to reflect the current WE&T structure.
Using all of the data compiled in the past three tasks, we will develop: (1) a working definition of partnerships,
(2) a classification system to catalog partnerships into homogeneous categories, and (3) updated PTLMs that
reflect the proposed changed reflected in the business plans and current thinking. We will codify our findings
from the past three tasks in a memo. We will vet the definition, taxonomy, PTLMs and crosswalk with the
Commission and the WE&T Program Coordinating Group (PCG).
Deliverables: Draft and final memo, draft and final presentation for PCG meetings
Task 4: Conduct Evaluability Assessment
An evaluability assessment (EA) is essentially a pre-evaluation process that examines the extent to which a
program (or in this case program strategy) can be evaluated in a reliable and credible way. It is aimed at three
major purposes:
◼ Is there an adequate theory of change?
◼ Is there a logical theory of change?
◼ Are there measurable goals?
◼ Are the long-term impact and outcomes clearly identified?
◼ Does it provide plausible causal links toward goals?
◼ Are there clear terms and agreements that define the partnership relationship?
◼ Has the partnership theory been implemented as planned?
◼ Have the planned activities been completed in the manner required by the theory?
◼ Are there adequate data to test the causal connections and outcomes?
◼ Do the program records/tracking systems contain appropriate information including participants,
near-participants and their relevant contact information as well as data to inform proximal and distal
indicators?
◼ Is there data on a potential control group?
◼ Do data exist or is it possible to collect outcome data that can test goal attainment?
To determine the best areas of focus for examining impacts given a limited evaluation budget, we will conduct
an evaluability assessment of each of the partnership types identified in the taxonomy described in Task 3.
10 e.g. Southern California Edison proposes consolidating the three traditional WE&T subprograms--Connections, Planning, and
Centergies--into one program called Integrated Energy Education and Training (IEET); PG&E proposes the same consolidated program
(IEET) which will include Core Energy Education and Technical Upskill Interventions.
Workforce Education & Training (WE&T)
opiniondynamics.com Page 20
Through this evaluability assessment, we will determine which partnership types (defined in Task 3) and what
specific partnerships have the most potential in meeting long-term outcomes such as energy savings, job
creation, and job placement. This information will be codified in an interim memo; findings from which will
heavily influence subsequent research tasks.
Deliverables: Draft and final memo
Task 5: Survey Current Partners
In order to verify our partnership taxonomy and help to inform the development of our impact evaluation plan,
we will conduct an online survey of entities currently partnering with PA WE&T programs. In developing survey
instruments, we will rely heavily on information gleaned from previous tasks, in particular the evaluability
assessment (Task 4). We will attempt a census of all partners collecting the key partner representative and
his or her contact information through a data request. We will conduct this survey in Year 2. We estimate that
these surveys will take approximately 15 minutes. Since what we define as a partner will be determined in
Task 3, we are not sure as to the number of partners. For budgeting purposes, we are assuming 20 partners
per year.
The objectives of this survey are to understand, from the partner perspective, the functioning and impact of
partnerships, validate information received from the PA’s, understand from the partner perspective what data
is available, capture any concerns regarding sharing data, understand the purpose of a partnership,
understand the goals of the partnership, characterize perceived success of the partnership, identify concerns
with the partnership and uncover any lessons learned. We will utilize these data to verify the accuracy of the
data collected so far and inform the impact evaluation planning in Task 9.
Deliverables: Draft and final survey
Task 6: Develop Impact Evaluation Plan
Using the data from the evaluability assessment and the partner survey, Opinion Dynamics will work with the
Commission to finalize an evaluation plan for an indirect impact evaluation in accordance to the specifications
discussed in Task 9 of this work plan. We will focus on a plan to measure impacts that align with the outcomes
identified in the PTLMs developed in Task 3, along with indirect energy savings that may result in partnerships
with WE&T programs.
Deliverables: Draft and final work plans for basic indirect impact evaluation
Task 7: In-Depth Interviews with Employers
To assess job-related impacts, we will conduct semi-structured interviews with employers that work with
occupational training organizations currently partnering with PA WE&T programs. We will focus these in-depth
interviews on employers that work with organizations focused on job training and placement that we identify
in Task 5. The research team will conduct interviews by telephone with representatives of these employers to
measure program-related impacts, such as job placement success, employee performance, and retention.
Opinion Dynamics will coordinate with program and occupational training partner organizations to obtain a list
of employers where partners have successfully placed candidates from 2018 to the present.
The research team plans to conduct interviews with up to 12 employers engaged with partner organizations
that focus on job training and placement. Based on information gleaned through the partner survey (Task 5)
we will adjust the number of necessary interviews accordingly. To ensure sufficient response rates, we will
Workforce Education & Training (WE&T)
opiniondynamics.com Page 21
offer $100 incentives (in the form of an Amazon gift card, or a donation to pre-selected charitable
organizations on their behalf) to employers willing to complete interviews.
Deliverables: Draft and final interview guides
Task 8: Survey with End-Users Receiving Job Training
WE&T training programs often serve end-users and market actors. End users are defined as individuals who
are in a position to undertake direct behaviors that lead to energy savings actions or receive job-related
benefits as a result of WE&T program activities. For this task, we will focus on end-users of the partnership
who are in a position to experience the outcomes identified in the PTLM updates. Examples might include
increased rates of job placement, change their behavior to become more successful in their positions, and
take direct energy savings actions.
Opinion Dynamics will conduct surveys with these end-users that have received services from participating
partner organizations. We will work with partner organizations to obtain a list of the clients they served in
2018, 2019, and the first half of 2020 and will draw a representative sample of individuals across
participating institutions. The objectives of this survey are to further develop an understanding of the types of
services that partner organizations provide, assess how those services may be augmented through their
partnerships with WE&T programs, and gather data to measure job-related impacts from WE&T programs—
e.g., job placement, performance, retention, and other job-related metrics identified in the evaluability
assessment (Task 4). The research team will work with the Commission to select two types of partnerships
based on the results of the evaluability assessment (Task 4). We will work with the Commission to determine
appropriate selection criteria, such as size, length of relationships, and data availability to inform the selection
of the two partnerships for evaluation. We will strive for each partnership selected to represent a different
partnership type. The research team will then develop a sample frame based on the partnership types and the
populations they serve gleaned from the partner survey (Task 5).
Opinion Dynamics will administer the survey online and plans to complete up to 140 surveys with end-users.
However, upon receipt of tracking data from partner organizations, we will adjust the sample frame and
number of anticipated survey completes accordingly. To ensure sufficient response rates, we will offer $25
incentives (in the form of an Amazon gift card, or a donation to pre-selected charitable organizations on their
behalf) to employers willing to complete interviews.
Deliverables: Draft and final survey
Task 9: Conduct Impact Evaluation
To conduct an impact evaluation, we will use participant surveys and secondary data to inform analyses to
estimate job-related impacts, and other gross energy impacts. From our experience conducting impact
evaluations on WE&T programs, we are well aware of the challenge to collect sufficient information through a
survey effort. One of the primary challenges to this approach is the difficulty of collecting sufficient information
regarding energy savings actions taken (e.g. replacement and changes to O&M behaviors) within the short
time frame of a telephone or Internet survey. However, we have iterated survey content and question wording
over time to maximize the reliability and validity of the data collected through self-report surveys.
The estimation of job-related and energy impacts requires detailed information about the training participating
organizations received, the energy-saving actions that end-users took, and specifications about equipment
and the location where end-users took action. In addition, we will need to collect detailed information regarding
Workforce Education & Training (WE&T)
opiniondynamics.com Page 22
the change in job placement and retention rates of end-users, along with information on their performance
since participating in trainings supported by WE&T programs.
Since WE&T programs are often designed to channel attendees into other resource programs, we will also
measure cross-program attribution to ensure that gross savings from WE&T interventions are not also being
counted toward gross savings from other IOU EE programs or vice versa. We will assess cross-program
attribution of program savings through responses to the participant survey. Responses from this survey will be
used to identify whether any of the reported energy savings actions also utilized a rebate or incentive from an
IOU energy efficiency program. If so, the Opinion Dynamics Team will deduct the estimated energy savings
value from the total gross savings for the participant.
Gross BOC Savings = Sum of Energy Savings Actions Estimates – Energy Savings Rebated through IOU EE
program
To support this impact evaluation, we will rely heavily on previous tasks, specifically the evaluability
assessment, to outline theories of change and impact-related goals to measure. Further, we will gather data
from a variety of secondary sources to inform gross energy impacts estimates. This may include but are not
limited to the Commercial Buildings Energy Consumption Survey (CBECS), the California Commercial End-Use
Survey (CEUS), and the Database of Energy Efficiency Resources (DEER) to determine savings for specific
measures, as well as to gather information on baseline load intensities among other uses.
Deliverables: Estimates of job-related impacts and gross indirect energy impacts
Task 10. Reporting
The Opinion Dynamics Team will work with the Commission to develop draft and final reports in accordance
to the specifications discussed in this proposal.
Deliverable: Draft and Final Reports
1.3.4 Study Timeline and Milestones
In Table 6, we identify the timelines and milestones for the study.
Table 6. Deliverable 27 Timeline
Task Q4
2018
Q1
2019
Q2
2019
Q3
2019
Q4
2019
Q1
2020
Q2
2020
Q3
2020
Q4
2020
Q1
2021
Q2
2021
Q3
2021
Initial Data Requests,
Program Staff Interviews x x x
Update PTLMs, Define
Partnerships and
Catalog Partnership
Types
x x x
Conduct Evaluability
Assessment x
Survey Current Partners x x
Develop Impact
Evaluation Plan x x
Workforce Education & Training (WE&T)
opiniondynamics.com Page 23
Task Q4
2018
Q1
2019
Q2
2019
Q3
2019
Q4
2019
Q1
2020
Q2
2020
Q3
2020
Q4
2020
Q1
2021
Q2
2021
Q3
2021
In-Depth Interviews with
Employers and End-Use
Surveys
x x
Conduct Impact
Evaluation x x
Reporting x x
Table 7 presents the milestones and total budget for this study.
Table 7. Deliverable 27 Milestones and Budget
Milestone Deliverable Budget Schedule
Milestone 1: Landscape
Analysis, PTLMs, and
Partnerships Definition
(Tasks 1-3)
PCG Meeting January 2020
Milestone 2: Partner Survey &
Impact Evaluation Planning
(Tasks 4-6)
Survey and Impact Evaluation Plan September
2020
Milestone 3: Employer
Interviews and End-User
Survey (7 & 8)
Interview Guide and Survey March 2021
Milestone 4: Impact Evaluation
and Reporting (Tasks 9 & 10) Final Report
October
2021
Total $470,000 October
2021
1.4 Deliverable 28 (WE&T-3): WE&T Career Connections Process
Evaluation
Numerous studies and legislation have informed and shaped the strategy for WE&T as proposed in the
business plans, prompting program administrators to adjust their program offerings for better alignment and
coverages of California's energy efficiency industry, as well as redirect some of the program focus to
disadvantaged communities. For 2018, the PG&E WE&T program has been realigned to “better serve
California’s energy workforce” into three core strategies: K-12 Career Connections (CC), Career and Workforce
Readiness (CWR), and Integrated Energy Education & Training (IEET), with the other program administrators
having similar plans.11 IEET includes two sub-components: Core Energy Education and Technical Upskill. We
focus specifically on PG&E as the Career Connections program was designated a statewide program with PG&E
as the statewide administrator in D.16-08-019.
With the passage of SB350 in 2015, Section 8 of this bill ordered the Commission to consider green workforce
training specifically calling out "workforce development and job training for residents in disadvantaged
neighborhoods, including veterans, at-risk youth, and members of the state and local community conservation
11 Pacific Gas & Electric. “Energy Efficiency Business Plan 2018-2025.” Jan 2017. Accessed May 3, 2018.
https://docs.wixstatic.com/ugd/0c9650_cbeb1d9e14cf4575845e8d5cd6bce57f.pdf
Workforce Education & Training (WE&T)
opiniondynamics.com Page 24
corps” and coordination with the “California Workforce Investment Board, the Employment Training Panel, the
California Community Colleges, and other entities to ensure a qualified, well-trained workforce is available to
implement the program requirements.”12 The California’s Workforce Education and Training Needs
Assessment (Needs Assessment) report completed in 2011 outlined two goals regarding workforce training
issues:13
◼ Energy savings, which requires that participating contractors and workers have the skills they need to
ensure that equipment is properly installed, commissioned, and maintained, and that buildings are
designed, constructed, and retrofitted consistent with best practice and technical specifications for
energy efficiency; and,
◼ Inclusion of workers from disadvantaged communities in rewarding careers in energy efficiency.
To achieve these goals, the Needs Assessment offered three recommendations for the IOUs:14
◼ Incorporate a set of contractor and workforce standards and other interventions into the program
requirements for energy efficiency incentive programs.
◼ Redesign WE&T programs to better align with and leverage California's main training and education
institutions, in order to incorporate EE-specific skills and knowledge in the broader skills set of
workers in key occupations.
◼ Create an inclusion program to broaden opportunities for workers from disadvantaged communities
to enter rewarding careers related to energy efficiency.
This new structure aims to achieve the goals outlined in SB350 and the Needs Assessment by ensuring that
current and potential members of California's energy efficiency workforce have quality education and training
at every stage of development, while also specifically targeting members of disadvantaged communities to
give them opportunities for rewarding careers in the energy efficiency industry.
1.4.1 Program Background
According to the most recent implementation plan, filed in 2013 and renewed for the 2016/17 program, the
Connections program has historically consisted of seven program components, five of which were geared
toward teachers and students of different age groups within the K-12 sector and two of which were geared
towards the college sector. In line with the planned WE&T program restructuring in 2018, the updated Career
Connections program will cater specifically to teachers and students in the K-12 sector.
The purpose of this program is to support teachers and organizations training future generations of the energy
workforce by providing teaching materials and resources to schools and teachers to educate students about
energy and sustainability fundamentals, and by providing green career awareness and exploration resources.
As such, the Career Connections program has the following overarching objectives15:
12 California State Legislature. Clean energy and pollution reduction act of 2015 (SB-350). October 2015. Accessed May 3, 2018.
https://leginfo.legislature.ca.gov/faces/billHistoryClient.xhtml?bill_id=201520160SB350 13 Donald Vial Center on Employment in the Green Economy - Institute for Research on Labor and Employment - University of
California, Berkeley. “Workforce Issues and Energy Efficiency Programs: A Plan for California’s Utilities.” May 2014. Accessed May 3,
2018. http://laborcenter.berkeley.edu/pdf/2014/WET-Plan-Executive-Summary14.pdf 14 Donald Vial Center on Employment in the Green Economy - Institute for Research on Labor and Employment - University of
California, Berkeley. “California Workforce Education and Training Needs Assessment.” 2011. Accessed May 11, 2018.
http://laborcenter.berkeley.edu/pdf/2011/WET_Part1.pdf 15 Pacific Gas & Electric. 2018-2025 PG&E Energy Efficiency Business Plan. Jan. 2017. Accessed May 3, 2018.
https://docs.wixstatic.com/ugd/0c9650_cbeb1d9e14cf4575845e8d5cd6bce57f.pdf
Workforce Education & Training (WE&T)
opiniondynamics.com Page 25
◼ Educate K-12 students on energy and sustainability fundamentals;
◼ Increase awareness of energy and sustainability career pathways for high school students and
career-seeking adults; and,
◼ Support Title 1 K-12 schools as a way of supporting disadvantaged communities.
Study Objectives
The CC program was designated as a statewide program with PG&E as the statewide administrator. Current
plans indicate that an implementer for this program will not be selected until early 2021. Thus, this study will
focus on an evaluability assessment for this program. The research objectives of the CC process evaluation
are to:
◼ Develop/Refine a CC logic model; and,
◼ Assess the evaluability and adaptability of the CC Program.
1.4.2 Overview of Evaluation Methodology
The table below identifies the methods we will utilize to meet the study objectives.
Table 8. WE&T Deliverable 28 Study Methodology by Research Objectives
Research
Objectives
Secondary Data
Review
Program
Staff
Interviews
Program Theory
and Logic
Model
Evaluability
Assessment
Develop/Refine
a CC logic
model
✓ ✓ ✓ ✓
Assess the
evaluability
and
adaptability of
the CC Program
✓ ✓ ✓ ✓
1.4.3 Evaluation Methodology: Detailed Efforts and Deliverables
We provide detailed descriptions of each task below.
Task 1. Data Requests and Secondary Data Review
Opinion Dynamics will complete a review of program administrator business plans and program materials
including Program Implementation Plans (PIPs) and past process evaluations, key overarching studies such
as the Needs Assessment, and relevant policies such as SB-350. We will submit an initial request to the
program administrator (PG&E) for relevant program materials, budgets, and participant data as available.
These documents may also include updated program performance metrics as well as associated goals,
participant tracking databases to date, and all available curriculum plans.
Reviewing these materials will allow the team to:
Workforce Education & Training (WE&T)
opiniondynamics.com Page 26
◼ Develop a base understanding of the current program design and implementation including any sub-
components and how they fit into the marketplace;
◼ Identify and develop a base understanding of current program theory and logic models; and,
◼ Identify key program management and implementation staff we want to interview in Task 2 and
develop the questions that we want to pose to them.
As necessary, we will submit a secondary data request for additional material identified as relevant to this
evaluation through Task 2.
Deliverables: Initial data request
Task 2. Program Staff Interviews
We will complete semi-structured in-depth interviews via telephone with key stakeholders from the
Commission, the program administrator (PG&E), program implementers, and any other key design and
implementation staff identified in Task 1. We will use these interviews to build our understanding of the
planned design and implementation structure of the CC program and performance metrics beyond what can
be gleaned from available program documentation. Specifically, we will focus on the following topics:
◼ Understanding the planned program design and implementation process, changes planned, updates
to proposed program metrics, associated goals and budgets;
◼ Understand alignment between the CC program and other WE&T programs;
◼ Alignment between the program and Strategic Plan goals;
◼ Characterize workforce outcomes; and,
◼ Ensure that we identify all relevant documentation related to the Career Connections programs for
review.
Deliverables: Draft and final interview guides
Task 3: Program Theory and Logic Model
Opinion Dynamics will develop/refine a PTLM that accurately documents the program design, delivery, and
theory of change for this iteration of the CC program. A PTLM captures the program’s intended outcomes and
links those outcomes to specific metrics and activities, mapping the causal relationships between the different
program components. As such, our team feels that this is a critical step in an initial evaluation; one that will
create a foundation for program planning, service delivery, and future evaluations.
We understand that PTLMs are complicated and representing the Program’s components accurately and
thoroughly may take several different iterations. Our team will draw upon information gleaned from interviews
with program staff and the review of secondary data (as outlined in Task 1) to complete the initial logic model.
Our team will then present the draft logic model to the program team and revise the PTLM as necessary to
ensure that the final logic model accurately captures the CC Program’s design and theory of change. We will
continue to work with the program team until all agree that the PTLM is sufficiently detailed, and will therefore
be useful, before finalizing.
Deliverables: Draft and final program theory and logic models
Workforce Education & Training (WE&T)
opiniondynamics.com Page 27
Task 4: Evaluability Assessment
As part of the evaluation of the CC Program, Opinion Dynamics will complete an evaluability assessment to
help the Commission and the program team ensure that the CC Program addresses the needs of California’s
disadvantaged workers and make progress toward the Program’s intended outcomes.
Our evaluability assessment will cover three main topics: (1) assess if there is an adequate program theory
that provides plausible causal links toward program goals; (2) consider if the program theory has been planned
appropriately (e.g., if the program is properly staffed and being implemented as intended); and (3) determine
if the planned program tracking database will collect adequate data to support the causal connections and
outcomes. To complete this EA, we will draw upon information gleaned from previous research tasks to
determine if established metrics align with program goals, and if those metrics are measurable. During
interviews with program staff, our team will also assess how data are currently collected and recommend ways
that the program team may be able to augment data collection in the future. Additionally, through this task,
our team will recommend that the program capture any additional data necessary to adequately evaluate the
CC Program in the future. The goal of this exercise is to assist the Commission with building an evidence-base
to support future evaluations and to ensure that the CC Program is able to show progress toward program
goals.
Task 5. Reporting
Opinion Dynamics will work with the Commission to develop draft and final reports in accordance with CPUC
specifications.
Deliverable: Draft and final reports
1.4.4 Study Timeline and Milestones
The table below summarizes the evaluation timeline and deliverables for the Career Connections evaluation.
Table 9. Deliverable 28 Timeline
Task 2020 2021
Q4 Q1 Q2 Q3 Q4
Data Requests and Secondary Data Review x
Program Staff Interviews x
Program Theory and Logic Model x x
Evaluability assessment x x
Reporting x x
Table 10 presents the budget by milestone for this study.
Workforce Education & Training (WE&T)
opiniondynamics.com Page 28
Table 10. Deliverable 28 Milestones and Budget
Milestone Deliverable Budget Schedule
Milestone 1: Evaluability
Assessment Report Evaluability Assessment Report October 2021
Total $50,000 October 2021
1.5 Deliverable 29 (WE&T-4): Career and Workforce Readiness
Process Evaluation
The Career Workforce Readiness (CWR) Program is new to the WE&T portfolio of programs. The program will
be implemented primarily through partnerships with different community-based organizations (CBOs),
including building-related job training programs and workforce development agencies. The Career and
Workforce Readiness program was designated as a statewide program with PG&E as the statewide
administrator. Current plans indicate that an implementer for this program will not be selected until early
2021. Thus, this study will focus on an evaluability assessment for this program.
The new CWR sub-program was created in response to SB350, passed in 2015. SB350 states that, in addition
to establishing a regulatory framework and a comprehensive set of programs to achieve greater energy savings
for California residents, the Commission shall promote greater penetration of programs within disadvantaged
communities, including programs aimed at workforce development. In addition, SB350 specifically notes that
the Commission should work with existing workforce development organizations that serve disadvantaged
communities.16 Working in concert with the existing WE&T and workforce development programs, the central
purpose of the CWR program is to support members of the energy workforce through training-related
collaborations with workforce development organizations whose primary mission includes serving
disadvantaged communities. The CWR program is a statewide WE&T program.
The CWR Program aims to integrate existing workforce development organizations’ resources—case
management, soft skills training, job placement, etc.—with technical energy education and resources. Its
central objectives are as follows:17
◼ Support the integration of energy efficiency content into existing workforce development training
programs whose primary audience is disadvantaged communities and workers.
◼ Support trainers of programs supporting disadvantaged workers and communities in support of
overarching energy efficiency portfolio objectives.
◼ Provide energy and green career awareness resources to workforce development organizations
serving disadvantaged workers and communities.
◼ Provide energy and resource conservation education materials and resources to organizations who
train disadvantaged workers.
The CWR Program is intended to facilitate entry from disadvantaged communities into the energy efficiency
career pathway. As such, its funding cannot be used to fund energy efficiency education and training activities
that are already occurring, or to fund activities that workforce development agencies and organizations are
16 California State Legislature. Clean energy and pollution reduction act of 2015 (SB-350). October 2015. Accessed May 3, 2018.
https://leginfo.legislature.ca.gov/faces/billHistoryClient.xhtml?bill_id=201520160SB350 17 Pacific Gas & Electric. 2018-2025 PG&E Energy Efficiency Business Plan. January 17, 2017. Accessed May 3, 2018.
https://docs.wixstatic.com/ugd/0c9650_cbeb1d9e14cf4575845e8d5cd6bce57f.pdf
Workforce Education & Training (WE&T)
opiniondynamics.com Page 29
funded to do (i.e., case management, job placement, etc.). It will fund the addition and integration of energy
efficiency funding to some existing efforts.
As the impetus for the CWR Program is to support disadvantaged workers, coming to some agreement on a
definition for this core component of the program’s target audience was of particular importance. According
to California’s Health and Safety Code, the California State Legislature broadly defines “disadvantaged
communities” as those that live within areas that meet the following criteria:18
◼ Areas disproportionately affected by environmental pollution and other hazards that can lead to
negative public health effects, exposure, or environmental degradation.
◼ Areas with concentrations of people that are of low income, high unemployment, low levels of
homeownership, high rent burden, sensitive populations, or low levels of educational attainment.
For the purposes of developing programs to support SB350, however, the IOUs built on this definition to
specifically describe the CWR Program’s target audience.
In late 2015, with input from stakeholders, the IOUs arrived at a definition of a disadvantaged worker as an
individual who meets at least one of the following three criteria: 1) lives in a high unemployment zip code
where unemployment rate is at least 150% of the median unemployment rate for the county or for the state;
or 2) lives in a low income zip code where the average household income is 50% below Area Median Income
(AMI); or 3) has a referral from a collaborating community-based organization (CBO), state agency, or workforce
investment board.19
1.5.1 Study Objectives
The research objectives of the CWR process evaluation are to:
◼ Develop a CWR logic model; and,
◼ Assess the evaluability of the CWR Program.
1.5.2 Overview of Evaluation Methodology
Table 11 below provides a high-level overview of Opinion Dynamics’ suggested research approach and how
each activity aligns with the study objectives outlined above.
Table 11. WE&T Deliverable 29 Study Methodology by Research Objectives
Research Objectives Secondary
Data Review
Program Staff
Interviews
Program
Theory and
Logic Model
Evaluability
Assessment
Develop a CWR logic model
✓ ✓ ✓ ✓
Assess the Evaluability ✓ ✓ ✓ ✓
1.5.3 Evaluation Methodology: Detailed Efforts and Deliverables
18 California Health and Safety Code. Chapter 4.1. Section 39711 (a) 19 Pacific Gas and Electric Company. “Energy Efficiency Business Plan 2018-2025”. 2017. Accessed May 6, 2018.
https://docs.wixstatic.com/ugd/0c9650_cbeb1d9e14cf4575845e8d5cd6bce57f.pdf
Workforce Education & Training (WE&T)
opiniondynamics.com Page 30
In the remainder of this section, we describe each of our suggested research tasks in detail. For each, we
discuss our approach, how we envision each task addressing the research objectives outlined previously, and
any deliverables associated with that task.
Task 1. Review of Program Materials, Tracking Data, and Secondary Data
To begin, Opinion Dynamics will complete a review of secondary materials, including any existing program
materials, policy documents, business plans, program tracking data, and any other related research studies.
We will also gather any relevant secondary data that are publicly available.
Through this task, the research team will develop a base understanding of the current program design,
program goals, intended outcomes, and identify key program staff that we will interview during Task 2.
Additionally, we will use this task to gather additional policy context that inform the desired outcomes.
Materials reviewed as part of this task will include, but are not limited to:
◼ Program Implementation Plan and any other program manuals
◼ Final Business Plan
◼ Senate Bill 350
◼ Survey of Inclusion Workforce Landscape
◼ The California Long-term Energy Efficiency Strategic Plan
◼ Other regulatory materials or studies related to WE&T and disadvantaged communities
Deliverables: Initial data request
Task 2: In-Depth Interviews with Program Staff
We will complete semi-structured in-depth interviews with key program design and program management staff
identified in Task 1 via telephone. We will use these interviews to build our understanding of the theories of
change, the planned design and implementation structure of the CWR Program, and performance metrics
beyond what can be gleaned from available program documentation. We will also use this interview as the
primary data source in developing the CWR program theory and logic model (PTLM). Specifically, we will focus
on:
◼ Understanding the planned program design and implementation process, along with the initial
program goals;
◼ Accurately capturing the CWR Program’s intended outputs, outcomes and theory of change;
◼ Gathering information about synergies between the CWR Program and other WE&T programs;
◼ Alignment between the program and Strategic Plan goals, and best practices outlined in the Survey
of Inclusion Workforce Landscape; and
◼ Ensuring that we identify all relevant documentation related to the CWR Program for review.
Deliverables: Draft and Final Interview guides
Workforce Education & Training (WE&T)
opiniondynamics.com Page 31
Task 3: Program Theory and Logic Model
As this is the first process evaluation of the CWR Program, Opinion Dynamics will develop a PTLM that
accurately documents the program design, delivery, and theory of change. A PTLM captures the program’s
intended outcomes and links those outcomes to specific metrics and activities, mapping the causal
relationships between the different program components. As such, our team feels that this is a critical step in
an initial evaluation; one that will create a foundation for program planning, service delivery, and future
evaluations.
We understand that PTLMs are complicated and representing the Program’s components accurately and
thoroughly may take several different iterations. Our team will draw upon information gleaned from interviews
with program staff and the review of secondary data (as outlined in Task 1) to complete the initial logic model.
Our team will then present the draft logic model to the program team and revise the PTLM as necessary to
ensure that the final logic model accurately captures the CWR Program’s design and theory of change. We will
continue to work with the program team until all agree that the PTLM is sufficiently detailed, and will therefore
be useful, before finalizing.
Deliverables: Draft and final program theory and logic models
Task 4: Evaluability Assessment
As part of the first evaluation of the CWR Program, Opinion Dynamics will complete an evaluability assessment
(EA) to help the Commission and the program team ensure that the CWR Program addresses the needs of
California’s disadvantaged workers and make progress toward the Program’s intended outcomes.
Our evaluability assessment will cover three main topics: (1) assess if there is an adequate program theory
that provides plausible causal links toward program goals; (2) consider if the program theory has been planned
appropriately (e.g., if the program is properly staffed and being implemented as intended); and (3) determine
if is the planned program tracking database will collect adequate data to support the causal connections and
outcomes. To complete this EA, we will draw upon information gleaned from previous research tasks to
determine if established metrics align with program goals, and if those metrics are measurable. During
interviews with program staff, our team will also assess how data are currently collected and recommend ways
that the program team may be able to augment data collection in the future. Additionally, through this task
our team will recommend that the program capture any additional data necessary to adequately evaluate the
CWR Program in the future. The goal of this exercise is to assist the Commission with building an evidence-
base to support future evaluations and to ensure that the CWR Program is able to show progress toward
program goals.
Task 5. Reporting
The Opinion Dynamics Team will work with the Commission to develop draft and final reports in accordance
to the specifications discussed in this proposal.
Deliverable: Draft and Final Reports
1.5.4 Study Timeline and Milestones
The table below summarizes the evaluation timeline and deliverables for the Career and Workforce Readiness
process evaluation.
Workforce Education & Training (WE&T)
opiniondynamics.com Page 32
Table 12. Deliverable 29 Timeline
Task 2020 2021
Q4 Q1 Q2 Q3 Q4
Review of Program Materials, Tracking Data, and
Secondary Data x
Interviews with Program Staff x
Program Theory and Logic Model x x
Evaluability Assessment x x
Reporting x x
Table 13 presents the budget by milestone for this study.
Table 13. Deliverable 29 Milestones and Budget
Milestone Deliverable Budget Schedule
Milestone 1: Evaluability
Assessment Report Evaluability Assessment Report October 2021
Total $50,000 October 2021
1.6 Deliverable 30 (WE&T-5): Knowledge, Skills, and Abilities Market
Studies
The CALTEESP sets forth a goal of establishing "energy efficiency education and training at all levels of
California's educational system." As codified in their business plans, program administrators continue to drive
toward the CALTEESP vision for utility WE&T programs--"By 2020, California's workforce is trained and fully
engaged to provide the human capital necessary to achieve California's economic energy efficiency and
demand-side management potential.”20 WE&T is a key cross-cutting strategy for providing the knowledge to
recognize EE opportunities and the skills and abilities to take action on those opportunities. California has
passed ambitious energy efficiency and greenhouse gas legislation with SB350 requiring California to double
energy efficiency savings and reduce greenhouse gas levels 40% below 1990 levels by 2030. Similar
statewide mandates include Assembly Bill 758 and Assembly Bill 802, which both address energy efficiency
performance and reporting for buildings. These statewide mandates create a demand for new and more
precise energy efficiency and whole building skillsets in the California workforce.
Numerous studies have identified a critical gap between ex ante and ex post energy savings estimates. Many
of these studies identify poor quality work as a key contributor to this gap, especially in the areas of HVAC
installation and maintenance, advanced lighting control systems, and weatherization services. In the
Workforce Issues and Energy Efficiency Programs: A Plan for California Utilities report, the Don Vial Center
states that quality of work is an important, under recognized factor for "meeting short-term savings targets
and long-term energy efficiency goals." The report goes on to say that "persistent quality problems in Heating,
20 California Public Utilities Commission. “California Long Term Energy Efficiency Strategic Plan” Sept. 2008, updated Jan. 2011.
Accessed May 8, 2017. http://www.leginfo.ca.gov/pub/15-16/bill/sen/sb_0301-0350/sb_350_bill_20150911_enrolled.pdf
Workforce Education & Training (WE&T)
opiniondynamics.com Page 33
Ventilation, and Air Conditioning (and Refrigeration) (HVAC) and other key building systems point to a need for
greater efforts to ensure quality. Despite the common use of upfront workforce standards in other industries
like health care, there is a notable lack of substantive qualifications required of contractors and workers who
participate in most IOU programs. Reliance on upfront standards for participating contractors and workers can
be particularly effective at promoting work quality because they attract high-performing contractors and screen
out (or improve the quality of work of) lower performing contractors."21
In the recent Responsible Contractor Policy for EE Programs: Market Intelligence Study,22 nearly all of the 78
stakeholders Opinion Dynamics interviewed discussed the topic of training and the need for consistent job
definitions, skill requirements, and training standards to support energy and demand reduction goals.
Concerns centered on defining how jobs are defined and what key knowledge, skills, and abilities (KSAs) are
needed to perform a particular task. Who defines these KSAs? How do we ensure KSAs are kept current with
quickly changing technology? How do you keep knowledge and content up-to-date when faced with contractors
who are providing on-the-job training and instructors who are teaching courses for equipment that did not exist
when they were certified? How do you ensure that training includes skills that are relevant to energy efficiency
when energy efficiency is not necessarily a focus of credentialing organizations?
Reviewing program theory and logic models across resource programs finds that an implicit assumption of
most programs is that contractors and technicians--with experience--have the right KSAs to install measures
in a way that will realize and ideally maximize energy savings. Programs and WE&T efforts utilize training to
support this assumption; however, the key challenge is defining what the right KSAs are. In programs where
these training needs are incorporated into the PTLM, program staff often consult relevant industries for
guidance. This is a challenge as many industries do not have standard career paths and competency models,
nor does their training adequately incorporate energy efficiency components.
Let's briefly look at a specific example in the HVAC arena. The CALTEESP specifies the following HVAC goal:
"Quality HVAC installation and maintenance become the norm. The marketplace understands and values the
performance benefits of quality installation and maintenance." Unfortunately, the CALTEESP does not define
the terms "quality installation” or “quality maintenance." Stakeholders involved in developing the CALTEESP,
indicate that “quality” was defined in development discussions as adherence to industry standards, such as
the ACCA/ANSI Quality Installation Specification 5 (ACCA 5) and ASHRAE/ACCA/ANSI Standard 180.
Unfortunately, that detail was not codified in the CALTEESP. The HVAC Action Plan, which was designed to help
the HVAC sector achieve the CALTEESP goals, identifies training as a critical success factor, stating that
"California must set standards for quality work." This document mentions industry standards and the need to
align them with Title 24, but still does not define the terms “quality installation” and “quality maintenance”.
Given that many program administrators took part in the development processes for both the CALTEESP and
the HVAC Action Plan, many took relevant industry standards as the definition of quality installation and quality
maintenance and used them to design HVAC programs. For example, the HVAC Optimization program was
based on ASHRAE's Standard 180 Standard Practice for Inspection and Maintenance of Commercial Building
HVAC Systems. The Residential Quality Installation program was based on ACCA 5. These standards provided
an important framework for program design and implementation. However, standards traditionally define what
tasks to perform and sometimes what order to perform these tasks in, but they do not define how to perform
the tasks nor what tools to use to perform the task. Given that quality installation and maintenance is
21 Don Vial Center, “Workforce Issues and Energy Efficiency Programs: A Plan for California’s Utilities.” May 2014. Accessed May 8,
2017. http://laborcenter.berkeley.edu/pdf/2014/WET-Plan14.pdf
22 Opinion Dynamics, “Responsible Contractor Policy for EE Programs: Market Intelligence Study.” Accessed December 31, 2017.
http://www.calmac.org/publications/Responsible%5FContractor%5FPolicy%5FStudy%5FReport%5FFINAL%2Epdf
Workforce Education & Training (WE&T)
opiniondynamics.com Page 34
considered a highly technical activity in which improper execution of the necessary steps can lead to incorrect
diagnoses of problems as well as solutions that can decrease efficiency, these industry standards do not
necessarily lead to energy savings in isolation. The Western HVAC Performance Alliance (WHPA) has
recognized this gap and has worked to develop “User Guides” to operationalize the “how” of these standards.
Given that this is a significant undertaking, however, these passionate volunteers have only made moderate
progress.
In HVAC, this lack of clarity on what knowledge, skills, and abilities are needed to perform standards-based
work has led to challenges with conducting evaluations. One of the challenges with past HVAC impact
evaluations is a discrepancy between the evaluator’s perspective on how to conduct key maintenance tasks
and the procedures the program was teaching its contractors. This lack of consistency has recently led to
controversy regarding ex post savings estimates and evaluation recommendations, such as with Work Order
32.
The lack of clarity on workforce standards and identified KSAs both at the industry level, as well as at the
whole building design level will likely continue to undermine the effectiveness of investments in programs and
training. With all of this in mind, Deliverable 30 is crucial to advancing California’s ambitious energy goals, and
to putting a stake into the ground that all stakeholders can work toward.
1.6.1 Study Objectives
Jobs can be best understood as a series of tasks and responsibilities that an employee conduct. A task is an
action designed to contribute to a specified end result. KSAs are what enable employees to perform these
tasks. Often, these terms are used interchangeably; yet they represent distinctly different dimensions of a job.
Here we define these terms in context of this study:
◼ Knowledge is the body of information needed to perform a task. It focuses on the actual understanding
of particular concepts. It is theoretical and not practical. A person can read a state’s drivers manual
on how to drive a car, but have no practical experience driving a car.
◼ Skills reflect the practical application of the theoretical knowledge. They reflect capabilities or
proficiencies developed through training or actual experience. Once a person passes their Department
of Motor Vehicle’s written test, they obtain a learner’s permit to develop the proficiency of driving a car
prior to taking the driving test.
◼ Abilities are underlying, enduring traits useful for performing tasks. They are often related to personal
and social attributes that tend to be innate. There is a fine line between skills and abilities. Driving a
car would require the ability to safely perform a range of physical activity to manipulate the controls
for starting, stopping, and backing up.
The objective of this research is to identify the KSAs needed for implementers, contractors, technicians, and
contractor laborers on energy efficiency and whole building design concepts within specific industries.
1.6.2 Overview of Evaluation Methodology
We will conduct a Job Task Analysis to meet the objectives of this study. Job Task Analysis is the systematic
investigation of work activities, worker attributes, and work context. This study is particularly focused on
understanding what tasks (work activities) and what knowledge, skills, and abilities (worker attributes) are
Workforce Education & Training (WE&T)
opiniondynamics.com Page 35
essential to energy efficiency for implementers, contractors, technicians, and contractor laborers in specific,
high-impact industries. Work context, defined as situational opportunities and constraints that affect the
occurrence of behavior (i.e. time allowed to conduct maintenance on an HVAC system is a work context
attribute that affects what maintenance tasks are completed and with what thoroughness), is not a focus of
this research.
Significant research on job analysis has taken place in the past 75 years. There are many methods for
conducting job analyses. Some of the most common include observations, individual interviews, facilitated
meetings, and questionnaires. Time requirements, cost, and intended use of information must be considered
when choosing job analysis methods.
Given the timeframe for this study, and the goal of identifying work activities and worker attributes, Opinion
Dynamics has selected the DACUM (Developing a Curriculum)23 Job Task Analysis method as the foundation
for this research. We have selected this method as it is a time-tested approach that is effective, comparatively
inexpensive and relatively fast. This method also lends itself to conducting a job analysis to a specific portion
of a total job—in this case those portions involving energy efficiency and whole building principles. DACUM is
based on three premises:24
1. Expert workers can describe and define their job more accurately than anyone else. While some methods
suggest using supervisors, managers, or content experts, this method purports that the people working
full-time in their positions are the real experts on the job.
2. An effective way to define a job is to precisely describe the tasks the expert workers perform.
Understanding what expert workers do to be successful at is key to understanding a job and how to train
others for that job.
3. All tasks, in order to be performed correctly, demand the use of certain knowledge, skills, and abilities.
Knowledge, skills, and abilities are the enablers that make it possible for workers to be successful.
For each identified key area of study and job role (e.g. HVAC contractors, electricians), we will follow the
following process outlined in Figure 3. We have budgeted to conduct two job role analyses.
Figure 3. DACUM Job Task Analysis Process
23 The term DACUM is a misnomer. While initially developed as a method to conduct needs assessment to inform curriculum
development, it has been used for over 40 years to conduct job analyses for other purposes such as defining job descriptions,
developing HR strategies, making compensation decisions, creating selection assessments, meeting ISO 9000 requirements, and
developing career paths. It is used in schools, colleges, government agencies, military organizations, business and industry. 24 Wojton, M., Heimlich, J. “Professional Learning Framework: DACUM Panels.” 2016. Accessed May 12, 2018.
http://www.islframework.org/wp-content/uploads/2017/07/For-Posting-DACUM-Panel-Feedback-Report.pdf
Workforce Education & Training (WE&T)
opiniondynamics.com Page 36
1.6.3 Evaluation Methodology: Detailed Efforts and Deliverables
Task 1: Identify Key Areas of Study
Opinion Dynamics will coordinate with the Energy Division Project Manager for WE&T to select two high impact
areas, for which we will identify necessary knowledge, skills, and abilities (KSA) for contractors and
technicians/contract laborers. We will also include implementers who act as contractors/technicians as
participants in the study process. Opinion Dynamics will base its recommendations on past evaluation
guidance, policy direction, program administrator business plans, labor market data, and expert opinion.
Potential areas of study include KSAs on HVAC installation and maintenance, heat pumps, home area
networks (HAN) or smart technology, with a key focus on how each of these are incorporated into efficient
whole building design. Given the differing KSAs required for residential vs commercial equipment we will also
work with the Energy Division Project Manager to focus the scope of each study area on one sector or the
other.
Task 2: Conduct Pre-Inventory Data Analysis
We will conduct a pre-inventory data analysis to gain an understanding of existing information and data
sources relevant to the selected areas of study. This analysis will help ensure that we:
◼ Use appropriate and familiar language in the development of workshop stimuli and homework;
◼ Prioritize key functions within a job that focus on energy efficiency;
◼ Align this study with existing industry-specific job frameworks to increase industry buy-in; and,
◼ Maximize the return on research investment.
We will also use this secondary analysis to understand the complexity of the occupations under study to further
refine the measures, applications, and careers to be examined within the project scope and to inform our
sample design. For example, there are many workers employed in direct HVAC positions, while others are
scattered among thousands of workers in Sheet Metal, Pipes, and Stationary Engineering trades. While the
KSAs are likely similar across these trades, we will want to include a representative from these trades in our
expert panels to ensure we capture any key nuances.
Drawing on our extensive experience, we will utilize known sources of data as well as conduct a brief literature
review to ensure our analysis includes all key information. We will assess each source to ensure the
information is up to date. We have also budgeted for eight in-depth interviews to collect additional data and
context should the need arise.
Known Sources
We will review job information from labor market sources such as the Occupational Information Network
(O*NET). The O*NET database—sponsored by the U.S. Department of Labor--contains hundreds of
standardized and occupation-specific descriptors on almost 1,000 occupations covering the entire U.S.
economy. It encompasses the broadest scope of work information ranging from labor market data and wages
to knowledge, skills and required tasks. The O*NET content model is the foundation of the system that
identifies the most important types of information about work. Figure 4 displays O*NET’s content model, with
the four elements we will emphasize in this study highlighted in blue.
Workforce Education & Training (WE&T)
opiniondynamics.com Page 37
Figure 4. Content model for O*NET
For example, O*NET includes the job--Heating and Air Conditioning Mechanics and Installers—and identifies
the information for each of the elements in the above content model. It also identifies 37 alternate occupation
titles.
We will also consult the Department of Energy and the National Renewable Energy Laboratory’s Guidelines for
Home Energy Professionals Project. This project conducted job task analyses, defined standard work
specifications that address minimum acceptable outcomes, developed accredited training programs, and
created Home Energy Professional certifications. These guidelines are currently utilized by the DOE’s
Weatherization program (WAP) and potentially include roles that could be considered in this study. Through
Opinion Dynamics’ in-depth assessment of this work for the Responsible Contractor Policy Market Intelligence
study, we are able to leverage the information as well as the lessons learned from their process. Such lessons
include the fact that workers need both requisite specialized task level skill expectations and broad training
that allows workers to take a holistic view of the systems they work with, enabling them to make situational,
systems-based decisions. We will consult the Bureau of Labor Statistics and the California Labor Market
Information Division for additional labor market and economic data as well as Opinion Dynamics’ PY2013-
2014 California Statewide Workforce Education and Training Program Contractor Training Market
Characterization
In addition to national and state-based labor market resources, we will consult specific work on the identified
areas of study. For example, for HVAC, we will review key standards such as ACCA Standard 5 and
ACCA/ASHRAE Standard 180. We will also consult the Knowledge Areas of Technician Expertise (KATEs)
identified by North American Technician Excellence (NATE), as well as the competency and task lists developed
by HVAC Excellence. We will utilize the compendium of work developed by the Western HVAC Performance
Alliance with a specific focus on the whole building career lattice work and the WE&T Gaps Reports.
Workforce Education & Training (WE&T)
opiniondynamics.com Page 38
Additional Sources
Opinion Dynamics will conduct a brief literature review to understand any additional research that can expand
on what we already know. We will conduct a systematic search of the available literature regarding the areas
of study, including conference proceedings, regional report databases, Google Scholar, and academic
databases. We will use a set of key words to search each database in a consistent manner. At the outset of
this effort, we will define the criteria for inclusion in the literature review, which will ensure that the literature
review process is efficient, focused, and relevant. Opinion Dynamics will assess the results of keyword
searches with respect to relevance based on abstracts or executive summaries. Papers or reports deemed
applicable to the topic at hand will then be selected for a full text review.
Opinion Dynamics will conduct a thematic analysis of each of the identified data sources above using NVIVO—
a powerful qualitative analysis tool—to uncover evidence-based insights and findings. Using NVIVO, we will
mine the data fully and use the information to develop the pre-work for the participants in each DACUM
workshop.
Deliverables: Draft and final Pre-Work
Task 3: Workshop Sample Strategy and Recruitment
Recruiting the right participants is the foundation for an effective DACUM job analysis. For each job focus
within each area of study, we will identify ten to twelve participants, with the goal of the final group consisting
of seven to ten participants so that we have a sufficient size to function effectively. Since we will be asking
them for a thirty-minute phone call, completion of a homework assignment and two days of their time, we will
offer participants a $1000 stipend as a thank you for participating in the study. The DACUM workshop cannot
succeed without the help of expert workers because they can describe their daily performance in terms of
activities and tasks they perform better than anyone else. Since we will be focusing on the energy efficiency
and building performance elements of each job, we will likely recruit past program participants in each key
area of study. The following selection criteria will guide our sampling strategy:
1. Technical Competence: Individuals selected should be highly skilled and aware of current
developments and needs in their industry.
2. Energy Efficiency/Whole Building Experience: Individuals need to be experienced with energy
efficiency and whole building principles as this study will focus on those aspects of the job.
3. Effective Communicator: Individuals selected must be able to explain what they do in a precise
and accurate way. Since the workshop involves brainstorming and consensus seeking, individuals
must be able to actively listen and participate in small group discussions.
4. Commitment: Individuals must commit to the process and actively participate.
5. Freedom from Bias: Individuals should be open-minded and free of bias.
We aim to balance screener length with researcher objectives to ensure respondent burden is minimized—
placing major qualifiers and disqualifiers (except for more sensitive demographic questions) at the beginning
of the screener. Prior to commencing the recruiting effort, the contractor will develop a draft recruiting guide
for all identified targets, which we will finalize based on comments from the Energy Division Project Manager
for WE&T.
Deliverables: Draft and final sampling strategy memo, draft and final screener
Workforce Education & Training (WE&T)
opiniondynamics.com Page 39
Task 4: Design and Facilitate DACUM Workshop
Once we have recruited our participants, the facilitators will reach out to participants via telephone to establish
rapport and orient them to the DACUM process, talk them through what to expect, discuss our expectations of
them, review logistics, discuss the pre-work, and answer any questions. We anticipate this discussion to take
approximately 30 minutes. We have found that this initial phone call is a key step to ensuring the workshop
goes well.
Following this phone call, we will send the participants the pre-work materials. We have found that in order to
get the most out of the DACUM workshop, providing a pre-work exercise is a best practice. This will be a short
exercise asking them to review the output of the task and KSAs from the pre-inventory analysis and
determining if the task belongs to that job, is an energy efficiency or whole building opportunity, and if the
KSAs are appropriately defined. Through this pre-work, participants are given the opportunity to reflect on the
job under study, which often adds depth and dimension to the discussions during the in-person workshop. All
pre-work exercises will be finalized based on comments from the Energy Division Project Manager for WE&T.
We will facilitate two DACUM workshops and will utilize either a Commission or Energy Center meeting room
to save rental costs. The DACUM workshop is a two-day in-person exercise in which we identify general areas
of job responsibilities, identify specific tasks performed related to that job responsibility, and develop a list of
KSAs necessary to conduct the specific tasks. For example, a general area of job responsibility might be
“maintain automobile” and the tasks that make up that responsibility would include “change motor oil,”
“change windshield wipers,” and “fill gas tank.” We will design key questions and activities to solicit this
information focusing on:
◼ Problems that a worker must solve
◼ Products or services that a worker produces
◼ Decisions that a worker makes
◼ Equipment that a worker operates
◼ Information that a worker collects, analyzes, or prepares
The facilitator's role is to set the stage, develop an energetic and comfortable environment, orient the
committee to the process, to guide them through the analysis, to draw out ideas, to question each proposed
task statement, to gain consensus from the group on each item, and to keep the discussion on target and the
process on schedule.
During the session, the high-performers brainstorm and discuss their work, coming to consensus about areas
of responsibility and required tasks. The facilitator actively assists the panel members by providing guidance
in the formulation of tasks statements, ensuring equal participation by all members, and coaching them to
describe their work in measurable and observable terms. We will utilize a storyboarding process, where
participants are encouraged to develop activity and task statements that include a verb, modifier, and a noun.
The storyboard cards are temporarily attached to the wall or board to allow for rapid changes, sequencing, and
modification. The storyboard process is very fluid, flexible and visual—all of which helps maintain the group's
focus on describing their job duties and tasks (see Figure 5). Important to the moderation is ensuring that task
statements meet key criteria such as having the right verbs, objects, and appropriate qualifiers. For example,
while the task "mow the lawn" is quite clear, the task statements "develop a plan" and "assess performance"
are not. What type of plan is needed, who are we to assess, and in what area? A committee often has to be
asked repeatedly for one or more words which help make the task statement "crystal clear."
Workforce Education & Training (WE&T)
opiniondynamics.com Page 40
Figure 5. DACUM Storyboarding Process
Source: Saskatchewan Institute of Applied Science and Technology
The facilitators are aware of the common pitfalls associated with DACUM workshops, which are (1) the
unsuitable composition of “expert worker” committees, (2) lack of definitions or criteria for job tasks and task
statements, (3) heavy dependence on outdated literature reviews, (4) lack of high-quality tasks statements,
and (5) the all-too-common failure to separate out tasks, knowledge, skills, and abilities. The facilitators will
utilize effective orientation materials, handouts, examples and moderation techniques to ensure these
hazards are avoided.
We will end the workshop with an evaluation of the workshop proceedings so that we can continuously improve
on the workshop design and facilitation iteratively.
Deliverables: Draft and final pre-work exercises; Draft and final DACUM process and facilitation guide; Draft
and final DACUM workshop evaluation; Draft activities, tasks and KSAs
Task 5: Verification
Following the workshop, we will solicit feedback through an online survey to validate the activities, tasks, and
KSAs identified through the workshops. The objective of the verification process is to ensure that we covered
all key job responsibilities, associated the right tasks with these responsibilities, and identified the correct
knowledge, skills and abilities need to perform these tasks, and to ensure that the results are appropriate for
a wide audience. We will advertise these surveys using online forums as well as target installers for which we
have email addresses.
Deliverables: Verified activities, tasks, and KSAs
Task 6: Reporting
Opinion Dynamics will work with the Commission to develop draft and final reports in accordance with CPUC
specifications.
Deliverable: Draft and final reports
1.6.4 Study Timeline and Milestones
In Table 14 we identify the timeline for the study.
Workforce Education & Training (WE&T)
opiniondynamics.com Page 41
Table 14. WE&T 30 Study Timeline
Q4
2018
Q1
2019
Q2
2019
Q3
2019
Q4
2019
Q1
2020
Identify key areas of study x x
Pre-Inventory Data Analysis X X
DACUM Workshop X X
Verification x x
Reporting x x
Table 15 presents the budget by milestone for this study.
Table 15. Deliverable 30 Milestones and Budget
Milestone Deliverable Budget Schedule
M1: DACUM Workshop
Design
Final DACUM process and facilitation
guide May 2019
M2: Final Report Report January 2020
Total $345,000 January 2020