Effective Feedback for Performance Evaluation 1 Rutherford, Naylor, & Wheeler
NLC Instructor: Instructional Design
Kara Rutherford, Beverly Naylor, Tammy Wheeler
Effective Feedback for Performance Evaluation 2 Rutherford, Naylor, & Wheeler
TableofContentsProject Description .................................................................................................................... 4
Project title ................................................................................................................................ 4Information about the organization ........................................................................................... 4Information about the site client ................................................................................................ 5Information about the project .................................................................................................... 5Team members ......................................................................................................................... 7Project schedule ....................................................................................................................... 7
Performance Analysis ................................................................................................................ 9Gap Analysis ............................................................................................................................. 9Organizational Analysis .......................................................................................................... 10Summary: State the Case for Training ................................................................................... 18
Task Analysis ........................................................................................................................... 20Data Sources .......................................................................................................................... 20Completed Task Analysis ....................................................................................................... 22(3-5 things) of What Is Critical, Difficult, and Complex ........................................................... 34
Learner Analysis ...................................................................................................................... 35Brief Description of the Learners ......................................................................................... 35Sources and Methods ......................................................................................................... 35Data-Gathering Instruments ................................................................................................ 36
Findings and Implications ....................................................................................................... 37Relevant Skills ..................................................................................................................... 37Relevant Contextual Factors ............................................................................................... 40
Job-Focused objectives .......................................................................................................... 44Critical Tasks that the Training Will Address .......................................................................... 44Job-Focused Instructional Objectives ..................................................................................... 45
Performance Assessment Instrument .................................................................................... 47Objectives ............................................................................................................................... 47Type of performance assessment and rationale ..................................................................... 48Assessment instrument .......................................................................................................... 48Mastery level ........................................................................................................................... 49Job Aids .................................................................................................................................. 49Performance Assessment ....................................................................................................... 49
SET UP ................................................................................................................................. 49
Effective Feedback for Performance Evaluation 3 Rutherford, Naylor, & Wheeler
INSTRUCTIONS AND SCENARIOS ............................................................................................ 50ASSESSMENT CHECKLIST ...................................................................................................... 52
Instructional Plan Worksheet .................................................................................................. 55Team Project Status Review # 1 ............................................................................................. 64
Overall Project Status in Relation to Schedule .................................................................. 64Actual or Potential Roadblocks ........................................................................................... 64Team Work Successes ......................................................................................................... 65Team Work Improvements ................................................................................................... 65
NLC Instructor Team Project Status Review # 2 ................................................................... 66Overall Project Status in Relation to Schedule .................................................................. 66Actual or Potential Roadblocks ........................................................................................... 66Team Work Successes ......................................................................................................... 66Team Work Improvements ................................................................................................... 67
NLC Instructor Team Project Status Review # 3 ................................................................... 68Overall Project Status in Relation to Schedule .................................................................. 68Actual or Potential Roadblocks ........................................................................................... 68Team Work Successes ......................................................................................................... 69Team Work Improvements ................................................................................................... 69
Appendix A: Interview Notes ................................................................................................... 71Appendix B: Chevalier’s Updated Behavioral Engineering Model (BEM) Order Completed for NLC Training Specialist ..................................................................................................... 77Appendix C: Survey Questions/Results ................................................................................. 80Appendix D: Extant Data Examples ........................................................................................ 91Appendix E: Job Aids .............................................................................................................. 92References .............................................................................................................................. 101
Effective Feedback for Performance Evaluation 4 Rutherford, Naylor, & Wheeler
Project Description
Project title Effective Feedback for Performance Evaluations and Beyond
Information about the organization Northwest Lineman College (NLC) is an Idaho based for-profit vocational college offering training and education services to the power delivery industry - mainly training pre-apprentice to journey-level lineworkers. NLC currently has four campus locations: Meridian, Idaho (corporate headquarters); Oroville, California; Denton, Texas; Edgewater, Florida. NLC’s mission/vision and core values are important in providing insight into the organization: NLC’s mission/vision Northwest Lineman College will be a dynamic, visionary, and leading international educational institution providing the benchmark standard of safety, training, and products that exceed the expectations of customers in the power delivery and related industries. NLC’s core values Northwest Lineman College has an outstanding reputation throughout the power delivery industry, thanks to the high standards of its company, staff, and students. The Core Values and Mission Statement are used as the guiding principles by which NLC conducts training, business, and all endeavors. P.I.E. = PASSION, INTEGRITY, EXCELLENCE* PASSION Love what you do with PERSISTENCE, MOTIVATION, and the ATTITUDE to succeed. INTEGRITY Do the right thing—even when no one is watching—so TRUST, HONOR, and DEPENDABILITY are your hallmarks. EXCELLENCE Innovate always, leveraging ORIGINALITY and TEAMWORK, so your LEGACY of positive contribution to the industry is guaranteed. *(PIE is a mnemonic for the power formula, P (watts) = I (current) X E (voltage), which is a well-known and used formula in the power delivery industry): NLC hires qualified journey-level lineworkers as Training Specialists and is required by internal policy and state/accreditation agencies to train and certify Training Specialists to teach and train in all of its programs. An internal training department, Grid University, is responsible for designing and implementing all training for the Training Specialists. Currently, new-hire Training Specialists complete a 40-hour training program introducing education and training principles and practices that will get them started teaching in the
Effective Feedback for Performance Evaluation 5 Rutherford, Naylor, & Wheeler
classroom and field. After the initial certification training, all Training Specialists are required to complete 40 hours of continuing education a year that builds on the introductory content of the initial certification.
Information about the site client The site client is the Chief Education Officer (CEdO) for the organization and holds a PhD in vocational education and has 37+ years of experience in higher education, both in teaching and administration. His major responsibilities are to oversee and supervise Grid University operations - specifically the instructional design process for faculty and staff training courses and programs - as well as review and advise on all externally offered program curriculum and training/education services. Client Stakeholders* Training Leads Campus Training
Specialist Campus
Lonnie M. Texas Charlie C. Idaho
Nick J. Idaho Mark G. Idaho
Rick C. Florida Darrow L. Idaho
Travis H. California Joe M. California
Jim L. California *The client stakeholders are listed for informational purposes only. The ID team is not responsible for organizing stakeholders.
Information about the project Request Background & Summary The initial request came to the Educational Leadership Committee (ELC membership consists of Training Leads for each campus, Director of Grid University, CEdO, and the VP of Curriculum & Instruction) directly from the President of the company. The NLC President based his request on feedback from several large industry employers who questioned the relevance of the behavior assessment course in the Electrical Lineworker Program (ELP). Specifically, their major concerns are: graduates with high behavior marks aren’t passing behavior-based interviews; graduates with high behavior marks are failing in apprenticeships. The behavior assessment grades are based on results from formal performance evaluations conducted throughout the term. There are several potential causes of this problem, some of which are:
• The test itself may be faulty. This would mean that the test isn’t an authentic measure of ELP skills: the test may not be valid. Addressing this issue is outside this team’s scope of work.
• Instructors may not be administering the test consistently. It could be that some instructors aren’t administering the entire test. Or, that different parts of the test are being administered in different ways. Or, that the checklists used to determine mastery are not clear. This may be within the scope of this project.
Effective Feedback for Performance Evaluation 6 Rutherford, Naylor, & Wheeler
• Instructors may not be providing consistent feedback. This is within the scope of this project.
To summarize the request in the words of the President and the ELC (not direct quotes):
• Training Specialists need to provide feedback to students during evaluations in a way that:
o Promotes accountability o Is thoughtful and supportive o Doesn’t break a student down o Is truthful
Business Need The ELP program objective is to provide training and education that ensures graduates are highly qualified for initial employment, highly successful in completing formalized apprenticeship and training programs, and are prepared to pursue long, rewarding careers as lineworkers. This request is linked to the ELP program objective and to the overall mission/vision and core values of the company. The NLC President is concerned that the low quality of performance evaluations is harming the reputation of NLC and its graduates within the industry and is not meeting the program objective. The demand for NLC graduates in the industry is what drives enrollment in the ELP, which is a major revenue source; therefore, maintaining their reputation in the industry and exceeding customer expectations is of upmost importance. Performance Gap Summary
Target Population Performance Standard
Desired Performance
What NLC wants their Training Specialists
to do is provide effective feedback during formal student evaluations
that is: truthful feedback based on specific and noted observations of the student’s progress in the program; setting the expectation for student accountability for their actions/non-actions; empathetic yet firm on discussed plans for improvement if necessary.
Actual Performance
What NLC Training Specialists
are doing now is reading through evaluations and providing standardized feedback from the forms
that is: generic and not indicative of actual student performance.
Effective Feedback for Performance Evaluation 7 Rutherford, Naylor, & Wheeler
Team members Team Member Email Phone Emergency Contact
Beverly Naylor [email protected] (H) 613-834-9927 (C) 705-817-5628
Dan Naylor (husband) 613-834-9927
Kara Rutherford [email protected] (C) 208-392-8097 Jonathan Cano (husband) 208-409-7456
Tammy Wheeler [email protected] (C) 804-564-4308 David Wheeler (husband) 804-874-9266
Team Contact Escalation Plan If a team member disappears from contact, Team NLC follows this contact escalation plan:
1. Send an email and copy other team members. If the team member in question does not respond within 24 hours, proceed to next step.
2. Send text message to cell phone. If the team member in question does not respond within 12 hours, proceed to next step.
3. Call team member and leave a message. If no answer, immediately contact other team members to formulate alternate plan which the team then expedites.
4. Email the team member in question to discuss the team’s decision to proceed with alternate plan.
5. If no response is made within 24 hours after emailing alternate plan, a team member calls the emergency contact.
Project schedule Team NLC Instructor recognizes that the dates listed for site client review are tentative dates that may need to be adjusted to comply with the client’s schedule. Team review Site client
review Resubmission Final course
deadline
Project description Thursday 8 September
N/A Thursday 15 September
Friday 9 September
Project status review Thursday 8 September
N/A N/A Friday 9 September
Team charter Thursday 8 September
N/A N/A Friday 9 September
Team meeting 1 with Steve Friday 8 September
N/A N/A Sunday 11 September
Performance analysis Sunday 25 September
Wednesday 12 October
Friday 7 October
Friday 30 September
Effective Feedback for Performance Evaluation 8 Rutherford, Naylor, & Wheeler
Task analysis Tuesday 27 September
Wednesday 12 October
Friday 7 October
Friday 30 September
Project status review 2 Thursday 29 September
N/A Friday 7 October
Friday 30 September
Team meeting 2 Sunday 25 September
N/A N/A Sunday 2 October
Learner analysis Sunday 9 October
Wednesday 12 October
Friday 21 October
Saturday 15 October
Objectives Sunday 16 October
Wednesday 19 October
Friday 28 October
Saturday 22 October
Performance assessment Sunday 23 October
Wednesday 26 October
Friday 11 November
Saturday 29 October
Instructional planning worksheet
Tuesday 8 November
Wednesday 9 November
Friday 18 November
Friday 11 November
Project Status Review 3 Tuesday 8 November
N/A Friday 18 November
Friday 11 November
Team meeting 3 with Steve Sunday 6 November
N/A N/A Sunday 13 November
Instructional plan Sunday 20 November
Wednesday 23 November
Friday 9 December
Monday 28 November
Instructional materials (optional)
Sunday 20 November
Wednesday 23 November
Friday 9 December
Monday 28 November
Formative evaluation (Review of another team)
Sunday 4 December
N/A Friday 9 December
Monday 5 December
Effective Feedback for Performance Evaluation 9 Rutherford, Naylor, & Wheeler
Performance Analysis Effective Feedback for Performance Evaluations & Beyond
Gap Analysis
Target Population Performance Standard
Desired Performance
What NLC wants their Training Specialists
to do is provide effective feedback during formal student evaluations
• Assesses cumulative performance level including grades, competencies, certifications, safety, attitude, housekeeping, excellence, and integrity with at least a moderate level of inter-rater reliability (based on Cohen’s interpretation of kappa, moderate level of agreement, k = 0.41 – 0.60);
• Completes the ELP performance rubrics based on assessment in a timely manner (completing forms in no more than 15 minutes per student – for 32 students);
• Conducts the face-to-face formal evaluation, in an empathic, firm, and consistent manner, using the completed ELP performance rubric effectively (completing each student meeting in 10 minutes or less; student’s report 80%
Effective Feedback for Performance Evaluation 10 Rutherford, Naylor, & Wheeler
satisfaction with evaluation via post evaluation survey).
Actual Performance
What NLC Training Specialists are doing now is reading through evaluations and providing standardized feedback from the forms
that is generic and not indicative of actual student performance.
Organizational Analysis About NLC Northwest Lineman College (NLC) is an Idaho based for-profit vocational college offering training and education services to the power delivery industry - mainly training pre-apprentice to journey-level lineworkers. NLC currently has four campus locations: Meridian, Idaho (corporate headquarters); Oroville, California; Denton, Texas; Edgewater, Florida. NLC’s mission/vision is: Northwest Lineman College will be a dynamic, visionary, and leading international educational institution providing the benchmark standard of safety, training, and products that exceed the expectations of customers in the power delivery and related industries. NLC hires qualified journey-level lineworkers as Training Specialists and is required by internal policy and state/accreditation agencies to train and certify them to train in all of its programs. An internal training department, Grid University is responsible for designing, implementing, and assessing all training for the Training Specialists. Grid University assesses Training Specialists during their initial certification process to determine readiness to begin training in the educational programs and services. Currently, new-hire Training Specialists complete a 40-hour training program introducing education and training principles and practices that will get them started teaching in the classroom and field. After the initial certification training, all Training Specialists are required to complete 40 hours of continuing education a year that builds on the introductory content of the initial certification. Currently, Training Specialists complete a 25-minute lesson on the ELP performance rubric form and evaluation process as part of 1-2 hour session on the purpose of assessments and evaluations in education. This 25-minute lesson provides only a brief overview of the ELP performance rubric and is general information: it does not provide instruction on how the rubric is to be completed or the processes involved in completing the ELP performance rubric.
Effective Feedback for Performance Evaluation 11 Rutherford, Naylor, & Wheeler
Information About the Site Client The NLC President has identified the site client for this project as the Chief Education Officer (CEdO) for the organization. The CEdO holds a PhD in vocational education and has 37+ years of experience in higher education, both in teaching and administration. His major responsibilities are to oversee and supervise Grid University operations - specifically the instructional design process for faculty and staff training courses and programs - as well as review and advise on all externally offered program curriculum and training/education services. Why Closing the Performance Gap is Important to NLC Based on informal feedback from industry employers of NLC graduates, the NLC President has identified (in an informal non-qualitative/non-quantitative analysis) that there is a current performance gap of instructors not providing consistent feedback to students, which affects NLC’s reputation as an educational leader in the electrical lineworker industry. NLC relies on this reputation to maintain enrollments in its educational programs and services as most incoming students report that they heard about NLC from an NLC graduate or someone in the industry. Additionally, NLC’s mission/vision statement indicates a clear commitment to customer satisfaction, indicating they want to “exceed the expectations of customers in the power delivery and related industries.” Therefore, closing this performance gap is an important step to increasing customer satisfaction in the marketplace. Measures Associated with Performance Currently, NLC does not collect specific data on the student performance evaluation process or how it measures its reputation perception. The NLC team will measure success by:
● Collecting and analyzing the performance forms to determine consistency using the Kappa statistic, which measures inter-rater reliability.
● Surveying students to analyze student satisfaction of the evaluation process. ● Observing Training Specialist preparing for and conducting the student performance evaluations.
Team NLC Instructor collected data from the sources listed below to analyze both the performance and causes of the stated performance gap.
Data sources Data gathering methods Findings
Phone interview BSU ID Team interviewed Frank S. (pseudonym), Training Specialist for NLC (based out of Texas campus),
Environment Information: Each performer felt improvement could be made in regards to providing feedback. The exemplary
Effective Feedback for Performance Evaluation 12 Rutherford, Naylor, & Wheeler
identified as a newer Training Specialist
performer noted expectations are verbally communicated but not written out.
Resources: Both performers felt data was available. There was some discrepancy on how to go about finding the data. Processes and procedures have changed over time.
Incentives: Both performers indicated that no incentives, monetary or non-monetary are in place. Occasional verbal praise is given. The organization does not have any overt rewards or punishments for desired or undesired behavior; however, the performance form and process itself may be covertly punishing desired behavior because rating a student as unacceptable requires more time and effort to complete the forms and provide feedback.
Motives: Both performers felt verbal and written feedback is an essential part of the education they provide to future linemen.
Capacity: There are no issues in regard to capacity or ability to learn.
Knowledge/Skills: It’s difficult to truly know a student’s knowledge, skill, and behavior if you don’t interact with the student on all levels in which they are evaluated.
Phone interview BSU ID Team interviewed Sam M. (pseudonym), Training Specialist for NLC (based out of Idaho campus), identified as an exemplary performer
Survey BSU ID Team conducted 3 surveys (see Appendix C):
• Survey 1 was sent to all Training Specialists and had questions based on Chevalier’s Updated Behavioral Engineering Model (BEM) for NLC Training Specialist (Appendix C).
Survey 1 (survey sent to 24 Training Specialists, 18 responded – 75% response rate). Survey highlights/findings:
• The majority of Training Specialists have performed 4 or more performance evaluations (83%) and report they know what is expected of them when conducting performance evaluations (89%).
• There is somewhat of split on whether regular and meaningful feedback is given on their performance (28% disagree/strongly disagree that they get feedback while 39% agree/strongly agree they get
Effective Feedback for Performance Evaluation 13 Rutherford, Naylor, & Wheeler
feedback).
• While Training Specialists report they are very comfortable providing feedback to students during performance evaluations (94%), the response to how they feel about providing written and verbal feedback during evaluations indicates an inconsistency (33% feel average or fair and 67% feel good or excellent).
• 61% of training specialist report they agree/strongly agree that they’ve received adequate training while 39% report they are neutral or disagree that they’ve received adequate training.
• Survey 2 was sent to all Campus Directors and Training Leads, who are responsible for their campus’ Training Specialists ability to perform as expected and had questions based on the Job Aid / EPSS Analysis (Appendix C).
Survey 2 (survey sent to 11 Campus Directors/Training Leads, 8 responses – 72.3% response rate). Survey highlights/findings:
• 75% of Campus Directors/Training leads report that it’s important to provide feedback to Training Specialists about their performance level conducting student performance evaluations.
• Campus Directors/Training leads report are evenly split on whether they think Training Specialists can provide feedback to students without a significant amount of practice (50% agree, 50% disagree).
• Campus Directors/Training Leads are split on whether a job aid could be used in place of formal training sessions (38% disagree while 62% agree/strongly agree).
Effective Feedback for Performance Evaluation 14 Rutherford, Naylor, & Wheeler
• Survey 3 was sent to all Training Specialists to collect information for a learner analysis with more specific questions about completing the ELP performance rubric form.
Survey 3 (survey sent to 24 Training Specialists, 15 responded – 62.5% response rate). Survey highlights/findings:
• The majority of Training Specialists report that the behaviors criteria is the most difficult to accurately assess using the ELP performance rubric form (60%).
• The majority of Training Specialists report their performance level on accurately assessing student behavior as average or good (87%).
• 93% of Training Specialist report they are very open to receiving feedback on performance.
• 100% of Training Specialists report that training on assessing student performance is relevant and meaningful to them and their job.
• 67% of Training Specialists report there are barriers to applying what they learn to the workplace.
• 53% of Training Specialists report that it’s difficult to get enough information about unfamiliar students in order to accurately assess behavioral performance.
NLC Webpage Extant Data • NLC history
• NLC spirit
• NLC core values, mission
• Electrical Lineworker Program (ELP) objective
Electrical Lineworker Program (ELP) Performance Rubric and Coordinating Improvement Planners for Academics, Skills, and
Extant Data • Criteria (Knowledge - Grades, Skills - Competencies, Certifications, Behaviors - Safety, Attitude, Housekeeping, Excellence, Integrity)
• Performance Level (Unacceptable, Expected,
Effective Feedback for Performance Evaluation 15 Rutherford, Naylor, & Wheeler
Behavior Exceptional)
• Required Probationary Action Steps for Academic, Skills, and Behavior Improvement
Updated Behavioral Engineering Model for Training Specialists Environment Information Resources Incentives
• 89% of Training Specialists surveyed agree they know the level of performance expected when conducting, and providing effective feedback during formal student evaluations.
• 39% of Training Specialists surveyed get regular and meaningful feedback on level of performance on providing feedback during formal performance evaluations.
• 88% of Campus Directors and Training Leads surveyed agree Training Specialists are expected to provide feedback to students during formal student performance evaluations quickly and smoothly.
• 63% of Campus Directors and Training Leads surveyed disagree that the environment or situation is unpredictable.
• 50% of Training Specialists surveyed agree that there are adequate resources (time, forms, help from others) available to help provide feedback to students during formal student performance evaluations.
• Electrical Lineworker Program (ELP) Performance Rubric and Coordinating Improvement Planners for Academics, Skills, and Behavior provide a guide for the Training Specialists during the student performance evaluations.
• There is no mentor program in place to provide coaching to new Training Specialists.
• 61% of Training Specialists surveyed agree there are non-monetary incentives for good or high levels of performance in providing feedback to students during formal student performance evaluations.
Effective Feedback for Performance Evaluation 16 Rutherford, Naylor, & Wheeler
Individual Knowledge/Skills Capacity Motives
• 83% of Training Specialists surveyed have conducted 4 or more formal student performance evaluations.
• 50% of Training Specialists surveyed agree that the processes and procedures for conducting student evaluations have remained consistent over time.
• 60% of Training Specialists surveyed
indicate the behaviors criteria are the most difficult to assess.
• Training Specialists have the capacity or ability to learn.
• All Training Specialists are
previous linemen that possess knowledge of regulations that apply to electrical linework (OSHA, NESC, etc.). All possess a Journeyman lineman certificate of completed apprenticeship.
• Educational level breakdown:
o H.S. diploma/GED: 45.8%
o Some college or Vocational/Technical program: 37.5%
o Associate degree: 4.2% o Bachelor degree: 12.5%
• 66% of Training Specialists surveyed feel good or excellent about providing written and verbal feedback to students during formal student evaluations.
• 17/18 Training Specialists surveyed are comfortable with providing feedback to students that need improvement in the knowledge, skills, and behavior phase.
Effective Feedback for Performance Evaluation 17 Rutherford, Naylor, & Wheeler
Job Aid / EPSS Analysis Question Yes No How do you know? What evidence (data/findings) can you provide
to support your answer? 1. Is it important for people to practice the
task to mastery? Or can people perform the job task without a significant amount of practice?
X The Training Specialists reported in the survey conducted that the task of conducting formal student evaluations involves numerous challenging tasks such as: working with a tight schedule, being organized and prepared, and getting other Training Specialists together to discuss student behaviors. Additionally, survey results from Campus Directors and Training Leads indicate a strong agreement (62.50%) of the importance in practicing providing feedback to students during formal evaluations to a level of mastery. 50% of Campus Director and Training Leads believe a significant amount of practice is needed to provide effective feedback to students.
2. Is it important for people to obtain both: ° Coaching (including error detection
and correction) as they perform a task?
° Delayed feedback (about the extent to which performance met standards and how to improve it) after they perform a task?
X The target audience for this training worked in the field as linemen prior to becoming Training Specialists. The typical performer noted he’s not as comfortable in the classroom. Learners will need to have multiple opportunities to see a variety of scenarios where a student needs improvement and coaching of how an exemplary performer guides the student to providing feedback needed for improvement based upon the performer’s observations and documentation. Opportunities should be provided to practice scenarios with coaching and feedback prior to working in the classroom to ensure mastery of providing feedback to linemen. The training will provide confidence and preparation necessary to respond to various student scenarios the Training Specialist faces during the evaluation process.
3. Is it important to provide learners with a safe, controlled environment in which they can make mistakes as they learn?
X The training environment provides the Training Specialist with a safe controlled environment where areas of weakness can be improved upon and provided feedback without affecting the linemen students. 80% of Training Specialists indicate that the learning environment is a safe and positive environment.
4. Is there an expectation of interactive learning?
X Results from Survey 2 indicate the campus/program leaders agree (75%) and strongly agree (25%) that there is an expectation of interactive learning when it comes to the student performance
Effective Feedback for Performance Evaluation 18 Rutherford, Naylor, & Wheeler
evaluations. 5. Are people required to perform the task
quickly and smoothly in the workplace? X Performers mentioned they are given 10 minutes per student during the
formal evaluation process. 6. Are people required to perform the task
in situations that are unpredictable? X Training Specialists perform in a relatively predictable environment. As
such, these performers could benefit from the use of a job aid. 7. Does one or more of the following
reasons for NOT relying on a job aid alone exist in the on-the-job environment: ° A physical barrier that makes it
difficult to use a job aid on the job ° A social barrier that makes it
difficult to use a job aid on the job
X We believe the ELP Performance Rubric and coordinating planners are a form of a Job Aid already. Although we have identified possible Job Aids not in place today to help facilitate the process of delivering effective feedback. Topics for Job Aids include:
● Performance Levels (Unacceptable, Expected, Exceptional)
● Assessment Decisions Flowchart
● Evaluation Site Setup (Outside Team’s Scope of Work)
● Veteran Instructor Guide (Outside Team’s Scope of Work)
Summary: State the Case for Training Based on the Gap, Cause, and Organizational analysis results, Team NLC Instructor concludes that the performance gap can be closed with training and job aids.
• Currently, typical performers, especially novice Training Specialists, conduct student performance evaluations in a very generic and impersonal manner. There is a tendency to rate students as having met the expected performance level because it is easier to do in the amount of time provided and with the amount of information available. Exemplary performers conduct performance evaluations in a very personal and specific manner and tend to rate students more reliably using their time to focus on consulting with other Training Specialists to get information when required and accurately using the ELP performance rubric to rate students appropriately.
• The performance analysis indicates that experienced Training Specialists are adequately trained, however, there are indications that novice Training Specialists require more information, training, feedback, and time/collaboration to compile the evaluations, especially for students they don’t know.
• There is currently a brief introduction to the ELP performance rubric form and evaluation process; however, there are no opportunities to practice or experience the many varied situations that arise when preparing and conducting the performance evaluations.
• Training Specialists report that the behavior criteria assessments are difficult and complex to complete accurately. The task of assessing behavior requires complex decision making and situation recognition. A training program with a focus
Effective Feedback for Performance Evaluation 19 Rutherford, Naylor, & Wheeler
on using job aids to help complete the behaviors criteria section of the ELP performance rubric as well as practicing complex decision making and situation recognition is necessary to assess behavior needed to close the gap.
Closing this performance gap is important to the organization because it’s directly linked to maintaining or increasing customer (graduates and industry employers) satisfaction. Graduates want a return of investment on their training. Industry employers want to be assured that the graduates are performing at a level that meets recognized industry standards.
Effective Feedback for Performance Evaluation 20 Rutherford, Naylor, & Wheeler
Task Analysis Effective Feedback for Performance Evaluations and Beyond
Data Sources Selected Task
Analysis Method Rationale Data Sources
Procedural task analysis with if-then and complex tables
• This process consists of a series of sequential steps and decision-making processes;
• The task analysis uses if-then decision to represent simple decision-making processes;
• The task analysis uses complex tables to represent the questions that exemplary performers ask themselves to reach a decision.
• Interview with NLC exemplary instructor (Charlie,10+ years of experience as a Training Specialists at NLC);
• Interview with NLC novice instructor (George, little over 1-year experience as a Training Specialist at NLC);
• Survey of NLC Training Specialists;
• NLC documentation.
Task Characteristics
Task Characteristic Description Major tasks The major tasks involved in providing feedback
include:
• Reviewing the Student File Packet and verifying contents with other instructors to assess student performance;
• Completing the NLC Electrical Lineworker Program performance (ELP) rubric correctly and consistently;
• Preparing the interview site;
• Debriefing the student based on the completed ELP rubric.
Goals for each major task • Ensure the students’ performance is assessed consistently based on the data provided from the Student File Packet and from interviewing other instructors;
• Ensure the ELP performance rubric is completed correctly based on data provided/collected;
• Ensure the students receive objective feedback based on actual performance results.
Effective Feedback for Performance Evaluation 21 Rutherford, Naylor, & Wheeler
Task Characteristic Description Cues • The formal student performance evaluations are
scheduled to occur on the Thursday of weeks 6, 10, and 15 of the ELP (program is 15 weeks total);
• Preparation and completion of forms is scheduled to occur the day prior to the scheduled formal evaluations.
Resources • The ELP performance rubric is very detailed and has evolved to be a sort of job aid in and of itself;
• The Student File Packet contains: o A blank ELP performance rubric with
student name labeled at top; o Student grades in all academic courses up
to the date of the evaluation; o Student progress in all competency skill
areas up to the date of the evaluation; o List of Safety Council members; o Student spot evaluations for:
§ Recognition of Exceptional performance
§ Sticker records § Probation (academic, skill, or
behavioral) § Not meeting written program
requirements § Absence counts § Tardy counts
• 8.5 hours of preparation time, scheduled to occur the day prior to giving the feedback;
• 8 hours of time scheduled to conduct the face-to-face performance evaluation meetings.
Standards that the completed task should meet • Consistently assess student’s cumulative
performance level including grades, competencies, certifications, safety, attitude, housekeeping, excellence, and integrity with at least a moderate level of inter-rater reliability (based on Cohen’s interpretation of kappa, moderate level of agreement, k = 0.41 – 0.60);
• Complete the ELP performance rubrics based on assessment in a timely manner (completing forms in no more than 15 minutes per student – for 32 students);
• Conduct the face-to-face formal evaluation, using the completed ELP performance rubric effectively (completing each student meeting in 10 minutes or less; student’s report 80% satisfaction with evaluation via post evaluation survey).
Effective Feedback for Performance Evaluation 22 Rutherford, Naylor, & Wheeler
Task Characteristic Description Prerequisite skills • Be able to organize/sort and reference information
provided; • Be able to communicate clearly (enunciate,
smooth flow/not stumbling with words, interact with others, verbally, in writing, and through active listening.
Additional factors to consider related to Task Analysis: Consequences of not accurately completing the tasks
• Students are provided with misleading feedback.
• Students may believe that they are performing to a higher standard than what they are actually attaining.
• Weak students may feel isolated and alienated, feeling that they do not have support.
• Industry leaders hiring NLC are dissatisfied with the performance of NLC graduates.
Assumptions made by Team NLC Instructor • Instructors want to aid in student learning and development.
• An instructor may not have observed a specific student performing in the field or interacted with a specific student prior to the formal assessment.
• An instructor may have to depend on the Student File Packet and other instructor input for information to include in the ELP performance rubric.
Key to symbols
' Hot tip: Hints or advice about performing the task
F Note: Additional information
M Caution: Use for safety and potential mistakes
How to read this task analysis:
Yellow activities are performance activities that are critical to providing feedback.
Blue tasks are activities that do not require training and can easily be managed by the use of job aids or fall outside the scope of the team’s 2-3 hour training session.
Completed Task Analysis Conducting Formal Student Performance Evaluations 1.0 Prepare to assess student performance based on Student File Packet data.
1.1 Obtain the list of students assigned to you for evaluation and their corresponding Student File Packets.
1.2 Verify that the Student File Packet contains all required documents necessary to assess performance for each student on the assigned list.
F Note: The Student File Packet is provided by the academic assistant and contains:
Effective Feedback for Performance Evaluation 23 Rutherford, Naylor, & Wheeler
• A blank ELP performance rubric form with student’s name labeled at the top • A list of Safety Council Members • Academic Records (current grade report) • Skill Records (competency and certification progress report) • Behavioral records (spot evaluations, probations, attendance records)
1.2.a If any documents are missing ask the academic assistant to provide the necessary information.
1.3 Review the list of assigned students to identify unfamiliar students (unfamiliar students are those with whom you’ve had little to no interaction with in the classroom, field, and/or behaviorally). 1.3.a Highlight unfamiliar student names on the list to indicate interviews with
other instructors will be necessary to complete the assessment and completion of the ELP performance rubric form.
1.3.b If there are no unfamiliar student names on the list, proceed to step 2.0 and follow directions for familiar students throughout the task analysis.
1.4 Conduct interviews about unfamiliar students.
2.0 Use the Student File Packet to complete the ELP performance rubric for each student on the assigned list (see Appendix XX (not included in this document) to view entire form).
F Note: If you have a list of 32 students, you will have just under 16 minutes to prepare and complete the ELP performance rubric form for each student on your list. If you have 16 students, you will have just over 30 minutes to prepare and complete the form for each student on your list. Don’t rush, but stay focused and remember to consult and collaborate with other instructors to get all the information you need to complete the forms accurately and consistently. The majority of instructors report that the full 8.5 hours is enough time to prepare and complete the forms.
2.1 Check the appropriate box in the Formal Evaluation section to identify the type of formal evaluation.
F Note: Week 6 evaluations are 1st of term and Week 10 evaluations are 2nd of term. Disregard the spot evaluation checkboxes for the formal evaluation.
Image for step 2.1
2.2 Complete the Knowledge Criteria Section
2.2.a Review the student’s current grade report to assess Knowledge criteria performance.
2.2.a.1 If the student name is highlighted as unfamiliar, consult with instructors who are familiar with the student.
2.2.a.1.i Obtain additional feedback on performance in the classroom.
Effective Feedback for Performance Evaluation 24 Rutherford, Naylor, & Wheeler
2.2.a.1.ii Take written notes of any information that will be relevant to provide feedback for the student verbally during the formal evaluations.
2.2.b Based on academic course scores, circle the appropriate box on the ELP performance rubric form.
Image for Step 2.2.b
2.2.c If student is marked “Unacceptable” see step 2.6 to complete the ELP Academic Improvement Planner prior to moving on to Step 2.3.
2.3 Complete the Skill Criteria Section
2.3.a Review the student’s current competency and certification progress report to assess Skill criteria performance.
2.3.a.1 If the student name is highlighted as unfamiliar, consult with instructors who are familiar with the student.
2.3.a.1.i Obtain additional feedback on performance in the field.
2.3.a.1.ii Take written notes of any information that will be relevant to provide feedback for the student verbally during the formal evaluations.
2.3.b Based on competency and certification scores, circle the appropriate box on the ELP performance rubric form.
F Note: In week 6, students should be 90% complete with level 1 competencies/certifications. In week 10, students should be 100% complete with level 1 competencies/certifications and 80% complete level 2 competencies/certifications.
Image for Step 2.3.b
2.3.c If a student is marked Unacceptable, use decision table to proceed:
If the student was Then
also marked Unacceptable in the Academic criteria section of the ELP performance rubric form
Don’t proceed completing the Skills Improvement Planner – instead:
Bring the Student File Packet and ELP performance rubric to Training Lead (the
Effective Feedback for Performance Evaluation 25 Rutherford, Naylor, & Wheeler
Training Lead is responsible for completing the rest of the form and conducting the evaluation – the student may be terminated per the satisfactory progress policy of the program).
not marked Unacceptable in the Academic criteria section of the ELP performance rubric form
Proceed to step 2.7: Complete the ELP Skills Improvement Planner prior to proceeding to Step 2.4.
2.4. Complete the Behaviors Criteria Section 2.4.a Use the following decision table to assess behaviors criteria on the ELP
performance rubric:
Precursor Action Results Interpretation Exceptional Behavior
2.4.a.1 • Student has one or
more spot evaluations for recognition of Exceptional performance/stickers.
F Hot tip: Reference Appendix XX (not included in this document) for sticker descriptions, taken from the program orientation guide provided to student during first week of class).
M Caution: If a student has one or more Exceptional spot evals, but also has one or more Unacceptable spot evals, bring the student information and the evaluation to the Training Lead to assist with the rest of the assessment. This situation is very rare and requires additional scrutiny by
• If spot eval form(s) was
completed by another instructor, review any notes provided and consult with instructor who completed the form to obtain any additional facts, etc.;
• If spot eval form(s) was completed by you, think back to the circumstance that led you to perform the spot evaluation;
• Read the Expected and Exceptional rubric boxes carefully to determine the appropriate rubric box to mark – depending on which category the spot evaluation is for, some categories may still be marked as Expected;
• Identify which behaviors category the spot evolution/or own assessment fits into:
Safety examples • Serves as a member of
safety committee; • Consistently observed
pointing out safety issues even when they don’t know they are
• Instructor
accurately and consistently assesses student behavior when spot evaluations are present for Exceptional behavior;
• The correct behavior category is identified consistently amongst students being evaluated.
• One or more
spot evaluations for recognition of Exceptional performance, indicates the student is performing at the Exceptional level for one or more of the given categories;
• Reviewing the spot eval forms with the instructor who completed them allows the evaluating instructor to become more familiar with the student’s behaviors and skilled in giving consistent and accurate
Effective Feedback for Performance Evaluation 26 Rutherford, Naylor, & Wheeler
leadership to avoid mistakes in the process.
being observed; • Consistently observed
watching out for his/her crew and others – attentive and engaged in the field at all times.
Attitude examples • Has one or more
passion sticker spot evals;
• Consistently positive and upbeat – encourages his/her team and others;
• Takes on role of peer teaching, coaching without prompting;
• Takes on challenges positively and rallies his/her crew to succeed;
Housekeeping examples • Cleans and/or picks up
shed/tool room/field area/classroom without prompting;
• Checking equipment, tools, and materials to ensure they are put away and not damaged without prompting;
Excellence Examples • Has one or more
excellence and/or knowledge and/or skills sticker spot evals;
• Consistently engaged in classroom, field, lab environments – answers questions when asked, asks appropriate questions, leads group discussions or activities;
• Has aced all academic course tests in one or more subjects – verify on grade report.
Integrity Examples • Has one or more
integrity sticker spot evals;
• Has been observed
feedback.
Effective Feedback for Performance Evaluation 27 Rutherford, Naylor, & Wheeler
turning in lost tools, making sure sheds are locked.
• Circle the correct rubric
box based on the overall assessment for each Behavior criteria category.
Areas for Improvement 2.4.a.2 • Student has one or
more spot evaluations for probations, not meeting written program requirements, excessive absences or tardies.
• If the spot eval form was
completed by another instructor, review any notes provided and consult with instructor who completed the form to:
• Obtain any additional facts, etc.
o Determine if the student met the requirements of any previous spot evaluations
o OR if the spot eval was completed by you, think back to the circumstance that led you to perform the spot evaluation;
• Read the Unacceptable and Expected rubric boxes carefully to determine the appropriate rubric box to mark – depending on which category the spot evaluation is for, some categories may still be marked as Expected.
M Caution: It’s very rare that a student with an Unacceptable spot eval will be assessed as Exceptional in any other Behavior criteria. Consult with the Training Lead to avoid making
• Instructor
accurately and consistently assesses student behavior when spot evaluations are present for behavior that requires improvement;
• The correct behavior category is identified consistently amongst students being evaluated;
• If any category is marked Unacceptable, the behavior improvement planner is completed to offer the student additional feedback and options for improvement – see step
• One or more
spot evaluations for probation, not meeting requirements, etc. indicates the student is performing at the Unacceptable level for one or more of the given categories; however,
• If progress is shown in follow-up documentation and the student is following improvement guidelines, the student is performing at the Expected level for one or more of the given categories;
• Reviewing the spot eval forms with the instructor who completed them allows
Effective Feedback for Performance Evaluation 28 Rutherford, Naylor, & Wheeler
mistakes in your assessment if this occurs.
• Identify which behaviors category the spot evolution/or own assessment fits into:
Safety examples • Not wearing gaff guards,
climbing with proper gloves, or wearing the required personal protective equipment (PPE);
• Injury occurred after coaching/feedback from instructor, e.g. cut to hand/thigh in underground, large splinters/injuries due to gaff outs on pole;
• Not completing tailboard per NLC standards/requirements.
Attitude examples • Negative or short
tempered with his/her crew or others;
• Complains regularly and/or blames others;
• Reluctant to do the what’s expected in class, field or lab – answering questions with “I don’t know” saying “I don’t want to”, etc.;
• Often uses foul language after being warned;
Housekeeping examples • Misplaces name tag or
doesn’t place on tables in classrooms without prompting;
• Crew or others are regularly (2 times or more) picking up after the student because student skips out, e.g. tools, equipment, etc.
2.8. the evaluating instructor to become more familiar with the student’s behaviors and skilled in giving consistent and accurate feedback.
Effective Feedback for Performance Evaluation 29 Rutherford, Naylor, & Wheeler
Excellence Examples • Not ready to perform
competencies or certifications during verification/check-off weeks;
• Excessive absences or tardies (3 or more absences, 5 or more tardies).
Integrity Examples • Does not participate in
community service projects (indicated in grade report);
• Caught lying, cheating, or stealing;
• Continues to use foul language after being warned;
M Caution: It’s rare that a student with a spot evaluation for an integrity violation of cheating or stealing will still be enrolled in the program by the time the formal evaluation occurs. Refer to the student conduct policy when assessing this Behavior criteria category (Appendix XX -not included in this document). Consult with the Training Lead to avoid making mistakes in your assessment if this occurs.
• Circle the correct rubric box based on the overall assessment for each Behavior criteria category.
2.4.a.3 • Student has no spot
evaluations for Exceptional behavior or behavior that
• If student is unfamiliar to
you, consult with other instructors to get more
• Instructor
accurately and
• The lack of
documentation in the
Effective Feedback for Performance Evaluation 30 Rutherford, Naylor, & Wheeler
needs improvement. information, feedback, etc.;
• If the student is familiar to you, try to recall specific interactions with the student to note for verbal feedback;
• Read the Unacceptable and Expected rubric boxes carefully to assess the student’s behavior accurately and consistently
F Note: It’s rare that a student is performing at the Unacceptable level if there are no previous spot evaluations or probations in the Student File Packet.
• Identify which behaviors category the spot evolution fits into (safety, attitude, housekeeping, excellence, integrity);
• Circle the correct rubric box based on the overall assessment.
consistently assesses student behavior when no documentation is present in the student file;
• The correct behavior category is identified consistently amongst students being evaluated;
• If any category is marked Unacceptable, the behavior improvement planner is completed to offer the student additional feedback and options for improvement – see step 2.8.
student file, indicates the student is performing in the Expected rating for one or more of the given categories;
• Reviewing the student’s performance level with instructors who are familiar with the student allows the evaluating instructor to become more familiar with the student’s behaviors and skilled in giving consistent and accurate feedback.
Effective Feedback for Performance Evaluation 31 Rutherford, Naylor, & Wheeler
Image for step/table 2.4.a
2.4.b If a student is marked Unacceptable use decision table to proceed:
If Then
The student was also marked Unacceptable in the Academic OR the Skill criteria section of the ELP performance rubric form
Don’t proceed completing the Skills Improvement Planner – instead:
M Bring the Student File Packet and ELP performance rubric to Training Lead (the Training Lead is responsible for completing the rest of the form and conducting the evaluation – the student may be terminated per the satisfactory progress policy of the program).
If the student was not marked Unacceptable in the Academic OR Skill criteria section of the ELP performance rubric form
Proceed to step 2.8: Complete the ELP Behavior Improvement Planner prior to proceeding to Step 2.5
2.5 Complete the accomplishments section 2.5.a Check the box if the student is the Student Body President or Vice
President (indicated on grade report) and/or check the Others box if the student is on the Safety Committee – write Safety Committee on the indicated line.
2.5.b Check boxes for every spot evaluation for stickers present in the Student File Packet and tally results on the indicated line.
2.5.c Complete the Others box to specify behavior that you think justifies special or additional note beyond what is verbally conveyed based on the rubric form assessment.
Image for steps 2.5.a – 2.5.c
2.6 Complete the total score boxes 2.6.a There are no points awarded for Unacceptable ratings, an improvement
plan must be completed.
2.6.b Count the number of boxes circled in the Expected column.
2.6.b.1 Add 5 points for each box circled.
2.6.b.1 Write total in the Score box in the Expected column.
2.6.c Count the number of boxes circled in the Exceptional column.
Effective Feedback for Performance Evaluation 32 Rutherford, Naylor, & Wheeler
2.6.c.1 Add 10 points for each box circled.
2.6.c.2 Write the total number in the Score box in the Exceptional column.
Image for steps 2.6.a – 2.6.c
2.7 Complete ELP Academic Improvement Planner (ONLY if student has been marked as Unacceptable in the Academic criteria category) – see Appendix XX (not included in this document) to reference the form.
2.7.a Check the box that best reflects the specific area of needed improvement
2.7.b Provide comments that support the academic improvement plan.
2.8 Complete the ELP Skill Improvement Planner (ONLY if student has been marked as Unacceptable in the Skill criteria category; if student has an Unacceptable in Academics refer to step/decision table 2.3.c) – see Appendix XX (not included in this document) to reference the form. 2.8.a List the competency groups and/or certifications that need improvement,
in terms of completion progress.
2.8.b Check the box(s) that apply for the Required Probationary Action Steps for Improvement.
2.8.c Provide comments that support the skill improvement plan.
2.9 Complete the ELP Behavior Improvement Planner (ONLY if student has been marked as Unacceptable in the Behavior criteria category; if student has an Unacceptable in Academics OR Skill refer to step/decision table 2.3.c) – see Appendix XX (not included in this document) to reference the form. 2.9.a Based on the assessment on the ELP performance rubric form, circle the
correct area of Unacceptable behavior.
2.9.b Write an improvement/assignment due date 1 week from the date of the evaluation.
2.9.c Provide comments that support the skill improvement plan.
M Caution: These forms can be subpoenaed so ensure your comments are professional, truthful and that you check for spelling and grammar (google search words/grammar if you have doubts).
' Hot Tip:
o After completing the assessments and forms for all the students on your list, try to prepare for the face-to-face meetings by either practicing with other instructors or mentally envisioning going through the meetings.
o Put yourself in the student’s shoes and think about how you would want feedback – either for Exceptional, Expected or Unacceptable behavior, but also remember
Effective Feedback for Performance Evaluation 33 Rutherford, Naylor, & Wheeler
that the student is accountable for their ratings and they have to make the choice to improve.
3.0 Set-up interview site (A job aid can be used).
3.1 Arrange chairs to facilitate face-to-face contact.
4.0 Debrief student using ELP performance rubric. Use the following table (this step is outside of the team’s scope for a 2-3 hour training session).
If Then
If exemplary student Typically, the student nods in agreement and signs the assessment form. The session is short as there are typically few to no questions.
If average student Typically, the student is in agreement and may have a few questions on how to improve performance. The session is average in length.
If weak student The instructor must be able to discuss the student’s performance and provide recommendations on how to improve performance. The supporting assessment forms must substantiate a report of poor performance.
The instructor will be expected to exchange more information and as a result this session may take longer than the average session.
4.1 Present the feedback in a non-threatening way
4.2 Specify details when giving examples.
4.3 Ask open-ended questions.
4.3 Normalize the conversation at the end of the session.
' Hot Tip: 1. Remain calm during the feedback session.
2. When meeting with a disappointed student:
o Listen attentively and patiently to the student’s concern
o Remain calm
o Place the “burden of proof” on the student. Do not placate the student but rather ask the student to explain how their work fulfills the requirements of grading criteria
o Be future oriented
o Let the student know that you are there to aid in their development.
Effective Feedback for Performance Evaluation 34 Rutherford, Naylor, & Wheeler
(3-5 things) of What Is Critical, Difficult, and Complex 1. Completing the behaviors criteria section of the performance evaluation forms
consistently and accurately. a. Interpreting documentation in the student file that was not submitted by the
evaluating instructor; b. Consulting with other instructors to accurately assess students’ performance.
2. Circling the correct behavior criteria rubric boxes based on the information provided in the student’s file, information provided by the familiar instructors, and common examples of behaviors for each situation. This is very difficult for novice instructors.
3. Adequately compiling notes, including student-specific examples for each behavior criteria rubric box rating based on instructors own observations, file documentation or consultations with other instructors, for each assigned student within the time frame provided.
Effective Feedback for Performance Evaluation 35 Rutherford, Naylor, & Wheeler
Learner Analysis Effective Feedback for Performance Evaluations and Beyond
Brief Description of the Learners
The learners are Training Specialists at Northwest Lineman College (NLC), a for-profit vocational college. The Training Specialists
are located in Meridian, Idaho; Oroville, California; Denton, Texas; and Edgewater, Florida. At least 15 of the students are
new/novice instructors.
The learners are qualified journey-level lineworkers who have been trained and certified as Training Specialists. These learners are
all men who have a significant amount of experience in the trade with 59% of learners having over 10 years in the trade. Their level
of education is: 45% have high school/GED; 37.5% have completed vocational or technical training; 4.2% have an associate degree;
and 12.5% have completed a Bachelor’s degree. Forty-six percent of learners have 4 years or more at NLC, 25% have 1-3 years
with NLC, and 29% have less than 1 year at NLC. Currently, new-hire Training Specialists complete a 40-hour training program
introducing education and training principles and practices that will get them started teaching in the classroom and field. After the
initial certification training, all Training Specialists are required to complete 40 hours of continuing education a year that builds on the
introductory content of the initial certification.
The learners have stated in interviews and surveys that verbal and written feedback is an essential part of the education they provide
to future linemen. The Training Specialists have indicated that processes and procedures have changed over time and that it is
difficult to truly know a student’s knowledge, skill, and behavior if they don’t interact with the student on all levels in which they are
evaluated. Specifically, they indicated that completing the behavioral form and providing feedback to weak students was the most
difficult.
Sources and Methods
Data Sources Data Collection Method(s)
Training Specialist, exemplary performer - SME Phone interview
Face-to-face interviews
Training Specialist, novice performer - SME Phone interview
Training Specialists Surveys
Directors / Leaders Survey
NLC Training Specialists and staff On-site observations
Effective Feedback for Performance Evaluation 36 Rutherford, Naylor, & Wheeler
Human Resources Reporting Extant data
Data-Gathering Instruments
Data was collected using the following instruments:
• Phone Interview questions were based on the Updated BEM and were used when interviewing both an exemplary instructor
and a novice instructor. The questions generated responses that provided information on the learner, their work, and their
environment.
• Face-to-face interviews were used to gather more in depth information about the learner, work processes, and the
environment.
• Observations were conducted informally.
• Extant data was provided by HR on the demographics of the learners. • Surveys were based on the Updated BEM to gain an understanding of the learner, the work, and the environment. Survey
questions and responses also served to confirm findings from the phone interviews. Surveys included:
o Training Specialist Learner Analysis Survey - The survey, conducted 12 October 2016, sampled 24 Training
Specialists and had a 62.5% return rate. See Appendix C for compiled raw data.
o Performance Evaluations for the ELP Student Survey - The survey, conducted 29 September 2016, sampled 24
Training Specialists and had a 75% return rate.
o ELP Student Performance Evaluation Survey - The survey, conducted 29 September 2016, sampled 10 directors and
had an 80% return rate.
Effective Feedback for Performance Evaluation 37 Rutherford, Naylor, & Wheeler
Findings and Implications
Relevant Skills
Learner Characteristic Findings: What you found out about each characteristic
Data Source/Method: List a data source/method for each finding
Instructional Implications: Ideas for the instruction in the form of
“therefore” statements Prior Knowledge—what
the target population
already knows and does
All learners have received some
training from Grid U. 20% are
either neutral or disagree that this
training has helped to develop
knowledge and skills.
Survey - Training Specialist
Learner Survey
Performance Evaluations for the
ELP Student Survey
Therefore, training should address
the areas that are not previously
covered in Grid U training and/or
should cover what is difficult or
complex.
There is a wide range of prior
knowledge when completing
student performance evaluations.
Experience ranges from
completing nil terms to 10 or
more terms.
Learners have conducted student
performance evaluations:
• Nil terms: 12.5%
• 1-3 terms: 6.25%
• 4-6 terms: 12.50%
• 7-9 terms: 6.25%
• 10 or more terms: 62.5%
Performance Evaluations for the
ELP Student Survey
87.5% of Training Specialists
surveyed have conducted formal
student performance evaluations.
Therefore, to narrow the scope of
the training, the instruction should
focus on one particular group of
learners, the novice learner.
A potential consequence of
training all students together is that
the experienced learner will be
bored and may develop a negative
attitude toward the training.
Conversely, if training is focused at
the more experienced learner, the
training may be too challenging for
the novice learner. The novice
learner requires opportunities for
success if confidence is going to
be developed.
Prerequisite skills that
learners should possess
before they can
Learners have conducted student
performance evaluations:
• 4-6 terms: 12.50%
Performance Evaluations for the
ELP Student Survey
Therefore, these experienced
individuals will not be required to
attend class with the novice. The
Effective Feedback for Performance Evaluation 38 Rutherford, Naylor, & Wheeler
challenge the training. • 7-9 terms: 6.25%
• 10 or more terms: 62.5%
Learners know the expected level
of performance when conducting,
and providing effective feedback
during evaluations:
• 44% agree
• 44% strongly agree
learners who have conducted
assessments for at least 4 terms
and who know the expected level
of performance when conducting
and providing effective feedback
during evaluations will be able to
challenge the training.
The super-performers may coach
the novice learners. The super-
performers will also aid in
determining the level of mastery.
Targeted Skills—what the
target population will
specifically need to
learn—and what’s critical,
difficult, or hard about
that
60% of all respondents indicated
that the behavior component of
the assessment is the most
difficult to complete because it
requires complex decision
making.
13% state that it’s difficult to
assess student performance in
the behavior category using the
ELP performance rubric for
familiar students.
Survey - Training Specialist
Learner Survey
Interview with novice instructor
Therefore, the scope of this
training will focus on the behavior
component of the ELP
Performance Rubric.
Additionally, the learner will be
provided with a behavior
performance decision-table job aid
that the learner will use during the
training and later on the job.
Therefore, using the ARCS model,
the learners should practice
assessing behavior in a safe
learning environment until they feel
confident that they can do the job.
Effective Feedback for Performance Evaluation 39 Rutherford, Naylor, & Wheeler
Assessing student files must be
conducted within 20 minutes.
Feedback from respondents is
that this is difficult to do.
Survey - Training Specialist
Learner Survey
Interview with novice instructor
Therefore, during the application
phase, the training will use
scaffolding so that the learner can
achieve the 15-20 minute limit.
The first and second practice
activities will have a 25 minute
time limit to complete one
assessment, the third and fourth
practice activities will have a 30
minute limit to assess two
assessments.
The learner will be given 30
minutes to complete the
assessment activity.
Over 50% of respondents
indicated that obtaining enough
information about unfamiliar
students in order to accurately
assess performance is difficult.
Survey - Training Specialist
Learner Survey
Interview with novice instructor
Therefore, during the application
phase, the learner will be provided
4 practice scenarios to assess
behavior performance. The
Student File Packet for each
scenario will be missing relevant
behavior information. The learner
will have to identify the missing
data on behavior performance,
identify where to obtain missing
data and will then have to obtain
the missing data prior to assessing
the behavior level of performance.
Ideally, SMEs will be available to
assume the role of a source of
missing information. This will
ensure that valid and reliable data
is provided to all learners. The
SMEs may also be able to model
how to provide information.
Respondents indicated that Therefore, the training will
Effective Feedback for Performance Evaluation 40 Rutherford, Naylor, & Wheeler
training would be most helpful for:
• Training on how to manage
weak students and what
happens if they don’t improve
• Termination
emphasize an objective and
systematic approach to the
assessment.
A flowchart will be provided as part
of the job aid to guide the learner
in completing the assessment.
Provide the learner with a
Worksheet job aid for taking notes
on the factors that lead to
decisions on the behavior
performance level.
Respondents self-reported that
their own level of performance in
assessing student performance in
the behavior categories on the
ELP performance rubric was as
follows:
• 13% rate their performance to
be excellent;
• 60% rate their performance
as good; and
• 27% rate their performance
as average.
This reflects that there is a wide
range of experience and self-
reported expertise amongst the
Training Specialists.
Therefore, focusing on only the
novice will ensure that those
Training Specialists with over 10
terms are not subjected to the
same training as the novice. It is
assumed that the experienced
performers self-rated as excellent
and good.
Relevant Contextual Factors
Learner Characteristic Findings: What you found out about each characteristic
Data source/method List a data source/method for
each finding
Instructional Implications: Ideas for the instruction in the form of
“therefore” statements Orienting Context—
Learner motivations
related to the job and
any corresponding job
Reception to feedback:
• 60% of Training Specialists
indicate that they are open to
receiving feedback.
Survey- Training Specialist
Learner Survey
Therefore, the instruction should
provide appropriate feedback.
Experienced Training Specialists
will be able to challenge the
Effective Feedback for Performance Evaluation 41 Rutherford, Naylor, & Wheeler
Learner Characteristic Findings: What you found out about each characteristic
Data source/method List a data source/method for
each finding
Instructional Implications: Ideas for the instruction in the form of
“therefore” statements training • 33% indicate that they strongly
agree that they are open to
receiving feedback on
assessing student
performance.
• 7% disagree.
training. Super-performers will be
encouraged to coach the less-
experienced.
Therefore the instruction will
provide the novice learner with
feedback on one’s performance.
The novice will also be able to self-
assess by noting the time it takes
to complete the performance
activities, as well as, their
improvement to identify missing
pieces of data.
Relevance of assessing student
performance and providing
feedback during formal student
evaluations as part of the job:
• 86% indicated that they agree
or strongly agree that it is
relevant.
• 7% are neutral
• 7% strongly disagree
Survey- Training Specialist
Learner Survey
Therefore, the instruction should
be focused in the application
phase where feedback skills can
be developed during realistic and
varied performance assessment
scenarios.
Therefore the learner will be
directed to take notes on key
behavior factors that led to their
decision on performance level.
This will aid in providing feedback
on the ELP Behavior Improvement
Planner.
72.2% of Training Specialists rated
their comfort level with providing
feedback to students that need
improvement in the behavior
Therefore to maintain Attention,
Relevance, Confidence, and
Satisfaction, the training should be
directed toward the novice who
Effective Feedback for Performance Evaluation 42 Rutherford, Naylor, & Wheeler
Learner Characteristic Findings: What you found out about each characteristic
Data source/method List a data source/method for
each finding
Instructional Implications: Ideas for the instruction in the form of
“therefore” statements phase as ‘Excellent’.
need to develop their confidence
level.
Instructional Context—
Any factors in the
learning environment
that may affect learning
Learners indicate that they prefer:
• 47% group classroom
environment
• 20% e-learning programs
• 33% one-on-one classroom
• 33% self-directed/self-paced
• 74%-realistic work environment
with mock scenarios to practice
and learn from mistakes
Survey - Training Specialist
Learner Survey
Therefore, instruction during the
application phase should be based
on realistic scenarios involving a
range of borderline acceptable to
unacceptable behaviors, with
realistic Student File Packets, for
the learner to assess.
The learning environment could be
a traditional classroom setting,
online, or OJT. The environment
will be determined by factors such
as time and cost.
The learners will be able to
complete practice activities with
diminished coaching under
realistic time constraints.
The learning environment for a
Training Specialist is safe and
positive:
• 80% agree or strongly agree
• 20% are neutral or disagree
Survey - Training Specialist
Learner Survey
Transfer Context—Any
factors related to the
workplace that could
facilitate or hinder
application of learned
skill on the job
Barriers to applying what is
learned in Grid U sessions to the
workplace include:
• 33% not enough time to apply
what I have learned
• 7% It’s too hard to apply what I
have learned to my work.
• 33% It doesn’t seem like
anyone else really applies what
is learned to the job.
• 7% Not enough support or
Survey - Training Specialist
Learner Survey
Therefore, during the integration
phase, the learners will prepare a
plan for using the new knowledge
and skills on the job. Emphasis will
be given on how the job aid can
assist the learner when completing
the ELP Performance Rubric on
the job.
Provide Training Leads with the
job aid and worksheet for
Effective Feedback for Performance Evaluation 43 Rutherford, Naylor, & Wheeler
Learner Characteristic Findings: What you found out about each characteristic
Data source/method List a data source/method for
each finding
Instructional Implications: Ideas for the instruction in the form of
“therefore” statements reinforcement from colleagues
or leaders to apply what is
learned to my job.
• 40% indicate that there are no
barriers.
observing future performance
evaluations.
Mode of Delivery
Delivery Mode (select one or more) Rationale
ü Instructor-led training. Can include:
ü Traditional classroom
ü Virtual classroom
ü Self-paced training. Can include:
o Print-based workbooks
ü Online/eLearning that learners complete individually
o Structured on-the-job training
o Flipped classroom
The primary delivery mode for this training will be instructor-led training
at the Boise campus. This mode was selected as it meets the criteria
of transferring the knowledge and skills to the new instructor, cost, and
time factors.
New NLC instructors attend a Grid U training program at the Boise
campus before assuming their duties at their home campus. It has
been identified that there is time in the schedule to add this training
component. Thus, there are no additional costs for travel and lodging.
The traditional classroom with instructor-led training will enable the
training to focus on the targeted tasks that the learner has identified as
being critical and difficult:
• Obtaining information on unfamiliar students
• Preparing feedback for weak students
• Completing the assessments within 15-20 minutes
We considered other modes of delivery that would support learners
using realistic scenarios for practice. However, we determined that by
narrowing the scope of this training to the less than 15 novice learners,
the time and costs involved in providing an eLearning program would
be too prohibitive for any immediate training.
In creating the training for an instructor-led training program, we
recommend that consideration be given to making this an e-learning
program in the future, should the number of potential learners
Effective Feedback for Performance Evaluation 44 Rutherford, Naylor, & Wheeler
increase.
We also recommend that experienced Training Specialists and
Training Leads provide coaching and mentoring to the novice Training
Specialist.
Job-Focused objectives Providing Feedback for Performance Evaluations and Beyond
Critical Tasks that the Training Will Address Task 2.4
4. Completing the behaviors criteria section of the performance evaluation forms consistently and accurately.
a. Interpreting documentation in the student file that was not submitted by the evaluating instructor;
b. Consulting with other instructors to accurately assess students’ performance.
5. Circling the correct behavior criteria rubric boxes based on the information provided in the student’s file, information provided
by the familiar instructors, and common examples of behaviors for each situation. This is very difficult for novice instructors.
6. Adequately compiling notes, including student-specific examples for each behavior criteria rubric box rating based on
instructors own observations, file documentation or consultations with other instructors, for each assigned student within the
time frame provided.
Effective Feedback for Performance Evaluation 45 Rutherford, Naylor, & Wheeler
Job-Focused Instructional Objectives # Performance (include TA #) Conditions on the job Criteria on the job
1 (2.4 ) Assess student behavior
performance
Cues • Formal student performance
evaluation process schedule
(preparation occurs Wednesday of weeks 6, 10, and 15, the day prior to the formal student performance evaluations which are scheduled to occur on the Thursday of weeks 6, 10, and 15 of the ELP – program is 15 weeks total);
• Receives the Student File Packet
and student list to be evaluated
from the Academic Assistant.
Resources • Student File Packet;
• Worksheet (a job aid used to collect missing information and compile notes for the Student File Packet);
• ELP performance rubric form
• Academic assistant
• Other instructors
• Mission window policy
• Confidentiality policy
• Uses worksheet job aid to gather
data;
• Verifies that Student File Packets
are complete;
• If incomplete, identifies the
missing information (e.g. missing behavioral spot evaluation forms, forms lack enough detail, and/or instructor has not worked directly with the student in the field, classroom and/or behaviorally);
• Consults with the academic
assistant and/or other instructors
to collect missing information and
complete the Student File Packet.
• Consultation conversations:
o Are focused on the student
behavioral issues in
relation to successful
completion of the ELP;
o Compliant with NLC’s
mission window
o Compliant with NLC’s
confidentiality policies;
• Behaviors criteria ratings on the
ELP performance rubric form are:
o Align with the information
in the Student File Packet
and worksheet job aid;
o Marked based on
information in the Student
File Packet and the
worksheet job aid;
Effective Feedback for Performance Evaluation 46 Rutherford, Naylor, & Wheeler
# Performance (include TA #) Conditions on the job Criteria on the job o Objective;
o Legally defensible;
• Complete ELP performance rubric
forms;
• If necessary, complete behavior
improvement planners;
• Complete worksheet fields.
Effective Feedback for Performance Evaluation 47 Rutherford, Naylor, & Wheeler
Performance Assessment Instrument Providing Feedback for Performance Evaluations and Beyond Objectives
Performance Conditions on the job Criteria on the job
(2.4) Assess
unacceptable
student
behavior
performance
Cues • Formal student performance evaluation
process schedule (preparation occurs Wednesday of weeks 6, 10, and 15, the day prior to the formal student performance evaluations which are scheduled to occur on the Thursday of weeks 6, 10, and 15 of the ELP – program is 15 weeks total);
• Receives the Student File Packets and
student list to be evaluated from the
Academic Assistant;
• Receives a Student File Packet that is
missing information (e.g. existing data is missing from Student File Packet or existing data lacks detail/wasn’t filled out by evaluating instructor, and/or instructor has not worked directly with the student in the field, classroom and/or behaviorally).
Resources • Student File Packet;
• Worksheet (a job aid used to collect missing information and compile notes for the Student File Packet);
• ELP performance rubric form
• Academic Assistant
• Other instructors
• Mission window policy
• Confidentiality policy
• Uses worksheet job aid to gather data;
• Uses worksheet job aid Table A to verify that Student
File Packets contain all existing written documentation;
o If not collects any existing written documentation
missing from Student File Packet from the
Academic Assistant;
• Uses worksheet job aid Table B to determine if the
existing written contains enough information to make a
decision on rating behavioral performance;
o If not, uses worksheet job aid Table C to
§ consult other instructors to collect
information;
§ make substantiated decisions on
behavioral ratings;
• Consultation conversations:
o Are focused on the student behavioral issues in
relation to successful completion of the ELP;
o Compliant with NLC’s mission window
o Compliant with NLC’s confidentiality policies;
• Worksheet job aid is complete;
• Transfers rating decisions from the worksheet job aid to
the behaviors criteria section on the ELP performance
rubric form.
• Ratings are:
o Categorized based on criteria indicators (e.g.
safety, attitude, housekeeping, excellence,
integrity;
o Substantiated based on information in the
Student File Packet and the worksheet job aid;
Effective Feedback for Performance Evaluation 48 Rutherford, Naylor, & Wheeler
Performance Conditions on the job Criteria on the job o Objective;
• ELP performance rubric forms, and if necessary
behavior improvement planners, are complete.
Type of performance assessment and rationale Type of assessment Rationale X Product assessment X Process assessment
As indicated in the task analysis, completing the behaviors criteria section of the ELP performance rubric
requires analyzing information to determine completeness, collecting additional required information and
then making complex decisions to accurately assess student behavioral performance on the ELP
performance rubric form.
In order to assess learner performance in assessing unacceptable student performance, team NLC
designed an assessment with both process and product assessment elements. The team chose both
elements because, while the majority of the objective criteria can be measured by reviewing the finished
job aids (i.e. the finished products/product assessment) during or after the assessment, the consultation
conversations part of the objective criteria is more process oriented and needs to be observed during
the assessment.
Because the process assessment element is only one part of the overall objective, which includes the
consultation conversations between the evaluating instructors and other instructors to gather information,
the team feels both process and product elements can be included in one assessment without creating a
time slip within the instructional plan.
Assessment instrument Team NLC instructor will utilize a checklist to assess instructor’s performance in rating unacceptable student behaviors. The checklist will
indicate whether or not the instructor utilized consultations with other instructors and the job aids (the worksheet and ELP performance rubric
form) correctly and completely to rate student behavioral performance1.
F 1
A facilitator and a group of exemplary performers (SMEs) will use the checklists to assess performance during the final assessment, but will also use the checklist to guide feedback
and error detection during the practice exercises.
Effective Feedback for Performance Evaluation 49 Rutherford, Naylor, & Wheeler
Mastery level The process tasks are critical to completing the behaviors criteria section of the ELP performance rubric correctly and reliably and there are
a variety of situations that come up when assessing unacceptable performance therefore Team NLC instructor will utilize one assessment
with two Student File Packet scenarios to measure mastery2.
A facilitator and 1 or 2 exemplary performers (SMEs) will assess instructor performance using a checklist. Some of the SMEs will also act as
the Academic Assistant or other instructors within the scenarios when information must be collected via consultation. The instructors will
have one attempt at each Student File Packet scenario in the assessment and must successfully complete both with a 90% or higher rating
on the checklist. If the instructors/learner fail one or both Student File Packets, additional instruction and practice with a facilitator and/or
SME will be arranged at a later date with future assessments to follow.
Job Aids See Appendices:
o Appendix E: Flowchart job aid/Advance Organizer
o Appendix E: Worksheet job aid
o Appendix XX (not included in this document): ELP Performance Rubric
o Appendix XX (not included in this document): Behavior Improvement Planner
Performance Assessment
SET UP
The performance assessment will include 2 Student File Packet scenarios for the learners to use to demonstrate desired performance and
mastery of the performance objectives.
The learners will have the following resources available:
o 2 incomplete Student File Packets with indications of familiarity with student noted.
o Flow chart and worksheet job aid
o ELP Performance Rubric and Behavior Planner
o Instructor to consult if needed
o 30 minutes to complete the assessment
F 2
The team will also utilize 4 practice exercises to aid in achieving mastery.
Effective Feedback for Performance Evaluation 50 Rutherford, Naylor, & Wheeler
The facilitator or SMEs will use the assessment checklist to assess the instructor’s performance and will not provide coaching or feedback
during the assessment. The facilitator and SMEs will have the following resources:
o Scenario Handout and Instruction sheet
o Instruction sheet for assessor/observer
o Instruction sheet for consulting instructor
o 1 specific scenario card for each Student File Packet for facilitator or SME acting as consulting instructor for the scenario (if
they are different)
INSTRUCTIONS AND SCENARIOS
Instructions for Facilitator and/or Subject Matter Expert (SME – exemplary training specialists): The Facilitator will
1. Separate the learners into two or more groups to be assessed in different classrooms/rooms.
2. Depending on class size, ensure that no more than 4 (individuals are in a room and ensure there is at least two SMEs per classroom
group. The Facilitator may be a SME for the group he/she is observing.
3. Ensure that each individual is located in separate corners of the room to avoid excessive disturbance/listening-in.
4. Read the learner instructions and inform of resources and time limitations. Indicate who the “Instructor” and/or “Academic Assistant”
is for the group (you can have two SMEs, but they both should play either part to avoid wait times).
5. Remind the learners that the assessment checklist used during practice activities will be the same instrument used to assess their
performance on the assessment.
6. Provide the scenario cards to each SME (they are pre-trained/practiced on the cards) or keep for your own use.
The SME will:
o Use the scenario cards to provide missing existing documentation/information as noted and provide answers to questions asked
based on the information provided in the scenario cards. Provide answers verbally to the learner.
Scenario Cards: • Student File Packet # 1 has unacceptable excellence, will be missing two pieces of information related to attitude and excellence. • Student File Packet # 2 has unacceptable housekeeping, will be missing three pieces of performance information, safety, attitude,
and housekeeping. Instructions for Learner: You will receive 2 Student File Packets to review. Use the resources provided to complete the ELP performance rubric, and improvement
planner if necessary, for each student.
Resources:
• Student File Packet # 1 and scenario card
Effective Feedback for Performance Evaluation 51 Rutherford, Naylor, & Wheeler
• Student File Packet # 2 and scenario card
• Flow chart and worksheet job aid
• ELP Performance Rubric and Behavior Planner
• “Other Instructor” and/or “Academic Assistant” to ask for missing information if needed (other Training Specialists or Training Lead in
the assessment room)
• 30 minutes to complete the assessment
The training Facilitator and “Instructor” and/or “Academic Assistant” will not provide hints or coaching as you progress through the
assessment. The Facilitator will designate who will act as the “Instructor” and/or “Academic Assistant” for you to consult as necessary. The
Facilitator will observe and evaluate your performance based on the Assessment Checklist. If the Facilitator will also act as the “Instructor”
and/or “Academic Assistant” for your group, he/she will inform you prior to starting the assessment.
The Facilitator will provide any additional instructions at the time of the assessment.
Student File Packet Contents and Scenario Cards:
You will receive the following information to complete your assessment:
Student File Packet # 1:
• Contents:
o Good standing academics (mid-B’s in all classroom subjects)
o Good standing skills progress
o Blank absence/tardy counts
o No spot evaluations
• Scenario card:
o You have interacted with this student mainly in the classroom environment. In the classroom, you’ve noticed that the student
is doing ok. He answers questions when asked and is reasonably engaged during classroom activities. You have heard some
students grumble when he is in their group and you’ve overheard some complain about his negativity, but you haven’t
observed anything specific in the classroom environment.
o Because of your limited time in the field prior to this formal student evaluation, you haven’t seen much of the student’s
behavior performance in the field.
Student File Packet # 2:
• Contents:
o Good standing in academics (high B’s and low A’s in all classroom subjects)
o Skills progress has been a little behind (as noted in the spot evaluation)
o Absence/tardy counts are present (2 absences, 1 tardy)
o Student has one spot evaluation for field performance (provided) – no progress/follow up documentation exists
Effective Feedback for Performance Evaluation 52 Rutherford, Naylor, & Wheeler
• Scenario card:
o You have interacted with this student in the classroom briefly and don’t really have a good sense for his behavior.
o You’ve interacted with this student in the equipment and underground areas in the field and you’ve noticed he takes more
time than the rest of his crew mates to get through his practice attempts. While his other crew mates are practicing, instead of
observing and taking notes – he is often dazing off into other parts of the field.
You will have 30 minutes to complete both scenarios. You should be able to complete each scenario in 15 minutes or less.
ASSESSMENT CHECKLIST
Task
Number Task Yes No Comments
Upon verifying the Student File Packet is missing information/is incomplete:
1 Uses worksheet job aid Table A to verify that Student File
Packets contain all existing written documentation
2 • If the instructor finds there is missing existing information,
he/she collects the information from the Academic Assistant
(if the scenario observed does not include this circumstance, please write N/A in the comments section and do not include this as a point in the Final Score section of this checklist)
3 Uses worksheet job aid Table B to determine if the existing
written documentation contains enough information to make a
decision on rating behavioral performance (both Student File Packets will NOT contain enough information to make a decision)
4 Uses worksheet job aid Table C to consult other instructors to
collect information
Consultation conversations:
5 • Are focused on the student behavioral issues in relation to
successful completion of the ELP
6 • Compliant with NLC’s mission window
7 • Compliant with NLC’s confidentiality policies
Effective Feedback for Performance Evaluation 53 Rutherford, Naylor, & Wheeler
Task
Number Task Yes No Comments
8 Uses worksheet job aid Table C to make substantiated
decisions on behavioral ratings
9 Worksheet job aid is complete
10 Accurately transfers rating decisions from the worksheet job aid
to the behaviors criteria section on the ELP performance rubric
form.
Ratings are:
11 • Categorized based on criteria indicators (e.g. safety,
attitude, housekeeping, excellence, integrity)
12 • Substantiated based on information in the Student File
Packet and the worksheet job aid
13 • Objective
14 Signs and dates the ELP Program Performance Rubric
15 The instructor completes the ELP Behavior Improvement
Planner:
16 • Enters an improvement due date
17 • Writes comments
18 • Signs and dates
19 ELP performance rubric forms and behavior improvement
planners are complete.
Total Yes/Nos
Final Score:
• Each Yes is worth 1 point. Each No is worth 0 (zero) points.
• If the scenario observed includes the circumstance described in task 2,
the total possible points are 19.
• If the scenario observed does not include the circumstance described
Effective Feedback for Performance Evaluation 54 Rutherford, Naylor, & Wheeler
Task
Number Task Yes No Comments
in task 2, the total possible points are 18.
• The instructor must achieve a score of 90% or higher to pass the
assessment (to achieve this score the checklist can have no more than
1 No). If the instructor falls below this threshold, refer the instructor to
the Training Lead for further instruction.
Note that based on the results of pilot testing, NLC may want to consider
using weighted scores, giving some items a higher value than others.
Effective Feedback for Performance Evaluation 55 Rutherford, Naylor, & Wheeler
Instructional Plan Worksheet A. OveralllengthofthetrainingprogramDesignNotes:
1. Theinstructor-ledtrainingwithscenariocardsis180minutesinlength.2. LearnersareallnoviceTrainingSpecialists.3. Thistrainingoccurswithinthecontextofa40-hourtrainingprogramintroducingeducationandtrainingprinciplesandpractices.4. ExperiencedTrainingSpecialistsorTrainingLeads(SMEs)toprovidecoachingandfeedbacktothenoviceTrainingSpecialist(twoSMEsperclasswillbeeasilyavailableper
client).Howlong(inminutes)istheentiretrainingprogram? 180minutesHowmanyunitswillyouincludeinthetrainingprogram? 1unitB. Usethesetablestobriefly(2–3sentenceseach)describeeachelementofthetrainingprogram
IntroductiontothetrainingprogramHowmuchtime(inminutes)willbeallocatedtotheintroduction?
5minutes
Writeabrief(2-3sentences)descriptionoftheintroduction
Thefacilitatorbeginsby:a) Welcomingtheparticipantstothetraining;b) Introducinghim/herselfandexperienceatNorthwestLinemanCollege;c) IntroducingexperiencedTrainingSpecialists(SMEs)assistinginpracticeexercisesand
assessments;d) Explainingwhythistrainingisimportant,thetrainingobjectiveandhowitsupports
businessobjectives(customersatisfaction);ande) Communicatinglengthoftheprogram,numberofpracticesexercises,andanassessment
uponcompletionofpracticeactivities.Unit1: AssessingUnacceptableBehaviorPerformance Listtheobjective(s)forthisunitPhase Timeallocatedto
eachphaseDescriptionofthephase
Activation 5minutes Thefacilitatorwill:• Remindthelearnerthattheyhavehadsomeexposuretorubricsandtheirroleinevaluationduringtheirinitial
training.• Introducetheadvanceorganizer(Flowchart)andWorksheetasavisualforthenewlearning.• ExplainhowtheELPPerformancerubricandperformanceevaluationsareliketheonestheyusedtogetonthejoband
Effective Feedback for Performance Evaluation 56 Rutherford, Naylor, & Wheeler
willbesimilartowhatthestudentswillgetonthejob.• Remember,thatitiscriticaltoassessbehaviorwellbecauseofthedangersinvolvedonthejob.Identifytheissues
nowsothatthelearnercancorrectthem.Ifissuesareidentifiedbutnotpointedout,itisessentiallycondoningthebehavior.Doyouwantpoorperformanceonthejob?
Demonstration 20minutes Thefacilitatorwilldemonstrate3scenarios.• Thefirstscenariowilldemonstratetheassessmentofacceptablestudentbehavior.Thefileismissinginformationon
student’ssafetybehavior.• Thesecondscenariowilldemonstratetheassessmentofunacceptablestudentbehaviorwithaclearviolation.The
studentisshorttempered,argumentative,andrefusedtoparticipateinthecommunityserviceproject.Thefileismissinginformationonthestudent’shousekeeping.
• Thethirdscenariowilldemonstratetheassessmentofunacceptablestudentbehaviorthatisdifficulttoevaluate.Thestudent’sbehaviorisinconsistent.Thefileismissinginformationonthestudent’sbehaviorinintegrity.
Uponcompletionofthethreescenariosthefacilitatorwillcomparethedemonstrations.Duringeachofthedemonstrations,thefacilitatorismodelingthebehaviorofanexemplaryperformerconductingtheassessment.Ineachscenariothefacilitatorwillusetheadvanceorganizerasaguidetowalkthroughstepsofassessingastudent’sbehaviorperformance:
a) UponreceivingaStudentFilePacketverifythatitiscompleteandcontainsinformationonbehaviorcriteriaofsafety,attitude,housekeeping,excellence,integrity.Identifyifanyinformationismissinginformation.
b) Usetheflowchartandworksheetdecisiontablestoobtainandrecordtheinformation.c) Emphasizetheimportanceofkeepingtheconsultationconversationsprofessionalandin-linewiththemission
window/confidentialitypolices.Theconversationistofocusonactualknowledgeofthestudentratherthanassumptions.
d) Assessthelevelofstudentbehaviorperformancee) CompletetheELPPerformanceRubric.
Application Time:140minutestocompletetheactivitiesandassessment.Resourcesrequired:PracticeActivities:12x8StudentFilePackets.Eachfileistorepresentadifferentlearnerat
Numberofpractices:4(110minutes),1attemptforeachpractice.Numberofassessments:1(30minutes),1attempt.PracticeActivity1:StudentFilePacket-studentbehaviorhasaclearintegrityviolation,missing2piecesofinformation(safetyandhousekeeping).PracticeActivitytime:25minutesResourcesrequired:StudentFilePacketsMissingFileInformation:ProblemBehaviors:Environment:Thelearnerswillbeinoneclassroom.Thelearnerswillidentifythemissingpiecesofinformationasagroup.Thefacilitatorwillprovidethemissinginformationtothegroupoflearners.Facilitator:
• WillprovideeachlearnerwithaStudentFilePacketthatismissing2piecesofinformation(safetyandhousekeeping)
Effective Feedback for Performance Evaluation 57 Rutherford, Naylor, & Wheeler
eitheranunacceptableorexpectedratingbehavior
necessarytoconductacompleteanalysis.• Thefacilitatorwillcoachthelearnersstep-by-stepthroughtheactivityusingtheadvancedorganizerandworksheetjob
aidtoobtainmissinginformationandmakeratingdecisionstocompletetheELPperformancerubric.• Asagroup,thelearnerswillidentifythemissinginformation,thefacilitatorwillprovidetheinformationtoalllearners.• Asthelearnersworkthroughtheworksheet,thefacilitatorwillprovidefeedbackanderrorcorrectionateachstep.
Learners:Asagroup,thelearnerswillverifythatthestudentfileiscompleteandcontainsinformationforsafety,attitude,housekeeping,excellence,andintegrity.Oncethemissinginformationhasbeenprovided,thelearnerwillindividuallyassesstheStudentFilePacket,usingtheworksheetjobtodeterminethelevelofbehaviorperformance.Thelearnerwill:
• Usingtheworksheetjobaid,analyzethedocumentation;• Takenotes(ontheworksheetjobaid)onthekeybehaviortocompletetheStudentFilePacket;• Makeadecision(rate)onthelevelofbehavior;• CircletheappropriateboxontheELPPerformanceRubric;• SignanddatetheELPPerformanceRubric;• AddcommentsontheELPBehaviorImprovementPlanner.
Debrief:
• Usetheperformanceassessmentchecklisttodiscusscommonerrorsnotedandtipstoavoidcommonerrorsmovingforward.
PracticeActivity2:StudentFilePacket-Student’sunacceptableattitudebehaviorisdifficulttoassess,1pieceofinformationismissing(housekeeping).PracticeActivitytime:25minutes.Resourcesrequired:StudentFilePackets,scenarioinformationcardsFacilitator:
• ProvideeachlearnerwithaStudentFilePacket.• Walkaroundtothelearnersprovidingguidance,feedback,andansweringquestions;• Coachlearnersiftheyattempttomoveforwardwithoutadequateinformation.• Usetheperformanceassessmentchecklisttodiscusscommonerrorsnotedandtipstoavoidcommonerrorsmoving
forward.SMEs(TrainingLeadsorExperiencedTrainingSpecialists)willbeintheclassroomintheroleofAcademicAssistantandotherinstructorstoprovidethemissinginformationwhenthelearnerrequestsit.TheSMEwillmodelbehaviorofprovidinginformation.SMEsmay:
• Providecoachingandfeedbackduringtheactivity;
Effective Feedback for Performance Evaluation 58 Rutherford, Naylor, & Wheeler
• Coachlearnersiftheyattempttomoveforwardwithoutadequateinformation.ThelearnerwillassesstheStudentFilePackettodeterminethelevelofbehaviorperformance.TheStudentFilePacketwillbemissinghousekeepingperformanceinformationandwillrequirethelearnertoobtaininformationaboutstudent’sperformance.Theanalysiswillincreaseindifficultyduetothemissinginformationandmorecomplexdecisionsrequiredaboutperformance.Thelearnerwill:
• Verifythatthepacketiscomplete,havinginformationforsafety,attitude,housekeeping,excellence,andintegrity;o Determinethereismissinginformationandusetheworksheetjobaidtoobtainthemissingdatafromsource
ofdata(AcademicAssistantorotherinstructor/SME);• Usingtheworksheetjobaid,analyzethedocumentation;• Takenotes(ontheworksheetjobaid)onthekeybehaviortocompletetheStudentFilePacket;• Makeadecision(rate)onthelevelofbehavior;• CircletheappropriateboxontheELPPerformanceRubric;• SignanddatetheELPPerformanceRubric;• Iflevelofbehaviorperformanceisunacceptable,addcommentsontheELPBehaviorImprovementPlanner.
Debrief:Usetheadvanceorganizerandworksheettoreviewtheprocessofbehaviorassessment.PracticeActivity3:StudentFilePacketx2,
• StudentFilePacket#1hasaclearsafetyviolationwith2piecesofinformationmissing(safetyandattitude).• StudentFilePacket#2hasacceptablebehavior,2piecesofinformationaremissing(safetyandattitude).
PracticeActivitytime:30minutesResourcesrequired:StudentFilePackets,scenarioinformationcardsThefirstStudentFilePackethasaclearsafetyviolationandismissing2piecesofbehaviorinformation(safetyandattitude).ThesecondStudentFilePackethasacceptablebehaviorbutismissing2piecesofbehaviorinformation(safetyandattitude).Thelearnerwillberequiredtoobtainthemissinginformation.Thepracticeactivityhasincreasedindifficultyasthereisanadditionalfilethatmustbeassessedwithinalimitedtimeperiodof30minutes.Environment:Thelearningenvironmentwillchangeasthelearnersaredividedintotwogroups.OneSMEwillbewitheachgrouptoprovidethemissingdata;thelearner’swillhavetoapproachtheSMEforthemissingdata.TheSMEwillusethescenariocardtoverballyprovidethemissinginformation/answerquestionswhenthelearnerrequestsit.IftheSMEsarebusywithanotherlearner,thelearnershouldproceedtothesecondStudentFilePackettodeterminethemissingdata.ThelearnermayrequestthemissinginformationforeitherfileorbothfilesateachinteractionwiththeSME.Thefacilitatorwill:
• ProvideeachlearnerwithaStudentFilePacket;• Walkaroundtothelearnersprovidingfeedbackasrequestedandansweringquestions;• Coachlearnersiftheyattempttomoveforwardwithoutadequateinformation;• Usetheperformanceassessmentchecklisttodiscusscommonerrorsnotedandtipstoavoidcommonerrorsmoving
Effective Feedback for Performance Evaluation 59 Rutherford, Naylor, & Wheeler
forward.TheSMEswill:
• Providefeedbackduringtheactivity;• Coachlearnersiftheyattempttomoveforwardwithoutadequateinformation;• Usetheperformanceassessmentchecklisttodiscusscommonerrorsnotedandtipstoavoidcommonerrorsmoving
forward.Thelearnerwill:
• Verifythatthepacketiscomplete,havinginformationforsafety,attitude,housekeeping,excellence,andintegrity;o Determinethereismissinginformationandusetheworksheetjobaidtoobtainthemissingdatafromsource
ofdata(AcademicAssistantorotherinstructor/SME);• Usingtheworksheetjobaid,analyzethedocumentation;• Takenotes(ontheworksheetjobaid)onthekeybehaviortocompletetheStudentFilePacket;• Makeadecision(rate)onthelevelofbehavior• CircletheappropriateboxontheELPPerformanceRubric.• SignanddatetheELPPerformanceRubric.• Iflevelofbehaviorperformanceisunacceptable,addcommentsontheELPBehaviorImprovementPlanner.
Debrief:Usetheadvanceorganizerandworksheettoreviewtheprocessofbehaviorassessment.PracticeActivity4:StudentFilePacketsx2,Complexdecision-making,difficulttoassess
• StudentFilePacket#1willbemissingtwopiecesofinformationrelatedtoattitudeandexcellence.• StudentFilePacket#2willbemissingthreepiecesofperformanceinformation,safety,attitude,andhousekeeping.
PracticeActivitytime:30minutesResourcesrequired:StudentFilePackets,scenarioinformationcardsTheanalysiswillincreaseindifficultywiththeamountofmissinginformationandthecomplexdecision-makingrequiredtoassessperformance.Thisactivitydrawsontheskillsandperformanceofthelastthreepracticeactivities.Environment:Thefacilitatordividestheclassintotwogroups.EachlearnerhastwoStudentFilePacketstoassess.OneSMEswillbewitheachgrouptoprovidethemissingdata;thelearner’swillhavetoapproachtheSMEforthemissingdata.TheSMEwillusethescenariocardtoverballyprovidethemissinginformation/answerquestionswhenthelearnerrequestsit.FacilitatorandSMEs:
• ProvideeachlearnerwithaStudentFilePacket;• Willprovidelearnerswiththemissingdata,uponrequest.Learnersmayrequestthemissinginformationforoneor
bothofthestudentfiles.(Somelearnerswillhavehadtoreviewthesecondstudentfilewhilewaitingtogainaccesstothemissinginformation.);
• Willnotprovidefeedbackorcoachingduringtheactivity;• Duringdebriefwillusetheperformanceassessmentchecklisttodiscusserrorsnotedandtips.
Effective Feedback for Performance Evaluation 60 Rutherford, Naylor, & Wheeler
Thelearnerwill:• Verifythatthepacketiscomplete,havinginformationforsafety,attitude,housekeeping,excellence,andintegrity;
o Determinethereismissinginformationandusetheworksheetjobaidtoobtainthemissingdatafromsourceofdata(AcademicAssistantorotherinstructor/SME);
o NotethatifSMEsarebusywithanotherlearner,proceedtothesecondStudentFilePacket.• Usetheworksheetjobaidtoanalyzethedocumentation;• Takenotes(ontheworksheetjobaid)onthekeybehaviortocompletetheStudentFilePacket;• Makeadecision(rate)onthelevelofbehavior;• CircletheappropriateboxontheELPPerformanceRubric;• SignanddatetheELPPerformanceRubric;• Iflevelofbehaviorperformanceisunacceptable,addcommentsontheELPBehaviorImprovementPlanner.
Debrief:Usetheadvanceorganizerandworksheettoreviewtheprocessofbehaviorassessment.Assessment:StudentFilePacketsx2:Complexdecision-making,difficulttoassess
• StudentFilePacket#1ismissingtwopiecesofinformation(attitudeandexcellence).• StudentFilePacket#2ismissingtwopiecesofinformation(safetyandhousekeeping).
AssessmentTime:30minutesResourcesrequired:StudentFilePackets,scenarioinformationcards,instructionsonhowtoperformtheinstructorrole.ThelearnerwillassesstwoStudentFilePackets.SMEs(TrainingLeadsorExperiencedTrainingSpecialists)willassumetheroleofAcademicAssistantandotherinstructorstoprovidethemissinginformationwhenthelearnerrequestsit.Thelearnerswillobtaintheidentifiedmissingdatafromthecorrectsourceofdata.TheassessmentisacombinationoftheprocessusedtocompletetheassessmentandtheproductofacompletedELPPerformanceRubricandELPBehaviorImprovementPlanner.ThelearnerwillfollowtheprocessoutlinedintheFlowchartjobaid,recordnotesintheWorksheetjobaid,circlethecorrectlevelofperformanceforthestudentbeingassessed,indicatethekeybehavior,andinsertcommentsontheBehaviorPlanner.Environment:Theassessmentwillbeheldintwoclassroomstoaccommodatetwogroupsoflearners.Thelearnerswillbedividedintotwogroupstofacilitatetheassessment.Thefacilitatorwill:
• ProvideeachlearnerwithtwoStudentFilePackets;• Providethelearnerfortherulesoftheassessment,particularlyfortherequestofmissinginformation;• Providethemissinginformationuponrequest.Learnersmayrequestthemissinginformationforoneorbothofthe
studentfiles.(Somelearnerswillhavehadtoreviewthesecondstudentfilewhilewaitingtogainaccesstothemissinginformation.)
• Willnotprovidefeedbackorcoaching.ThelearnerwillassesstheStudentFilePackettodeterminethelevelofbehaviorperformance.
Effective Feedback for Performance Evaluation 61 Rutherford, Naylor, & Wheeler
• Verifythatthepacketiscomplete,havinginformationforsafety,attitude,housekeeping,excellence,andintegrity;o Determinethereismissinginformationandusetheworksheetjobaidtoobtainthemissingdatafromsource
ofdata(AcademicAssistantorotherinstructor/SME);• Usetheworksheetjobaidtoanalyzethedocumentation;• Takenotes(onthejobaidworksheet)onthekeybehaviortocompletetheStudentFilePacket;• Makeadecision(rate)onthelevelofbehavior;• CircletheappropriateboxontheELPPerformanceRubric;• SignanddatetheELPPerformanceRubric;• Iflevelofbehaviorperformanceisunacceptable,addcommentsontheELPBehaviorImprovementPlanner.
Debrief:Eachlearnerwillbedebriefedindividually.ThefacilitatororSMEwillusethePerformanceAssessmentChecklistduringthedebrief.
Integration 5minutes Classroom:Facilitatorwillconsolidatewhatlearnershavecovered,specifically,verifyingthatthepacketiscomplete,howtheworksheetfacilitatesthegatheringofdata,andaidsincomplexdecisionmakingrequiredtocompletetheELPPerformanceRubriconthejob.Integrationactivity:Learnerswillidentifyatleasttwokeylessonslearnedthattheycanapplyonthejob.Facilitatorwillemphasizethefollowingpostcourseinformation:
• NLCexpectsthatlearnerswillusetheseresourcesforassessingstudents.• Afterthetraininghasended,learnerswillworkwiththeirTrainingLead.• TrainingLeadswillbeprovidedwithachecklistforobservingthelearners’futureperformanceevaluations–timewill
dependonELPscheduleforeachcampus.• TheTrainingLeadwillprovidefeedbackandcoachingtothelearnerastheyevaluatefuturestudents.• TrainingLeadshavereceivedcopiesoftheadvanceorganizerandworksheet.
Recommendationsforpost-trainingactivities:
• Brownbaglunchestodiscusswhat’sgoingwellandifanythingneedstobechanged.
ConclusiontothetrainingprogramHowmuchtime(inminutes)willbeallocatedtotheconclusion?
5minutes
Effective Feedback for Performance Evaluation 62 Rutherford, Naylor, & Wheeler
Writeabrief(2-3sentences)descriptionoftheconclusion
Facilitatorwillprovidesummaryofthecontentcovered.Advancedorganizerwillbedisplayedasavisual.FacilitatorsandSMEsencouragethenoviceTrainingSpecialiststoutilizetheworksheetjobaidwhenassessingbehavioralperformancewithaStudentFilePacketthatismissinginformationwhentheyareonthejobbecauseitaccomplishestwothings:completingtheStudentFilePacketforfutureperformanceevaluationsandprovidescriticalinformationtoassessdocumentedstudentbehavior.Thefacilitatorwrapsupthecoursebyadministeringalevel1trainingevaluationandindicatesthatalevel3evaluationwillbesentouttoalllearnersandtheirTrainingLeadsseveralweeksormonthsafterthetraining.
PracticeActivity1 PracticeActivity2 PracticeActivity3 PracticeActivity4 Assessment
1
Remindthelearnertoreviewthejobaids.(Flowchartandworksheet)
ü ü
Remindthelearneraboutwhatiscritical,difficult,andcomplex.
ü
Remindthelearnertoreviewtheworksheetjobaid.
ü
Remindthelearnertoreviewtheworksheetjobaid.
Providecoaching(immediateerrordetectionandcorrection)forwhatiscritical,difficult,andcomplex)
ü
Iflearnerattemptstomoveforwardwithoutallrelevantdocument,coachlearnertocheckfordocumentation
ü
Iflearnerattemptstomoveforwardwithoutallrelevantdocument,coachlearnertocheckfordocumentation
ü
Iflearnerattemptstomoveforwardwithoutallrelevantdocument,coachlearnertocheckfordocumentation
Coaching:Stateeachstepinthetasklistbeforeaskinglearnerstocompleteit.
ü ü
Chunking:Provideimmediatefeedbackaftereachdiscretestep.
ü
ü
Usetheperformanceassessmentchecklisttoprovidedelayedfeedback
ü ü ü ü ü
Effective Feedback for Performance Evaluation 63 Rutherford, Naylor, & Wheeler
ProblemProgression 1studentfilepacketwith:
• 1clearviolation
• 2piecesofmissinginformation
1studentfilepacketwith:
• Unacceptablebehaviordifficulttoaccess
• 1pieceofmissinginformation
2studentfilepacketswith:
• 1clearviolation
• 1acceptablebehavior
• 2piecesofmissinginformationinbothpackets
2studentfilepacketswith:
• Complexdecisionmaking
• 2piecesofmissinginformationinonepacket
• 3piecesofmissinginformationintheotherpacket
2studentfilepacketswith:
• Complexdecisionmaking
• 2piecesofmissinginformationineachpacket
Effective Feedback for Performance Evaluation 64 Rutherford, Naylor, & Wheeler
Team Project Status Review # 1
Project Name:
NLC Instructor
Date: 8 September 2016
Report
Author:
Kara Rutherford
Project Team
Members:
Beverly Naylor
Kara Rutherford
Tammy Wheeler
Overall Project Status in Relation to Schedule
Project Status Summary: X Green ____Yellow ____Red
Indicate the extent to which the team believes the ID project is producing quality work on
schedule.
• What parts of the project has the team completed?
Team Charter
Project Description – including project schedule
• What parts of the project are in progress?
Coursework for performance/cause analysis
• What parts of the project will the team work on next?
Performance analysis
Task analysis
Actual or Potential Roadblocks • What obstacles has the team run into so far and how did you work through or around them?
We had some trouble coming up with an initial meeting time that worked for all of us
considering the Labor Day holiday. We worked around it by accommodating schedules to let
members come in a little late with a review of the discussion as well as setting a date and
time for weekly meetings that will work for all of us going forward.
• What potential obstacles do you see related to what you will be working on next?
Since this is our first status and weekly assignment, we are not sure exactly what to expect,
but managing project time, work time, and family time can be an obstacle in completing our
next part of the project.
We don’t expect any obstacles in getting information from the client.
• What, if any, help will you need to overcome the obstacle(s)?
We don’t expect any outside help will be necessary to overcome potential time obstacles or
getting information from the client. We plan to use our weekly meeting to set realistic tasks
and timelines based on our weekly schedules and will evaluate how well our plans worked
prior to setting the next week’s schedule.
Effective Feedback for Performance Evaluation 65 Rutherford, Naylor, & Wheeler
Team Work Successes • What aspects of working as a team have gone well?
We are all very open and active communicators and listeners so we have been able to start
off running. During our initial meeting we used our agenda to cover the following points:
o Meet and greet- this gave us the opportunity to identify other key priorities that we
are facing.
o Personal expectations for the course;
o How we will manage any conflict that may arise over the duration of the project; and
o How we will manage and coordinate weekly meetings. We have a schedule that
identifies who will lead the meeting and who will take minutes. An agenda will be
distributed prior to a meeting. Minutes will be posted following the meeting to ensure
that all members are aware of the team’s expectations of them.
• What will the team do to build on those successes as the project continues?
We will commit to attending all of our weekly meetings and to be honest with challenges we
encounter as well as to be graceful in giving and accepting constructive feedback on work
provided.
Team Work Improvements • What aspects of working as a team could the team improve upon (roles, processes, tools,
etc.)?
It’s a little early to tell yet, but in upcoming project work, we need to set definitive task
assignments for each team member and/or for group task assignments in our weekly
meetings. With the short week and not being able to connect everyone for the full hour, we
were a little rushed at the end of the meeting.
• What steps will the team take to improve these aspects? What, if any, help will you need?
We plan to address this in our weekly meeting agenda and no outside help is needed right
now.
Effective Feedback for Performance Evaluation 66 Rutherford, Naylor, & Wheeler
NLC Instructor Team Project Status Review # 2
ProjectName:EffectiveFeedbackforPerformanceEvaluationsandBeyond
Date: 30September2016
ReportAuthor:BeverlyNaylor
ProjectTeamMembers:
KaraRutherfordBeverlyNaylorTammyWheeler
Overall Project Status in Relation to Schedule ProjectStatusSummary:___Green____Yellow____Red
IndicatetheextenttowhichtheteambelievestheIDprojectisproducingqualityworkonschedule.• Whatpartsoftheprojecthastheteamcompleted?
ProjectDescriptionProjectStatusReview1TeamCharterPerformanceAnalysisPass1
• Whatpartsoftheprojectareinprogress?PerformanceAnalysis-GuidedbytheBEM,theteaminterviewedanexemplaryinstructorandanoviceinstructor.Asurveywasthensentoutto34trainingspecialists(24instructors,10managers)andhada47%responserate.Inaddition,theteamhasreviewedtheNLCwebsiteandtheElectricalLineworkerProgramPerformanceRubric.TaskAnalysis–Theteamhasmetwithanexemplaryinstructorandanoviceinstructorinordertoidentifythetasksinvolvedwithprovidingfeedback.TheteamhasreviewedtheElectricalLineworkerProgramPerformanceRubric,andotherNLCdocumentation.
• Whatpartsoftheprojectwilltheteamworkonnext?LearnerAnalysis
Actual or Potential Roadblocks • Whatobstacleshastheteamrunintosofarandhowdidyouworkthroughoraroundthem?
Theteamlostonemember.Tomanagethisloss,wereallocatedthetasksthathadbeenassignedtothatindividual.Theteamcharterhasbeenamendedtoreflectthis.
• Whatpotentialobstaclesdoyouseerelatedtowhatyouwillbeworkingonnext?
Wecurrentlydonotforeseeanypotentialobstacles.
• What,ifany,helpwillyouneedtoovercometheobstacle(s)?
Therearenoobstaclesatthistime.
Team Work Successes • Whataspectsofworkingasateamhavegonewell?
Theteamhasadedicatedmeetingtimetodiscussthetasksthatneedtocompletedforthatweek.Inaddition,theteamhasdemonstratedflexibilityinitsabilitytoadjustmeetingtimes.Theweekof26September,theteammettwicetodiscussthedeliverablesdueon30September.Thereiseffectivecommunicationwithquickresponsetimes.Emailsareexchangedthroughouttheweektodiscusspointsandsharework.Theteammaintainsasenseofhumor.
• Whatwilltheteamdotobuildonthosesuccessesastheprojectcontinues?
Effective Feedback for Performance Evaluation 67 Rutherford, Naylor, & Wheeler
Tobuildonthesesuccesses,theteamwillcontinuetomaintainitsdedicatedmeetingtime,itsflexibility,itsfrequencyofcommunicationandsenseofhumor.
Teamwillmodifyandadjustmilestonescheduleasperclientrequest.
Team Work Improvements • Whataspectsofworkingasateamcouldtheteamimproveupon(roles,processes,tools,etc.)?
WhilewediscussourexperienceswithourworkplacesandotherOPWLcourses,we’veidentifiedthatweshoulddiscussthecoursereadings,discussions,andcontentinrelationtoourprojectaswellasinourroundtablediscussionsduringourweeklymeetings.
• Whatstepswilltheteamtaketoimprovetheseaspects?What,ifany,helpwillyouneed?
Theteamcurrentlydoesnotneedanyassistanceintheareaofteamworkimprovement.
Effective Feedback for Performance Evaluation 68 Rutherford, Naylor, & Wheeler
NLC Instructor Team Project Status Review # 3
ProjectName:EffectiveFeedbackforPerformanceEvaluationsandBeyond
Date: 11November2016
ReportAuthor:BeverlyNaylor
ProjectTeamMembers:
KaraRutherfordBeverlyNaylorTammyWheeler
Overall Project Status in Relation to Schedule ProjectStatusSummary:___Green____Yellow____Red
WehaveidentifiedthatweareintheYellowleaningtowardsGreenbecausewestillhavethePerformanceAssessmentPass1tosubmit.Whilewehavedevelopedadraftofthisdocument,itisnotreadyforsubmission.WeareconfidentthatwewillbeabletosubmitbothPass1andPass2ofalldocuments.
IndicatetheextenttowhichtheteambelievestheIDprojectisproducingqualityworkonschedule.• Whatpartsoftheprojecthastheteamcompleted?
ProjectDescriptionTeamCharterProjectStatusReview1ProjectStatusReview2PerformanceAnalysisPass1PerformanceAnalysisPass2TaskAnalysisPass1TaskAnalysisPass2LearnerAnalysisPass1LearnerAnalysisPass2ObjectivesPass1ObjectivesPass2InstructionalPlanWorksheetPass1ProjectStatusReview3
• Whatpartsoftheprojectareinprogress?1. PerformanceAssessmentPass1.Draftversion.
• Whatpartsoftheprojectwilltheteamworkonnext?1. InstructionalPlanWorksheetPass22. InstructionalPlanPass13. FormativeEvaluation
Actual or Potential Roadblocks • Whatobstacleshastheteamrunintosofarandhowdidyouworkthroughoraroundthem?
OnechallengethattheteamhasencounterediswithPass1assignments.NLCInstructorstrivestoproduceahighqualityproductandindoingsotheteam’stimelinehasslipped.Currently,thePerformanceAssessmentPlanisinprogress.Wedonotforeseeanyproblemswithhavingthesedocumentsdeliveredbytheduedateoutlinedbythecoursebuthavehadtoadjustourownteamschedule.Anotherchallengeisthatteammemberseachhaveaslightlydifferentinterpretationoftheproject.Theteamhashadtoworktoalignourpersonalvisionoftheprojectintoonesharedvision.While
Effective Feedback for Performance Evaluation 69 Rutherford, Naylor, & Wheeler
onlyaminorchallenge,itresultedintheteamcallingforanadditionalmeetingon10NovembertodiscussObjectivesPass2ratherthansubmittingwhatwasbelievedtobeafinalversion.
• Whatpotentialobstaclesdoyouseerelatedtowhatyouwillbeworkingonnext?
Theteamhasidentifiedtheneedtoclearlyarticulateone’svisionoftheprojectinordertodevelopamutuallysharedprojectvision.Maintainingclear,concisecommunicationwilleliminateorreducepotentialobstaclesthatcouldcausethescheduletoslip.
• What,ifany,helpwillyouneedtoovercometheobstacle(s)?
TeamNLCdoesnotcurrentlyneedanyhelptoovercomethedescribedchallenge.
Team Work Successes • Whataspectsofworkingasateamhavegonewell?
TeamNLCInstructorhasadedicatedmeetingtime,usingGoogleHangouts,todiscussthetasksthatneedtobecompleted.Emailsareexchangeddailytodiscusspointsandsharework.Whenemailsarenotsufficienttoresolveanissue,theteamreadilycallsforadditionalsynchronousmeetings.Themostrecentextrameetingwascalledon10November2016.Eachmemberoftheteamisfullycommittedanddedicatedtoaccomplishingthetasks.
TeamNLCInstructorstrivestocompletealldeliverablesinaccordancewiththeirteamschedule.Byaimingtoadheretotheirschedule,thereisflexibilitytoadjustthescheduleshouldadditionaltimebeneeded.TheteamfeelsconfidentthattheywillbeabletosubmitbothPass1andPass2submissionsforalldeliverables.
TeamNLCInstructorhasdevelopedajobaidthatconsistsofaflowchartandworksheet.ThisjobaidhasbenefittedtheteaminthedevelopmentoftheObjectives,PerformanceAssessment,InstructionalPlanWorksheet,andisanticipatedtobeofbenefitfortheInstructionalPlan.
TeamNLChasbecomemorecriticaloftheworkthatisbeingsubmitted.Priortosubmittingadeliverable,wearenowquestioningthedocumentstoverifythatthereisinternalandexternalalignment.
TeamNLCInstructorincorporatesthecourseweeklycontent,discussions,andreadingsintoteamdiscussions.Theresultisthatallteammembersbenefitfromparticularlyworthypointsthatmayhavebeenmissed.
Whilestrivingtoattaintheambitiousprojectschedulethatwedevelopedatthebeginningofthecourse,wehaverealizedthatitisalivingdocumentthatcanbeadjusted.Byaimingtoadheretoit,westayontrack,byacknowledgingthatthescheduleisaguideweareabletofocusontheactualdeliverableratherthanonthedatedue.
• Whatwilltheteamdotobuildonthosesuccessesastheprojectcontinues?
Tobuildontheteam’ssuccesswewillcontinuetomaintainopenlinesofcommunication.Wehavefoundthatdiscussionsconsistingofonlyafewminuteshasbeeneffective.
Tobuildonthesesuccesses,theteamwillcontinuetomaintainitsdedicatedmeetingtime,flexibility,frequencyofcommunicationandsenseofhumor.
Team Work Improvements • Whataspectsofworkingasateamcouldtheteamimproveupon(roles,processes,tools,etc.)?
Effective Feedback for Performance Evaluation 70 Rutherford, Naylor, & Wheeler
Wehavefoundthateachofushasspecificrolesthatwehaveadopted.Theprocessappearstobeworkingwell.Theteamcanimprovebysharingmoreoftheinformationthatwehavefromeachofourroles.
• Whatstepswilltheteamtaketoimprovetheseaspects?What,ifany,helpwillyouneed?
Theteammemberswillaskmorequestionsofoneanothersothattheonusisonallmemberstogaintheinformation.
Effective Feedback for Performance Evaluation 71 Rutherford, Naylor, & Wheeler
Appendix A: Interview Notes
Frank S. (pseudonym).: Cause Analysis: Variables Questions Typical performer – Novice Environment
1. Information Do the performance
standards make it
clear how you are
expected to provide
feedback?
Are you provided
feedback on this
specific performance?
No, should have had more training on
what to expect and how to answer
questions that they asked me.
No feedback on the specific performance.
2. Resources Are there adequate
resources available to
help you to provide
feedback?
Yes, I think so. But George didn’t know to
ask for it. I believe there are some
available.
3. Incentives Have you received
(are they available)
bonus around good
performance?
Are there non-
monetary incentives
for good performance?
No bonuses. No non –monetary
The only thing I really look for is like an
atta-boy or a good job.
Really enjoys his job and wants to
succeed and if they succeed that is
reward
Individual
4. Motives How do you feel about
providing verbal and
written feedback to
trainees?
I think it’s a good thing because if you
don’t give them feedback they don’t know
what to do to improve or succeed. It goes
both ways.
KRs follow up question -
Don’t see anything getting in the way.
Brand new instructor in the classroom
and asking for feedback from the
students on performance. In the field not
a problem on feedback on what they are
doing because of his experience and
expertise, but in the classroom not so
much
5. Capacity To Manager:
Does the individual
lack the capacity or
ability to learn?
Effective Feedback for Performance Evaluation 72 Rutherford, Naylor, & Wheeler
6. Knowledge/Skills Do you believe the
processes and
procedures have
remained consistent?
Do you feel you are
expected to know too
much information?
No.
Yes.
Frank S. (pseudonym).: Task Analysis: Preparing feedback: I got a form from Samantha and had their grades, attendance, rubrics previous, had every
student the entire semester.
I also had the previous knowledge from working with the students over the term.
Drew on what was on paper in front of me, but my semester experience.
Is there anything difficult or complex about form? No, the form is pretty much laid out, but the only think I had issue with is that I didn’t have
enough time per student and if they had questions, I wanted to be able to answer their
questions so I went over the time.
The hardest part of the whole thing was telling the students weren’t going to graduate it. I don’t
want to become hard-hearted and have no intention of becoming hard-hearted.
How do you prepare yourself to do that? Only way I can see it working where it does not keep me up at night, I think long-term when they
leave here, I don’t want them carrying bad habits into the trade that could cause harm to
themselves or others.
Meeting with student: Do you set up in a specific manner? I use the classroom, I am on one side of table and they are on the other. Maintain eye contact
and did best to not twiddle thumbs or be distracted. Have a grown up to grown up discussion
and not treat them like a child.
When you come across something difficult – giving honest feedback. Is it easier to skim over things or how do you ensure that they get all the necessary feedback? For one thing, I am going to ask them do you understand that you will not graduate. And not
condescending, but being honest and straight forward
What about the student that has done exceptionally well? I don’t think so, I do my best to treat all students equally.
After the conclusion of the evaluation process: Any follow up? Saw the students for two days after and would just check in with them.
Can you imagine a scenario where an instructor may not be able to meet that objective?
Effective Feedback for Performance Evaluation 73 Rutherford, Naylor, & Wheeler
Yes, they don’t care. Yes
In your experience, are you observing students that are not receiving that type of feedback?
Yes.
Is it that they don’t care or don’t know how to do it? Yes, to they don’t care and yes to they may not know how to do it, they may have passion that
is lacking.
Sam M. (pseudonym). Cause Analysis: Variables Questions Exemplary performer – Veteran
Environment
1. Information Do the performance
standards make it clear how
you are expected to provide
feedback?
Are you provided feedback on
this specific performance?
I think they are verbally communicated and
being part of the organization we are in they
can change readily, so understanding that if
things are written and if there is a plan of
instruction you can follow that and it brings
consistency – the other side of that is that when
it’s time to change you have updates and
managing the change.
Communicated yes, verbally – is it monitored,
checked, feedback – could be better.
Feedback: Seen it happen some, but not as much as we
should. When we train a new instructor there is
some feedback given, but after that we should
be giving feedback more often.
Because feedback should be consistent,
positive and
2. Resources Are there adequate resources
available to help you to
provide feedback?
Yes – the data is there and the data is
processed in order to be accessible and the
instructors are given time to digest the data and
make an accurate counseling based on that
data.
3. Incentives Have you received (are they
available) bonus around good
performance?
Are there non-monetary
incentives for good
performance?
Not monetarily – because there is not really
much feedback on the process so even non
monetarily process isn’t there
Effective Feedback for Performance Evaluation 74 Rutherford, Naylor, & Wheeler
Individual
4. Motives How do you feel about
providing verbal and written
feedback to trainees?
I think it’s a very important part of the
educational process. Three ways to learn,
knowledge, skills, behavior. Learning theory is
reinforcement and that comes from counseling
and feedback.
5. Capacity To Manager:
Does the individual lack the
capacity or ability to learn?
6. Knowledge/Skills Do you believe the processes
and procedures have
remained consistent?
Do you feel you are expected
to know too much
information?
Changed. They are important and there has
always been time allotted for formal feedback
and evaluation. Over the 10 years the process
has been polished and been refined to be
better for the student and the instructor.
No, not too much information. The challenge
with knowing for a TS is if you are going to
effectively counsel the student, it’s important to
know the student. If the student is performing
well, it’s easy but if they aren’t you have to get
beyond the data to really understand how to
counsel or encourage or connect with the
student beyond what’s on the data sheet.
You may not interact with the student in all
three areas so it can be hard to counsel
students. I can make circles and boxes but
don’t really know the student.
Sam M. (pseudonym). Task Analysis: How do you prepare for the meeting - walk us through the process?
The way now:
Data is collected the three areas of learning. academic scores, performance in the field –
competency completion criteria, and any counseling sessions that occur for exceptional and
unacceptable behavior.
The information is presented to the instructor for the group of students they are responsible for
counseling. The instructor has time to pour over that and pull the information to put the
information on the forms. The system is set for average and then there is an improvement plan
for those that are not.
In planning for that – there is that tie that needs to be made in knowing the students so without
that it’s hard to counsel the students when you haven’t observed them in all the phases of
education. The students you counsel are the students you work with each day.
When you don’t have a student in the field/classroom.
Effective Feedback for Performance Evaluation 75 Rutherford, Naylor, & Wheeler
An instructor should collaborate with other instructors that have worked with that student to get
information to help counsel the student and provide the best feedback.
Does the form get into detail or generic?
It’s a blend. It’s written so there are doors that you could expand on to be specific to the student.
Able to dig down to be specific to student?
If the student has done exceptionally well, there should be documentation in the student file and
you can mark that and then expound on that detail. If those things are in the folder…consistency
is the key and the documentation may or may not be in the folder.
We have a reward and recognition system with stickers on the hard hat and that happens on the
spot – there should be written documentation that matches that in the folder but the practice
now is that stickers are there but no documentation so getting detailed and specific is difficult.
The form is weak in the area for the stickers and the reward/recognition with specific details.
Challenges?
Safeguarding and the allotted time needed to prepare. The hardest ones are the ones that
require improvement. It’s easy for the exceptional student or the middle of the road.
Will some instructors opt for the middle of the road to avoid the difficulty with counseling improvement?
CS: Is there a tendency to want to push them in meets rather than NI – yes because they are on
the border and you aren’t really sure what to say to the student. May need another instructor to
ask for help or get advice.
Objective is good/ subjective is hard
Concrete data is good in the system, but when it comes to the improvement for the behaviors
can be more difficult in the moment.
If training provided, what processes or tasks, steps would be the most beneficial to focus on?
I think basic to intermediate to advanced communication techniques, understanding the
personality types of people. Also effective documentation skills – used to be open for them to
write comments so they system evolved to circles and checkmarks to make it a little more
standardized which has its pros and cons.
If we are sticking with the system we have now, some basic training on what each of those
boxes mean and maybe even a more advanced training on what those boxes mean. When we
bring someone in they get 1 or 2 hours training, but you need to model it, you need to see it,
observe it, so you need to get into those situations for all varieties of students so they can
develop the areas they are weak in counseling.
Assess where they are strong and weak and then assess them in a training environment before
they are doing it in a real situation.
Get foundational training and then it’s thrown in there and then you learn as you go.
Form: What is the most difficult aspect of that form? CS: the behavioral part, the classroom is cut and dry, skills can be more difficult because it’s a
little more subjective, but students will ask what they can do to be exceptional – they students
don’t know, does the new instructor know what to tell them? IF you have been doing it for a
while you will know better what to say. You just have to use your common sense if you don’t
have the experience.
Effective Feedback for Performance Evaluation 76 Rutherford, Naylor, & Wheeler
That carries over into behavior – there are a lot more areas to asses. Students will question how
they can do better. If you collaborate with instructor before-hand and then students will ask why
they are not exceptional. They think they are exceptional but they are clearly middle of the road.
So working with instructors to know how to answer that is very difficult. When students ask
questions, the evaluation is harder.
Human nature is to gravitate toward the middle of the road or positive to not have to deal with
the difficult feedback. The documentation that needs to be in place to substantiate behavior that
needs to be improved, if it’s not there, it’s really difficult.
If the instructors are not using the forms between the formal evaluations, you don’t have a
picture of the student in order to give them adequate feedback.
The forms can be legal forms and used for terminations, etc. so the instructor may not want to
write much on the forms to avoid any issues in the future.
Scheduling
Used to be two instructors, but now it’s down to one instructor and you have a group of students
assigned to you.
Changed because of man power. They try to schedule students you know or have worked with.
Effective Feedback for Performance Evaluation 77 Rutherford, Naylor, & Wheeler
Appendix B: Chevalier’s Updated Behavioral Engineering Model (BEM) Order Completed for NLC Training Specialist Chevalier’s Updated Behavioral Engineering Model (BEM) Order Completed for NLC Training Specialist
Variables Questions Exemplary performer interview Typical performer interview
Environment
1. Information Do the performance standards make it clear how you are expected to provide feedback?
Are you provided feedback on this specific performance?
I think they are verbally communicated and being part of the organization we are in, they can change readily. So understanding that if things are written and if there is a plan of instruction you can follow that and it brings consistency.
The other side of that is that when it’s time to change you have updates and managing the change.
Communicated yes, verbally – is it monitored, checked, feedback? – could be better.
Feedback:
Seen it happen some, but not as much as we should. When we train a new instructor there is some feedback given, but after that we should be giving feedback more often because feedback should be consistent and positive.
No, should have had more training on what to expect and how to answer questions that they asked me.
No feedback on the specific performance.
2. Resources Are there adequate resources available to help you to provide feedback?
Yes – the data is there and the data is processed in order to be accessible and the instructors are given time to digest the data and make an accurate counseling based on that data.
Yes, I think so. But I didn’t know to ask for it. I believe there are some available.
3. Incentives Have you received (are they available) bonus
Not monetarily. Because there is not really much feedback on the process non-monetary process
No bonuses. No non-monetary.
The only thing I really look for is like an
Effective Feedback for Performance Evaluation 78 Rutherford, Naylor, & Wheeler
around good performance?
Are there non-monetary incentives for good performance?
isn’t there. “atta-boy” or a good job.
I really enjoy my job and want students to succeed and if they succeed that is reward enough.
Individual
4. Motives How do you feel about providing verbal and written feedback to trainees?
I think it’s a very important part of the educational process. Three ways to learn: knowledge, skills, and behavior. Learning theory is reinforcement and that comes from counseling and feedback.
I think it’s a good thing because if you don’t give them feedback they don’t know what to do to improve or succeed. It goes both ways.
I don’t see anything getting in the way. As a brand new instructor in the classroom and asking for feedback from the students on performance. In the field, it’s not a problem on feedback on what they are doing because of my experience and expertise, but in the classroom not so much.
5. Capacity To Manager:
Does the individual lack the capacity or ability to learn?
No. No.
6. Knowledge/Skills
Do you believe the processes and procedures have remained consistent?
Do you feel you are expected to know too much information?
Changed. They are important and there has always been time allotted for formal feedback and evaluation. Over the 10 years the process has been polished and been refined to be better for the student and the instructor.
No, not too much information. The challenge with knowing for a Training Specialist is if you are going to effectively counsel the student, it’s important to know the student. If the student is performing well, it’s easy, but if they aren’t you
No.
Yes.
Effective Feedback for Performance Evaluation 79 Rutherford, Naylor, & Wheeler
have to get beyond the data to really understand how to counsel or encourage or connect with the student beyond what’s on the data sheet.
You may not interact with the student in all three areas so it can be hard to counsel students. I can make circles and boxes but don’t really know the student.
Effective Feedback for Performance Evaluation Rutherford, Naylor, & Wheeler 80
Appendix C: Survey Questions/Results Survey 1
Effective Feedback for Performance Evaluation Rutherford, Naylor, & Wheeler 81
Effective Feedback for Performance Evaluation Rutherford, Naylor, & Wheeler 82
Effective Feedback for Performance Evaluation Rutherford, Naylor, & Wheeler 83
Effective Feedback for Performance Evaluation Rutherford, Naylor, & Wheeler 84
Effective Feedback for Performance Evaluation Rutherford, Naylor, & Wheeler 85
Effective Feedback for Performance Evaluation Rutherford, Naylor, & Wheeler 86
Effective Feedback for Performance Evaluation Rutherford, Naylor, & Wheeler 87
Survey 2
Effective Feedback for Performance Evaluation Rutherford, Naylor, & Wheeler 88
Effective Feedback for Performance Evaluation Rutherford, Naylor, & Wheeler 89
Effective Feedback for Performance Evaluation Rutherford, Naylor, & Wheeler 90
Effective Feedback for Performance Evaluation 91 Rutherford, Naylor, & Wheeler
Appendix D: Extant Data Examples NLC’s mission/vision Northwest Lineman College will be a dynamic, visionary, and leading international educational institution providing the benchmark standard of safety, training, and products that exceed the expectations of customers in the power delivery and related industries. NLC’s core values Northwest Lineman College has an outstanding reputation throughout the power delivery industry, thanks to the high standards of its company, staff, and students. The Core Values and Mission Statement are used as the guiding principles by which NLC conducts training, business, and all endeavors. P.I.E. = PASSION, INTEGRITY, EXCELLENCE* PASSION Love what you do with PERSISTENCE, MOTIVATION, and the ATTITUDE to succeed. INTEGRITY Do the right thing—even when no one is watching—so TRUST, HONOR, and DEPENDABILITY are your hallmarks. EXCELLENCE Innovate always, leveraging ORIGINALITY and TEAMWORK, so your LEGACY of positive contribution to the industry is guaranteed. *(PIE is a mnemonic for the power formula, P (watts) = I (current) X E (voltage), which is a well-known and used formula in the power delivery industry):
Electrical Lineworker Program (ELP) Objective The ELP program objective is to provide training and education that ensures graduates are highly qualified for initial employment, highly successful in completing formalized apprenticeship and training programs, and are prepared to pursue long, rewarding careers as lineworkers.
ELP Performance Rubric and Coordinating Improvement Planners for Academics, Skills, and Behavior (Not included in this document)
Effective Feedback for Performance Evaluation 92 Rutherford, Naylor, & Wheeler
Appendix E: Job Aids
Advance Organizer Flow Chart
Rutherford, Naylor, Wheeler November 17, 2016 1
Worksheet Job Aid: Assessing Areas for Behavioral Improvement
Use this worksheet to collect and document data necessary to make performance assessment decisions for the behaviors criteria section on the ELP performance rubric. Note: This worksheet will also provide talking points and examples to discuss during the face-to-face performance evaluations with the student. Complete the steps listed below:
1. Determine if the Student File Packet contains all existing written information to assess behavioral performance: Table A If the Student File Packet: Then: Contains one or more spot evaluations for exceptional behavior and has absence and tardy counts listed
1. Assess the student using the ELP performance rubric form Hot Tip: The Student File Packet is likely complete and can be assessed using the ELP performance rubric form alone. It’s very rare that a student with spot evaluations for exceptional behavior will require additional consultations or information to assess behavior. If this does occur, consult with your Training Lead immediately to aid in completing the assessment.
Contains one or more spot evaluations for exceptional behavior but is missing absence and tardy counts 1. Ask the Academic Assistant to provide the absence and/or tardy counts. 2. Note any counts on the Student File Packet absence/tardy count sheet. 3. Assess the student using the ELP performance rubric form.
Hot Tip: Once you collect the absence/tardy counts, the Student File Packet is likely complete and can be assessed using the ELP performance rubric form alone. It’s very rare that a student with spot evaluations for exceptional behavior will have excessive absence/tardy counts and/or will require additional consultations or information to assess behavior. If this does occur, consult with your Training Lead immediately to aid in completing the assessment.
Is missing any one or combination of the following documentation: • Spot evaluations:
o For probation for violating behavior standards o For probation for not meeting written program requirements
1. Ask the Academic Assistant to check files/records to verify if the documentation exists or doesn’t. 2. Collect any existing documentation not previously provided by the Academic Assistant OR note that the
information doesn’t exist on Step 3. 3. Proceed to Table B under step
Is missing One or both of the following documentation (indicated by a blank space by either count on the absence/tardy sheet in the Student File Packet): • Absence Counts • Tardy Counts
1. Ask the Academic Assistant to provide the absence and/or tardy counts. 2. Note on the Student File Packet absence/tardy count sheet. 3. Proceed to Table B under step 2
2. Review the available behavioral data in the Student File Packet
Rutherford, Naylor, Wheeler November 17, 2016 2
3. Determine if there is enough existing behavioral data in the Student File Packet to make a decision (rate) the student’s behavioral performance:
Table B If the Student File Packet: Then: Has no existing spot evaluations for any one or combination of the following:
o Probation for violating behavior standards; o Probation for not meeting written program requirements; o Probation for excessive absence/tardy counts.
Use Table C to: 1. Document performance findings/notes for each behaviors criteria for the students you have personally
observed classroom and/or field; 2. Consult with other instructors to document performance findings/notes for each behaviors criteria for the
students you have NOT personally observed in the classroom and/or field. Contains one or more spot evaluations for probations for behavior that you didn’t personally fill out: Use Table C to:
1. Consult with the instructor who did fill out the spot evaluation to document performance findings/notes for the specified behavior criteria on the spot evaluation form.
Continued on page 3
Rutherford, Naylor, Wheeler November 17, 2016 3
4. Use the following tables to: a. Determine if behavioral issues exist for each behaviors criteria category; b. Document findings and examples; c. Rate student behavior on the ELP based on the findings and examples.
Table C: Safety
Behavior Criteria Category
Behavior Questions Record Findings Note answers below or indicate N/A if the criteria does not require additional information.
Record Examples Note examples of observed behaviors for question 1 and any other questions listed as necessary.
Rating Guidance
SAFE
TY
1. If the student has a spot evaluation for safety, is the student’s performance improving?
a. Yes: Examples b. No: Examples
2. Does the student always wear gaff guards?
3. Does the student always climb with proper gloves?
4. Does the student always wear required personal protective equipment (PPE)?
5. Has an injury or injuries occurred after coaching/feedback from instructor?
6. Are tailboards always completed per NLC standards/requirements?
If the answers to 3 of the 6 questions indicate ongoing poor performance and/or an unresolved spot evaluation(s) exists for safety violations, the student is performing at the Unacceptable level for safety. Otherwise the student is performing at the Expected level for safety.
1. Rate the student’s behavior for Safety using the information documented in the record findings space, the record examples space and the rating guidance information:
a. Circle the correct rating on the ELP performance rubric form;
b. Note here or highlight from the notes above the observed behaviors that substantiate the rating. (Note: this will also be useful during the face-to-face meeting with the student)
Rutherford, Naylor, Wheeler November 17, 2016 4
Table C: Attitude
Behavior Criteria Category
Behavior Questions Record Findings Note answers below or indicate N/A if the criteria does not require additional information.
Record Examples Note examples of observed behaviors for question 1 and any other questions listed as necessary.
Rating Guidance
ATT
ITUDE
1. If the student has a spot evaluation for attitude, is the student’s performance improving?
a. Yes: Examples b. No: Examples
2. Is student regularly negative or short-tempered with crew or others?
3. Does the student complain regularly and/or blame others?
4. Is the student regularly reluctant to do what’s expected in class, field or lab?
5. Does the student continue to regularly use foul language after being warned?
If the answers to 3 of the 5 questions indicate poor performance and/or an unresolved spot evaluation(s) exists for attitude violations, the student is performing at the Unacceptable level for attitude. Otherwise the student is performing at the Expected level for attitude.
1. Rate the student’s behavior for Attitude using the information documented in the record findings space, the record examples space and the rating guidance information:
a. Circle the correct rating on the ELP performance rubric form;
b. Note here or highlight from the notes above the observed behaviors that substantiate the rating. (Note: this will also be useful during the face-to-face meeting with the student)
Rutherford, Naylor, Wheeler November 17, 2016 5
Table C: Housekeeping
Behavior Criteria Category
Behavior Questions Record Findings Note answers below or indicate N/A if the criteria does not require additional information.
Record Examples Note examples of observed behaviors for question 1 and any other questions listed as necessary.
Rating Guidance
HOUS
EKEE
PIN
G
1. If the student has a spot evaluation for safety, is the student’s performance improving?
a. Yes: Examples b. No: Examples
2. Does student misplace name tag or doesn’t place on tables in classrooms without prompting
3. Does the crew or others regularly pick up after the student?
If the answers to 2 of the 3 questions indicate poor performance and/or an unresolved spot evaluation(s) exists for housekeeping violations, the student is performing at the Unacceptable level for housekeeping. Otherwise the student is performing at the Expected level for housekeeping.
1. Rate the student’s behavior for Housekeeping using the information documented in the record findings space, the record examples space and the rating guidance information:
a. Circle the correct rating on the ELP performance rubric form;
b. Note here or highlight from the notes above the observed behaviors that substantiate the rating. (Note: this will also be useful during the face-to-face evaluation meeting with the student).
Rutherford, Naylor, Wheeler November 17, 2016 6
Table C: Excellence Behavior Criteria Category
Behavior Questions Record Findings Note answers below or indicate N/A if the criteria does not require additional information.
Record Examples Note examples of observed behaviors for question 1 and any other questions listed as necessary.
Rating Guidance
EXC
ELLE
NC
E 1. If the student has a spot evaluation
for safety, is the student’s performance improving?
a. Yes: Examples b. No: Examples
2. Is the student always ready to perform competencies or certifications during verification/check-off weeks?
3. Does the student have 3 or more absences or 5 tardies?
If the answers to 2 of the 3 questions indicate poor performance and/or an unresolved spot evaluation(s) exists for attendance/tardy violations, the student is performing at the Unacceptable level for excellence. Otherwise the student is performing at the Expected level for excellence. NOTE: If the student has 3 or more absences or 5 or more tardies AND there is not already a spot evaluation for absence/tardy violations – the student MUST be marked Unacceptable in order to document notification and counsel to student.
1. Rate the student’s behavior for Excellence using the information documented in the record findings space, the record examples space and the rating guidance information:
a. Circle the correct rating on the ELP performance rubric form;
b. Note here or highlight from the notes above the observed behaviors that substantiate the rating. (Note: this will also be useful during the face-to-face evaluation meeting with the student).
i.
Rutherford, Naylor, Wheeler November 17, 2016 7
Table C: Integrity
Continued on page 8
Behavior Criteria Category
Behavior Questions Record Findings Note answers below or indicate N/A if the criteria does not require additional information.
Record Examples Note examples of observed behaviors for question 1 and any other questions listed as necessary.
Rating Guidance
INTE
GRI
TY
1. If the student has a spot evaluation for safety, is the student’s performance improving?
a. Yes: Examples b. No: Examples
2. Does student participate in community service projects?
3. Did student cheat, steal, or get caught in a lie?
4. Does student continue to use foul language after being warned?
If the answers to 2 of the 4 questions indicate poor performance and/or an unresolved spot evaluation(s) exists for integrity violations, the student is performing at the Unacceptable level for integrity. Otherwise the student is performing at the Expected level for integrity. NOTE: If the student cheated or was caught stealing – notify the Training Lead immediately to finish the evaluation and documentation.
2. Rate the student’s behavior for Integrity using the information documented in the record findings space, the record examples space and the rating guidance information:
a. Circle the correct rating on the ELP performance rubric form;
b. Note here or highlight from the notes above the observed behaviors that substantiate the rating. (Note: this will also be useful during the face-to-face evaluation meeting with the student).
Rutherford, Naylor, Wheeler November 17, 2016 8
5. If the student is rated Unacceptable in one or more criteria, complete the behavior improvement planner form. a. Circle the behavior criteria rated Unacceptable on the behavior improvement planner form; b. Indicate the improvement due date on the behavior improvement planner form where requested (typically 7 calendar days from the date of the face-to-face evaluation
meeting); c. Note the observed behaviors that substantiate the rating in the comments box on the behavior improvement planner form. d. If there are 3 or more areas rated as Unacceptable on the behavior improvement planner form, consult with the Training Lead to determine next steps.
6. Sign and date the ELP performance rubric form, and the ELP improvement planner if necessary, as indicated on the forms. Note: Once the formal performance evaluations conclude and the students’ have signed their evaluation forms and improvement planners (if applicable), make sure to provide a this completed worksheet along with they signed rubric/planner forms to the Academic Assistant so it’s included in future Student File Packets.
Effective Feedback for Performance Evaluation 101 Rutherford, Naylor, & Wheeler
References Chevalier, R. (2003). Updating the behavior engineering model. Performance Improvement, 42(5), 8-14. https://dx.doi.org/10.1002/pfi.4930420504