Formative Evaluation Plan Transporting Chemicals 1 Review Phase # of Reviewers Main Purpose 1. Expert Review 2-3 Subject Matter Experts Content Accuracy 2. Pilot Study (1 to 1) 1 Trainee, 1 instructor* Program Usability 3. Pilot Study (group) 6 Trainees, 1 instructor* Program Usability 4. Editorial Review 2-3 Editors Program Readability Expert Review Pilot Studies Editorial Review *The same trainers for both 1 to 1 and group pilot studies would be ideal.
2. Pilot Study (1 to 1) 1 Trainee, 1 instructor* Program Usability
3. Pilot Study (group) 6 Trainees, 1 instructor* Program Usability
4. Editorial Review 2-3 Editors Program Readability
Expert Review Pilot Studies Editorial Review
*The same trainers for both 1 to 1 and group pilot studies would be ideal.
Expert Review Formative Evaluation
A needs assessment was conducted in October, 2012. At this time I spoke with glove box heads from three separate research groups at
McGill University. A pre-meeting questionnaire was also distributed. (See results below) It was decided by the feedback received and in conjunction with the project sponsor
(Jeffrey Quesnel) to make the training accessible to beginner level frequent and beginner level infrequent users and to target four main training areas:
a. Using the glove box computer. b. Bringing chemicals into the glove box. c. Bringing chemicals out of the glove box. d. Preparing the workspace for the next user.
Decision-Making to Date
Identified Performance Gaps from Questionnaire # of People who Identified this
Not preparing the workspace for the next user 3
Using solvents and/or chemicals inappropriately 3
Increasing O2 or H2O levels 2
Not repairing holes in the gloves 2
Inability to do maintenance 1
Tightening valves and lids too tightly or too loosely 1
Mistakes in using the vacuum pump 1
Issues related to users overlapping 1
Identified Training Issues from Questionnaire # of People who Identified this
Students lack a basic understanding of the central concepts of glove boxes
4
The glove box is a common space used by multiple users 2
Not all techniques translate across labs & equipment 1
No consistent training protocol exists for glove box training 1
Not sure of the student’s prior knowledge 1
The trainer lacks knowledge himself or herself in some areas
1
Some techniques have very complex procedures 1
There is lots of information to cover in a limited timeframe 1
2
Expert Review Formative Evaluation
In order to ensure that the content being prepared for the glove box training program is accurate, logical, and safe. I will be asking several subject matter experts (SMEs) to review the content produced to date. A pilot study & editorial review will also be conducted.
For the purpose of this review a small questionnaire has been created to direct your attention to important aspects of the project, with a place for general feedback at the end of sections 2 & 4.
Your honest and full feedback is welcomed! The more information you provide the more likely your comments will translate into a better training experience for everyone involved, so please don’t be shy!
Should you have any questions, or should you like to follow up further on a particular point, please contact Jessica Haak by e-mail, [email protected] or by phone, 514.589.4251.
Background Information
Section One: Learner Package Accuracy
Instructions: Please review the materials making sure to assess each of the elements below. It is asked that you annotate the document using the “review comments” function in PowerPoint, located within the tools menu.
1. Safety information in the introductory section.
4. Sequence of steps relating to bringing a chemical into the glove box.
5. The 8 steps listed in relation to bringing a chemical into the glove box.
6. Bringing chemicals into the glove box safety review.
7. The 6 steps listed in relation to bringing a chemical out of the glove box.
8. Bringing chemical out of the glove box safety review.
9. The 6 steps listed in relation to preparing the glove box for the next user.
10. Worksheets 1, 2 & 3
11. Videos 1, 2, 3 & 4
12. Evaluation Section
13. Glossary
14. Job Aid
3
Expert Review Formative Evaluation
Section Three: Instructional Package Accuracy
Instructions: Please answer the questions below in a separate document & submit them to the instructional designer.
1. What 3 words would you use to describe the learning package you just viewed?
2. Name the elements that would think are most important to preserve in the final draft.
3. Name the elements that you would most like to see improved.
4. Name the visuals that you most appreciated.
5. Name the visuals that you would like to see improved.
6. What topic do you think was best articulated in the package?
7. What topic do you think was worst articulated in the package?
8. Were there any safety issues that you noted? If so, what advice would you give to
improve this issue?
9. How did you find the sequence of the program? Was it logical? If not, what advice
would you give to improve this issue?
10. Did you find it to navigate through the content? If not, what advice would you give
to improve these issues?
11. Please provide your general feedback on the learning package.
Section Two: Learning Package Overall Impressions
Instructions: Please review the materials making sure to assess each of the elements below. It is asked that you annotate the document using the “review comments” function in PowerPoint, located within the tools menu.
1. Instructions for the teaching the glove box computer.
2. Instructions for teaching how to bring a chemical into the glove box.
3. Instructions for teaching how to bring a chemical out of the glove box.
4. Instructions for teaching how to prepare the glove box for the next user.
5. Instructors for conducting the evaluations.
4
Expert Review Formative Evaluation
Instructions: Please answer the questions below in a separate document and submit them to the instructional designer.
1. What 3 words would you use to describe the learning package you just viewed?
2. Which instructions do you think would be most successful in a training setting?
3. What instructions do you think would be least successful in a training setting?
4. Is there any content that you would like to see added to the instructional package?
5. Is there any content that you would like to see removed from the instructional
package?
6. Were there any safety issues that you noted? If so, what advice would you give to
improve this issue?
7. How did you find the sequence of the instructor package? Was it logical? If not,
what advice would you give to improve this issue?
8. Please provide your general feedback on the instructional package.
Pilot Study (Trainee & Trainer Information) Formative Evaluation
Background Information
The purpose of this pilot study is to assess how easy it is to read through and use the content in the glove box training program.
Through this process it is hoped that any procedural gaps, poor labeling, logistical problems, and inappropriate teaching strategies written into the program are identified.
To best accomplish this, it is asked that you follow the steps in the training to identify problem areas (in the event that you notice any safety-related issues please always air on the side of logic & caution).
Your honest and full feedback is welcomed! The more information you provide the more likely your comments will translate into a better training experience for everyone involved, so please don’t be shy!
Lastly, it is important that trainees in the pilot study are able to complete the entry level tasks prior to participating.
Should you have any questions, or should you like to follow up further on a particular point, please contact Jessica Haak by e-mail, [email protected] or by phone, 514.589.4251.
Task ✔
Name the major parts of the glove box. ✔
Label a basic diagram of a glove box. ✔
Explain how the value on the vacuum gauge relates to the pressure inside the antechamber.
✔
Explain how the glove box removes oxygen and water from its atmosphere. ✔
Explain what types of chemicals are incompatible with the O2 purifier in the glove box.
✔
Explain what each of the four circular icons on the glove box computer main menu represent in relationship to the glove box.
✔
Explain what types of reactions are unsafe or impossible to conduct in a glove box.
✔
6
Instructions: Marking an ‘X’ in the appropriate column indicate whether you strongly disagree (SD), disagree (D), are neutral to (N), agree (A) or strongly agree (SA) with the following statements.
Statement SD D N A SA
1. The learner package was easy to follow. SD D N A SA
2. The visuals added to the experience of the program. SD D N A SA
3. I believe this program could be completed in 30 minutes. SD D N A SA
4. I found the order of the sections logical. SD D N A SA
5. I like the red highlighting of safety points. SD D N A SA
6. I found the large job aid poster useful. SD D N A SA
7. I thought the activities in the program engaging. SD D N A SA
8. I found the general layout of this program unappealing. SD D N A SA
9. The videos in the program were useful. SD D N A SA
10. The order of the elements in this program were not logical.
SD D N A SA
11. The tone of the writing in the program was appropriate. SD D N A SA
12. I found the introduction prepared me for the training. SD D N A SA
13. This program was hard to follow. SD D N A SA
14. The evaluation preparation section was useful. SD D N A SA
15. The material in this program was hard to follow. SD D N A SA
16. I found the glossary was useful. SD D N A SA
17. I would use the materials in the learner package after the training was done to help me in my work.
SD D N A SA
Rate Your Training Pilot Study (Trainee) Formative Evaluation
7
Pilot Study (Trainee) Formative Evaluation
Instructions: Please check off the boxes that apply, and provide additional feedback in the ‘other’ section.
1. Which sections of the program did you find confusing?
2. Which sections of the program did you find well-articulated?
Table of Contents Section 1 Evaluation Prep Page
Course Info Page Section 2 Worksheets
Training Prep Page Section 3 Videos
Safety & Materials Page Section 4 Glossary
Why did you find these sections confusing? Feel free to include other sections not listed above. __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Table of Contents Section 1 Evaluation Prep Page
Course Info Page Section 2 Worksheets
Training Prep Page Section 3 Videos
Safety & Materials Page Section 4 Glossary
Why did you feel that these sections were well-articulated? Feel free to include other sections not listed above. __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
8
Pilot Study (Trainee) Formative Evaluation
Instructions: Please check off the boxes that apply, and provide additional feedback the ‘other’ section.
3. What three words would you use to describe the program?
Instructions: Marking an ‘X’ in the appropriate column indicate whether you strongly disagree (SD), disagree (D), are neutral to (N), agree (A) or strongly agree (SA) with the following statements.
Statement SD D N A SA
The one-on-one training notes were logical. SD D N A SA
The in-class portion of this learning program could be completed in 30 minutes.
SD D N A SA
The layout of the instructor package was logical. SD D N A SA
I found the quick notes useful. SD D N A SA
I visuals for this program were difficult to read. SD D N A SA
The large job aid for this program was confusing. SD D N A SA
The group adaptation notes were useful. SD D N A SA
The program worked better 1 on 1 than in groups. SD D N A SA
The layout of this guide was user friendly. SD D N A SA
I was confused by the layout of the instructor guide. SD D N A SA
The order of sections in this guide was logical. SD D N A SA
I found the evaluation descriptions hard to follow. SD D N A SA
The program worked better in groups than in 1 to 1. SD D N A SA
This guide appeared complete to me. SD D N A SA
I was confused by some of the terms in the instructor guide.
SD D N A SA
The introduction helped prepare me to teach the program. SD D N A SA
This guide was missing critical sections. SD D N A SA
Rate Your Training
10
Pilot Study (Instructor) Formative Evaluation
Pilot Study (Trainer) Formative Evaluation
Instructions: Please check off the boxes that apply, and provide additional feedback in the ‘other’ section.
1. Which sections of the program did you find confusing?
2. Which sections of the program did you find well-articulated?
A glove box is a large piece of equipment roughly 7’ deep, 14’ long and 6’ tall.
A glove box is used to conduct chemical reactions that cannot be done in an oxygen or water saturated environment.
An antechamber with an inner and outer door helps to transport chemicals and equipment in and out of the box.
While in the antechamber, the oxygen and water can be “cycled out” and the chemicals or equipment can then be transported into the box.
What is a Glove Box?
Background Information
Although you may not be familiar with chemistry and/or glove box equipment your observations will help to ensure that the layout, editing and readability of the program are logical and accurate.
Your attention to detail is very important for the success of the program. Please note all anomalies to the best of your abilities.
Should you have any questions, or should you like to follow up further on a particular point, please contact Jessica Haak by e-mail, [email protected] or by phone, 514.589.4251.
13
Editorial Review Formative Evaluation
Instructions: Please review the copy making sure to assess each of the points listed below on all of the pages of the program. It is asked that you annotate the document using the “review comments” function in PowerPoint, located within the tools menu. Additionally please review videos 1, 2, 3 & 4 including the credit sections.
