Running Head: EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

     

Evaluation Plan: Apple Keynote Tutorials

Christina Churchill EDCI 577

Purdue University October 4, 2013

     

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Executive Summary

Since the adoption of the Evaluation Plan: Apple Keynote Tutorials, Apple has continued to remain a dominant corporation in the computing world. Although Keynote does not lead the presentation software arena as Microsoft Powerpoint, it still continues to bring positive admiration in the Apple software environment. The focus of this evaluation plan was to ensure that Apple’s Keynote tutorials continue to portray Apple’s reputation of simplicity and ease of use. Set skill objectives were outlined for the participant to be able to accomplish in the Keynote tutorials.

The key finding in the Level 1 survey has been that the participants are

overall satisfied with the tutorial. According to the participants Level 1 survey, 95% of the results stated that the tutorials communicated effectively, were easy to navigate and benefited from the tutorial. Moreover, 98% of participants stated that they would recommend the tutorial to others. These results mark success in the reactionary arena.

Results in the Level 2 assessment showed that 100% of participants in the

experimental group scored 90% or higher compared to 100% of participants in the control group answered 75% or below. These variances solidify the successfulness of the video tutorials. Skill level was predominately assessed in Level 3.

The Level 3 Behavioral assessment shifted from skill-based results to the

overall objective of maintaining Apple’s reputation of simplicity and ease of use. These assessments were administered 3 months after Level 2 and reassessment continued every three months for a year. Although this assessment was difficult to directly see the correlation between the tutorial and the overall opinion of Apple software, there seemed to be a 7% increase in the positive perception of the Keynote product with the experimental group compared to the control group. It is difficult to pin point this to the tutorial since there are many elements that could skew this data.

Based on the key findings of this evaluation, the recommendations for this

program is to continue to offer it as an option for participants to use on an as need basis. Further development of tutorials is also recommended to continue the positive perception of Apple software product. Further research needs to be conducted to possibly implement a certification of completion of these products.

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Evaluation Goals and Scope

Instructional Product Description  

Apple’s innovation and simplicity has been in the forefront of technology. With the attempt to keep their products simple, Apple also provides short video tutorials to train their customers on Apple products. Most of their video clips are less than 3 minutes long, allowing the customer to quickly pinpoint what they want to learn. The videos feature a well-spoken male voice that guides the customer with step-by-step visual aids and procedures to accomplish the task of the video. The purpose of these videos is to ensure that the Keynote user is able to accomplish these tasks and to maintain the perception that using Apple products is simple and easy.

The focus of this Instructional Product Evaluation is on the advanced features of Apple Keynotes, a product of iWorks. Due to the short length of the video clips, the full section “Making Your Presentation Look Great” will be assessed. This section features 6 video clips varying from 1-2 minutes in length. Table 1 provides the topics and links. Table 2 provides a list of the objectives and duration of each video tutorial.

Table 1 Making Your Presentation Look Great in Keynotes Topics and Links to Video Tutorials • Using Instant Alpha to Remove a Background • Add Shadows and Reflections • Using the Mask Tool to Crop and Compose Images • Add Transitions Between Slides • Animate Objects Between Slides with Magic Move • Using Advanced Builds to Animate Objects On and Off Slides

(Apple, 2013) Table 2 Video Clip Objectives Video Clip Topic Objectives Duration Using Instant Alpha to Remove a Back Ground

Understanding instant alpha 1 min 12 s

Using instant alpha on a photo Using instant alpha on a graphic Add Shadows and Reflections

Add basic shadow to text 1 min 8 s

Customizing shadow in text Add reflection to photo

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Customize reflection to photo Using Mask Tool to Crop and Compose Images

Using mask tool to crop images 1 min 11 s

Using mask tool to compose images Adding Transitions Between Slides

Turning on transitions 1 min 54 s

Selecting effects of transition Setting transition duration Animate Objects Between Slides with Magic Move

Creating 2 consecutive slide with same image

1 min 2 s

Turn on magic move option Select duration of magic move Using Advanced Builds to Animate Objects On and Off the Slides

Selecting build in effect 2 min 28 s

Selecting build out effects Playing the slide effects Creating complex builds with

timings deliveries and effects

Purpose of Evaluation and Objective As a third-party consultant for Apple, Inc., the purpose of this evaluation is not only to ensure that the Keynote user is able to accomplish the tasks set forth by the objective of the tutorial, but also to maintain the perception of Apple products of being simple and easy to use. With this perception, Apple software product sales will continue to increase and maintain their reputation. Learner Analysis and Context

The target consumers for Apple’s Keynote product are all audiences (Apple, 2013). Therefore, since these videos have been designed for all Apple Keynote users, the target audience for these training videos is also all audiences. The video tutorials cater to audiences ranging from the middle school student user to the business professional, from the new Apple user to the tech-savvy college student. The vocabulary used throughout the video is basic as well as the step-by-step procedures. The context of the tutorial is typically on the participant’s computer, in participant’s selected environment. The participant will be able to choose a

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

comfortable setting and will have the ability to review the tutorial as necessary. In order to practice the steps in the tutorial, the participant must have access to an Apple computer and the Keynote software. Ideally, the participant will practice the skill immediately after the tutorial the steps that are taught in the video. This will help solidify the materials covered in the training videos.

Evaluation Process and Instruments

The process to evaluate these tutorials will entail Kirkpatrick and Kirkpatrick’s Four Levels of Evaluation. (2006) Level 1 survey will be administered immediately following the viewing of each tutorial. Level 2 assessments will be administered randomly once the participants complete all six modules of the tutorial, along with the experimental group of registered Keynote software users.

Level 3 survey is administered a month after all six modules and the Level 2 survey is complete. This will ensure that the participant had the opportunity to use the skills acquired in the tutorials. The Level 3 survey will be re-administered in 3-month intervals, up to 1 year past the original Level 3 survey.

These surveys and assessments will be continuous until 5,000 participants have been administered the first Level 3 survey. Although the timeframe for this is difficult to pinpoint, since there is no control on how many participants visit the tutorials and complete the surveys, according to historical website traffic data to the tutorial in the past year, it is expected to take approximately 6 months.

Once all data is collected, Level 4 of the evaluation will be immediate since the data collection is automated through Qualtrics.com. The Gantt chart on Appendix A shows the expected timeline of the project.

Level 1 Reaction Evaluations

Immediately following each video tutorial, each participant will be asked to take a quick online survey. This survey will be the Level 1 – Reaction Survey that will be administered by Qualtrics.com. According to Kirkpatrick & Kirkpatrick (2006), the Level 1 – Reaction Survey will help determine the effectiveness of the training videos and how they can be improved. Unfortunately, due to the limitation of the trial version of the Qualtrics.com account, the online version of the survey is currently unavailable.

Evaluation Instrument The online survey will consist of 7 questions covering the essential

questions targeting subject content, instructor’s communication skills, navigation, and overall perception of the tutorial, as recommended by Kirkpatrick & Kirkpatrick (2006). The Level 1 Reaction Survey can be found in Appendix B.

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Evaluation Procedure

The Level 1 Reaction Survey will be monitored on a regular basis to

ensure that the results are positive. Results are automatically complied by Qualtrics.com as participants submit the surveys. A sample of result reports are shown in Appendix C or at the following link: https://qtrial.qualtrics.com/CP/Report.php?RP=RP_9S3fFoYTbv8PxhX.

Analysis Process Once 5,000 participants have submitted their surveys, the results on

Qualtrics.com will be reviewed for detailed analysis. Questions 1-4 should maintain a mean of 2 or less. Question 5 should maintain a mean of 5 or more. Any drop in the recommended mean should be further investigated. Furthermore, the listed suggestions in Question 6 should be regularly reviewed for possible consideration of changes in the video. Question 7 email addresses will be collected in a database for possible use in the Level 2 Survey. Any cumulative results that do not meet the recommended standard mean should be further investigated for possible changes in the training program. Results of survey are reviewed with the training department to adjust or further develop the Apple tutorials. Positive and negative results are reviewed and thoroughly analyzed

Rationale

Although the overall knowledge gained is not assessed at Level 1, the overall reaction of the tutorial is essential to recognize in order to ensure that the tutorials are providing a positive impression on the Keynote product. If there are negative responses on the tutorial, changing the video can be an immediate solution. The impression of the product and the tutorial is critical in maintaining Apple’s reputation.

Level 2 Learning Evaluations

The survey to assess learning from the tutorial videos will be administered to random participants once they complete all 6 modules. Email addresses from this experimental group will be collected at the time of the Level 1 assessment. An email will be sent to the experimental group within one week of the completion of the 6 modules. The email will contain an online link to a survey administered by Qualtrics.com. A control group of registered Keynote users that have not viewed the training videos will be emailed with the link of the same survey taken by the experimental group. The survey will be administered in the same method, allowing the capability to open multiple screens as the experimental group. This

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

control group will allow the assessment the average user’s knowledge in the content presented in the videos. The total number of participants in the experiment group must be the same as the control group.

Evaluation Instrument

The survey will have 19 open-ended, multiple-choice or true/false questions, each question focuses on the objectives listed in Table 2. Participants will be able to open multiple screens on their computer to visualize the steps to help answer the survey questions. Learned skills are being assessed rather than memorization of content. Screen shots of the Level 2 Learning Assessment Survey is available in Appendix D or can be found at: https://qtrial.qualtrics.com/SE/?SID=SV_dg1rx0faO4rBLwx.

Evaluation Procedure The Level 2 Learning Surveys from the experimental and control group will

be compared to the correct answers. Some of the questions with open text boxes will require additional review due to the nature of possible correct variations of the answer. The scores of the control group and the experimental group will be compared assess the effectiveness of the tutorials. If minimal difference, 5% or less, is noticed and the survey scores are 90% or higher, the need for the training videos will be considered. The possibility of increasing the skill level in the tutorial could be an option. If survey results in the experimental group are 80% or below, the overall content and delivery of the video should be reconsidered. The goal of the overall difference between the experimental and control group should result in a difference of at least 10%.

Analysis Process Once 5,000 participants in the experimental group have completed the

Level 2 assessment, scores will be compared to the 5,000 control group participants. Results will be complied and graded on a 100-point scale. Individual answers will not be analyzed since the purpose of the assessment is to review the overall skill level. This assessment will review the overall success of skills mastered by the participant. Qualtrics.com will be used for detailed analysis of data since all information is automatically complied in the Qualtric’s online database.

Individual questions will be complied for the training department to assess. If the same question incorrect over 30% of the time in the experimental group, the training department will review the tutorial to fine-tune the video at the end of the evaluation study.

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Rationale

The control and experimental group is maintained to ensure that the increase in knowledge is due to the tutorials. The assessment of the increase of knowledge will be an indirect method of analysis to measure the participant’s satisfaction.

Level 3 Behavioral Evaluations The evaluation of the Level 3 Behavioral change is difficult to thoroughly evaluation due to the other variables affecting the possible results. The objective in the Level 3 assessment is to ensure that the participant was satisfied in the learning that took place and the overall Keynote product. The possibility of skewed results could be the satisfaction of the tutorial but not the actual Keynote software, or the dissatisfaction of the Apple hardware versus the Keynote tutorial. Due to these possible variables, the evaluation period and number of participants to be evaluated needs to be significant. The recommendation of 5,000 participants with the evaluation period spanning for 12 months in the Level 3 evaluation will assist in aligning the results to focus on the Keynote tutorial.

Evaluation Instrument

The survey consists of 10 questions ranging from multiple-choice, rating and open-ended answers. The focus on the survey will be on the participant’s satisfaction of the Keynote tutorial, along with the overall satisfaction and perception of the Keynote software and Apple products. The survey is available in Appendix D or https://purdue.qualtrics.com/SE/?SID=SV_3gEJVR9jGP2ZTil.

Evaluation Procedure

The survey will be emailed to Level 2 participants from the control and the experimental group three months after the completion of the Level 2 survey, then again every 3 months for a year. This will ensure the long-term assessment of the product. In order to motivate the participant to participate in the program, a $5 iTunes gift card will be offered if the participant completes all of the surveys. In order not to skew the data, Level 3 of the assessment is to be outsourced to a different company. This needs to be stated to the participant since then they are rewarded with a gift card. The survey will be administered using Qualtrics.com. Analysis Process

Analyzing data can begin once 20% of participants have submitted their online surveys. Qualtrics.com gives detail reports on each survey question. Each question will be tallied and graphed. Since most of the questions reflect the perception of Apple’s reputation through the tutorial and the Keynote software, the focus will be to maintain a 90% positive result.

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Training department and upper management will review the Level 3 results to assist in the decision-making of the continuation of this and other video tutorial programs.

Rationale

By evaluating the Level 3 Behavioral assessment, the long-term affects of the tutorial are being analyzed. The Level 3 is conducted 3 months after the completion of the Level 2, followed by the same survey every three months for a year. Using this method will assist the overall satisfaction of the tutorial. Although some of the questions do not pertain to the tutorial, it focuses on Apple’s reputation. There will be some skewing of data since Apple’s reputation could cover many different elements. The evaluation needs to ensure that the tutorial does not cause a long-term negative effect but builds on the positive Apple image.

Level 4 Results Evaluation All results will be compiled through Qualtric.com offering a variety of reports to be analyzed. Immediate feedback will be available as the surveys are being completed. Various evaluation levels will be completed at various times during the evaluation period. Moreover, as noted in Appendix C, each question will be assessed individually. The difficulty in the evaluation of the overall return on investment (ROI) is the other variables influencing the results such as new operating, software or hardware releases. Product perception through advertising also plays a significant factor. Level 1 will be evaluated for the immediate reaction of the tutorial. The 7 questions are simple to assess and analyze. Immediate results are available for simple changes to the training videos such as voice and ease of use. Level 2 focuses on the mastery of skill level. Overall composite scores are collected graphed and analyzed on Qualtrics.com. Due to the nature of the overall goal to maintain Apple’s reputation of simple and easy to use, individual questions will not be assessed. Level 3 surveys will assess each individual questions. The goal is to ensure that the overall perception of the Apple tutorial, product and company is positive.

Data Collection Data collection will take place in real-time though Qualitrics.com. The results of the assessments of all levels can be accessed through the Qualtric.com account at any point in the evaluation period. Once the participant

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

submits the survey, it automatically sends the information into the Qualtric.com database. All typed answers will be complied and reviewed manually. Multiple choice and rating questions will be tallied.

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

References Keynote ’09. (n.d.). Retrieved September 11, 2013, from Apple, Inc. Website: http://www/apple.com/iwork/keynote Keynote tutorial: Making your presentation look great. (n.d.) Retrieved September 9, 2013 from Apple, Inc. Website: http://www.apple.com/iwork/tutorials/keynote/tutoriallist.html#keynote Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating Training Programs. San

Francisco: Berrett-Koehler Publishers, Inc.

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Appendix A Gantt Chart of Evaluation Timeline Further detail of the Gantt Chart is available at https://app.smartsheet.com/b/home?lx=Nj6b5YAg7PaucJdyerZBdA once user accepts share invitation. To receive a share invitation, send request to churchic@purdue.edu.

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Gant Chart of Evaluation Timeline (continued)

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Appendix B Level 1 Reaction Survey  Due  to  the  limitations  of  the  trial  account  from  Qualtrics,  the  link  to  this  survey  is  not  accessible.    

Screen Shot 1 of Level 1 Reaction Survey

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Screen Shot 2 of Level 1 Reaction Survey

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Appendix C Sample of Result Report from Level 1 Reaction Assessment Full details available at: https://qtrial.qualtrics.com/CP/Report.php?RP=RP_9S3fFoYTbv8PxhX

9/15/13 3:39 PMQualtrics Survey Software

Page 1 of 3https://qtrial.qualtrics.com/CP/Report.php?RP=RP_9S3fFoYTbv8PxhX

Initial Report Last Modified: 09/15/2013

1. The topics covered in the video were relevant to learning Keynote.

1 Strongly Agree 6 100%

2 Agree 0 0%

3 Neither Agree nor Disagree 0 0%

4 Disagree 0 0%

5 Strongly Disagree 0 0%

Total 6

# Answer Bar Response %

Min Value 1

Max Value 1

Mean 1.00

Variance 0.00

Standard Deviation 0.00

Total Responses 6

Statistic Value

2. The video was able to communicate effectively.

1 Strongly Agree 1 17%

2 Agree 2 33%

3 Neither Agree nor Disagree 2 33%

4 Disagree 1 17%

5 Strongly Disagree 0 0%

Total 6

# Answer Bar Response %

Min Value 1

Max Value 4

Mean 2.50

Variance 1.10

Standard Deviation 1.05

Total Responses 6

Statistic Value

3. The video allowed you to easily navigate throughout the content.

1 Strongly Agree 2 33%

2 Agree 2 33%

# Answer Bar Response %

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Appendix D Level 2 Learning Survey Level  2  Learning  Survey  is  available  at: https://qtrial.qualtrics.com/SE/?SID=SV_dg1rx0faO4rBLwx Screen Shot 1 of Level 2 Learning Survey

         

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Screen Shot 2 of Level 2 Learning Survey

Screen Shot 3 of Level 2 Learning Survey  

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Screen Shot 4 of Level 2 Learning Survey  

Screen Shot 5 of Level 2 Learning Survey  

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Screen Shot 6 of Level 2 Learning Survey  

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

Appendix D Level 3 Behavioral Survey Level  3  Behavioral  Survey  is  available  at:  https://purdue.qualtrics.com/SE/?SID=SV_3gEJVR9jGP2ZTil

Screen Shot 1 of Level 3 Behavioral Survey  

EVALUATION PLAN: APPLE KEYNOTE TUTORIALS

 

Screen Shot 2 of Level 3 Behavioral Survey