Web viewThe instructional design process parallels other processes of development when observing the...

Dalia Allencherry

EDTECH 503 (Summer 2013)

Instructional Design Project

Submitted to: Dr. Jennifer Freed

August 4, 2013

Project Title: Usability Testing for Websites

Copyright © 2013, Dalia Allencherry

EDTECH 503 – Instructional Design Project

Table of Contents Page

Reflection Synthesis paper 3Part 1. Topic 4 Part 1a. Goal statement 4 Part 1b. Audience description 4 Part 1c. Rationale 4Part 2. Analysis Report 6 Part 2a1: Needs assessment survey 6 Part 2a2: Needs assessment data 7 Part 2b1: Learning context description 8 Part 2b2: Transfer context description 8 Part 2c: Description of Learners 9 Part 2d: Task Analysis Flow Chart 10Part 3. Planning 10 Part 3a: List of instructional objectives 10 Part 3b: Objectives matrix table 12 Part 3c: ARCS table 17Part 4. Instructor Guide 19Part 5. Learner Content 23 Part 5a: Learning materials 23 Part 5b: Assessment materials 23 Part 5c: Technology tool rationale 23Part 6. Formative Evaluation Plan 24 Part 6a: Expert review plan 24 Part 6b: One-to-one review plan 25 Part 6c: Small group evaluation plan 25 Part 6d: Field trial plan 26Part 7. Formative Evaluation Report 26 Part 7a: Evaluation survey 26 Part 7b: Report of expert review 27 Part 7c: Designer’s response to review 27Part 8. Standards Grid 28Appendices 36

Copyright © 2013, Dalia Allencherry 2


Reflection Synthesis Paper

The instructional design process parallels other processes of development when observing the

broad stages of project inception, planning, and completion. For example, one can illustrate the

instructional design process via the lens of an architect. Akin to an instructional designer, an

architect commences with an inspiration: the desire to create a functional and aesthetic physical

structure to be used and enjoyed by society. Likewise, an instructional designer beckons her

faculties to generate novel methods for conveying knowledge. Such knowledge should also

serve a practical purpose and be appealing and beneficial to the learners that will eventually use

the materials. Next, an architect spends long hours at the drafting table--the blueprints should be

just right, they should be scalable to real space, and should comply with local building codes.

Similarly, an instructional designer maps out her plans and creates design documents. The

architect then fashions a miniature model of the building, presents it to the client, and makes

changes based on the feedback. Similarly, an instructional designer presents the training to a

sample audience, collects feedback, and makes the appropriate changes. These stages of

development will soon produce real platforms in order to engage actual students. However, such

instruments must fulfill agreed upon standards and survive the scrutiny of subject matter experts.

Finally, the day comes when the structure is erected, the ribbon is cut, and the space is utilized

for its intended purpose. The building is enjoyed by society and the architect is pleased having

journeyed with the idea from inception to completion. Likewise, the instructional designer is

gratified by harnessing an idea, guiding it through the planning stages while satisfying strict

standards, and ultimately releasing the product to the intended audience.


EDTECH 503 – Instructional Design ProjectI have worked as an instructional designer in the past. However, in this course I learned many

things that I should have used on the job, but was not aware of. This course was eye opening for

me, especially the ID project. I never paid attention to the motivational factor of learners prior to

this course. Therefore, I made no attempts to include motivational strategies in my design

documents and storyboards. My biggest takeaway from the course was learning about

motivational strategies. Sometimes I struggled with generating strategies to motivate learners

during the project. The course activities were extremely useful and I learned a lot from my

classmates’ posts and feedback. Even though I have submitted my work for SME reviews, I

never conducted one-to-one evaluations, group evaluations, or field trials prior to this course.

Now, I understand the importance of these evaluations. Extensive learner analysis and its

significance is another prime takeaway. In the past, I collected data about the target audience

from the client for most of my projects (i.e., the company or employer of the target audience paid

me to create the training materials and provided relevant data). I now grasp the importance of

learner analysis and how it should be conducted properly. Overall, this course was a great

opportunity for me to ‘unlearn’ activities that I performed poorly and to learn the appropriate

methods of instructional design processes.

There will be many occasions to incorporate knowledge from the course in my job. I review test

items and manage a team of content specialists that creates test items. I can apply my newfound

skills in learner analysis to effectively analyze the target audience and design assessments that

test the objectives. I am responsible for providing training to my team members based on their

needs. This arrangement will enable me to transfer my knowledge on the instructional design

process to design training for my team members.

Part 1: Topic

Part 1a: Goal statement

Learners will acquire the knowledge and conduct usability tests of a website.


EDTECH 503 – Instructional Design ProjectPart 1b: Audience description

The learners are web designers and developers of the company ALT. They have either an

associate degree or a bachelor’s degree in web design or a related field with an experience

ranging from three to eleven years.

Part 1c: Rationale

The sales through the company website are not improving as expected even after rigorous

marketing efforts. Therefore, the senior management team of ALT decided to take a look at the

factors that may stop potential customers from making a purchase through the website. They

identified one of the reasons as lack of ease to use the website. Usability analysis or testing was

never a part of the website development. In this case, the corporate decided to provide training to

the website designers and developers on usability testing and incorporate usability tests in the

regular work schedule. This course introduces the concept of usability to the learners and

provides them resources to conduct usability tests. It also gives them a chance to practice various

methods of usability tests in an instructor-monitored environment. This gives them the

opportunity to get feedback from the instructor and the peers during the mock sessions.

The overall project consists of about 80% supplantive and 20% generative strategy. The

instruction is mainly supplantive because of the procedures that need to be followed for usability

testing. Even though certain procedures need to be followed, the learners have opportunities to

use their interpretation of the information and detail to decide on the questions they want to ask

during interviews, website features they want to cover, type of method to choose, and the setting

in which they want to conduct the test. The primary type of learning that is concentrated on this

course is procedural. The instruction involves procedures to conduct various usability testing

methods.

The goal of the project is classified under three different strategies. They are strategies for

declarative knowledge instruction, strategies leading to concept learning, and strategies for

instruction to learning procedures, which is the major instructional strategy. The learners have

very minimal prior knowledge in usability testing. The instruction involves strategies of


EDTECH 503 – Instructional Design Projectdeclarative knowledge for the learners to understand the definition, context, purpose, and

importance of usability tests. The strategies of concept learning are used for the learners to recall

the reasons behind usability testing and apply them in the context of a website to determine the

most suitable usability test method for the site. The instruction of concept at an abstract level is

suitable in the context of this project. The strategies for instruction to learning procedures are

used for learners to understand and perform the step-by-step procedures involved in conducting

various usability testing methods.

Part 2: Analysis Report

Part 2a1: Needs assessment survey

The following need assessment survey was sent out to 13 people to collect data for the need

analysis.

Name:

Job title:

Male or Female:

Age:

1. How many years of experience do you have in website design or development?

2. What degree(s) do you possess related to website design or development?

3. Define usability testing.

4. Have you ever conducted a usability test?

5. Have you ever conducted a user acceptance test?

6. What is the difference between a usability test and a user acceptance test?

7. What motivates you the most about conducting usability tests for websites?

8. How motivated are you about conducting usability test? Rate 1-5

9. How relevant do you think is the need to conduct usability analysis? Not relevant,

Relevant, Very relevant

10. How confident are you that you can successfully perform usability analysis? Rate 1-5


EDTECH 503 – Instructional Design Project11. What characteristics of usability analysis interest you the most?

12. How important is usability testing when your job responsibilities are considered? Not

Important, Important, Very important

13. Do you expect any constraints regarding you performing usability tests?

14. What consequence do you expect if you perform an incorrect usability test?

15. What are the goals you plan to achieve by taking the usability test training?

16. How do you feel about the way the company offers trainings?

17. How seriously do you usually consider trainings offered by the company? Not serious,

Serious, Very serious

18. How often does the company follow-up with the training sessions provided? Never,

Sometimes, Always

Part 2a2: Needs assessment data

The need analysis survey revealed that the work-related experience of the learners range from

three to eleven years. They possess either an associate degree or bachelor’s degree in a field

related to their work. The learners have very low prior knowledge in usability testing. None of

them have conducted a usability test even though two of them correctly defined usability testing.

Some of the learners have conducted user acceptance testing, however, forty percent of the

learners think of usability testing a part of user acceptance testing.

The learners showed low motivation levels. None of the learners appeared to be highly motivated

(level 5). Eight-five percent of the learners think usability analysis is relevant to their work. The

company rarely follows-up on training sessions provided; therefore the measures to retain the

knowledge of usability testing need to be included in the course. Even though eight percent of

the examinees take trainings seriously, none of them seem to take it very serious. None of them

identified any constraints that could affect their performance in conducting usability test other

than proper training. As shown below, most of the learners are not sure about the importance of

usability testing when their job responsibilities are considered.


EDTECH 503 – Instructional Design ProjectImportance of usability testing when your job responsibilities are considered

Not important

Important

Very important

Don't know

Part 2b1: Learning context description

The learning environment is a training room in the company compound. The room is well lit,

well ventilated, and has enough room for learners to move around freely. There are desks and

seating for fifteen people in the room. There are fifteen computers with internet connection for

the learners to use. A laptop, a projector, and a display screen are available in the room. Note

books, pens, and markers are available in the room. The instructor is a usability expert. Learners

will receive emails about the instructor of the course and basic information about usability, in

advance. Thirteen employees will attend the course. The course will be delivered from 1 p.m. to

4 p.m. Water, coffee, and ice tea will be available throughout the course. Some snacks will be

provided during a break at 2:30 p.m.

Part 2b2: Transfer context description

The learners will transfer the knowledge and skills learned to their daily work. As a part of a new

corporate-wide initiative to improve processes, usability will be added to the regular work

schedule of the learners. Usability testing is scheduled to be conducted every six months as a part

of the regular work schedule for the website development team. It will also be conducted in the

case of any major revisions to an existing website. Additionally, the learners will subscribe to

feeds and newsletters related to usability, which will enable them to build on their knowledge


EDTECH 503 – Instructional Design Projectrelated to the topic and apply it in their usability analysis and testing. The result of the transfer of

knowledge and skills from the usability testing will be measured using Return on Investment

(ROI) calculations that involve the cost of the training and the change in sales through the

updated website after usability testing. Web analytics studies related to the website before and

after usability testing will also measure the transfer of knowledge and skills as a result of the

training.

Part 2c: Description of Learners

The leaners are web designers and developers of the company ALT. They possess either an

associate degree or a bachelor’s degree in a field related to their work. They have a work

experience of three to eleven years in the field. The experience related to website design and

development will be at the benefit of the learners as usability testing deals with various parts of a

website. The learners consist of 30% women and 70% men with an age range of twenty-eight to

fifty years. They show low motivation and very minimal prior knowledge of the training topic.

Define usability testing

Usability testing definition0

1

2

3

4

5

6

7

8

CorrectPartially correctWrong



Part 2d: Task Analysis Flow Chart

Click here to open the graphic in a window.

Part 3: Planning

Part 3a: List of instructional objectives

1 Learners will explain usability testing

1.1 Learners will define usability testing in their own words

1.2 Learners will explain the purpose of usability testing

1.3 Learners will explain the importance of usability testing

1.4 Learners will list three methods of usability testing

1.5 Learners will list the characteristics of a website that need to be evaluated

1.6 Learners will explain hallway usability testing

1.7 Learners will explain remote usability testing

2 Learners will describe usability testing using interviews

2.1 Learners will explain usability testing using interviews in their own wordsCopyright © 2013, Dalia Allencherry 10

https://docs.google.com/file/d/0B_bNv8YQaboNNnpBTm5SUkxJTEE/edit?usp=sharing

EDTECH 503 – Instructional Design Project2.2 Learners will recall the advantages of usability testing using interviews

compared to other methods

3 Learners will describe A/B usability testing

3.1 Learners will explain A/B testing in their own words

3.2 Learners will recall the advantages of A/B usability testing compared to other

methods

4 Learners will describe usability testing using an eye tracking software

4.1 Learners will explain usability testing using an eye tracking software

4.2 Learners will recall the advantages of usability testing using an eye tracking

software compared to other methods

5 Learners will apply usability testing using interviews

5.1 Learners will identify participants

5.2 Learners will identify the observer

5.3 Learners will create the scenarios to test

5.4 Learners will schedule the interviews with participants

5.5 Learners will prepare the test room

5.6 Learners will greet the guest

5.7 Learners will inform the purpose of the usability test to the participant

5.8 Learners will collect answers to the pretest questionnaire

5.9 Learners will inform the scenarios to perform to the participants

5.10 Learners will collect the data

5.11 Learners will collect answers to the posttest questionnaire

5.12 Learners will thank the participant

5.13 Learners will collect data from the observer

6 Learners will apply A/B usability testing

6.1 Learners will identify participants

6.2 Learners will set two wireframes or interfaces ready for testing

6.3 Learners will develop questions for the participants to answer

6.4 Learners will identify participants to conduct the test

6.5 Learners will inform the participants of the questions


EDTECH 503 – Instructional Design Project6.6 Learners will schedule the participants to take the test

6.7 Learners will collect the answers to the questions

7 Learners will apply usability testing using an eye tracking software

7.1 Learners will identify the software to use

7.2 Learners will develop the scenarios for the participants to perform

7.3 Learners will identify the participants

7.4 Learners will set the interface to test

7.5 Learners will schedule the participants to take the test

7.6 Learners will collect the data

8 Learners will summarize the results of usability testing

8.1 Learners will make conclusions for each test goal

8.2 Learners will summarize biographic information of the participants and other

metadata

Part 3b: Objectives matrix table

Learning

Objectives

(a)

Bloom’s

Taxonomy

Classification (b)

Format of

Assessment (c)

Description of

test form (d)

Sample items

(e)

1.1 Knowledge Paper-and-Pencil Short Answer Define usability testing.

1.2 Comprehension Paper-and-PencilMultiple-choice

question

Choose the correct responses

of a multiple-choice multiple

response (MCMR) question

on the purpose of usability

testing.

1.3 Comprehension Paper-and-Pencil Short AnswerExplain the importance of

usability testing.

1.4 Knowledge Paper-and-Pencil Multiple-choice

question

Choose the correct responses

of a multiple-choice multiple

responses (MCMR) question


EDTECH 503 – Instructional Design Projectto select the three methods of

usability testing.

1.5 Knowledge Paper-and-PencilMultiple-choice

question

Choose the correct response

of a MCSR question on the

characteristics of a website

that need to be evaluated.

1.6 Comprehension Paper-and-Pencil Short answerWhat is hallway usability

testing?

1.7 Comprehension Paper-and-Pencil Short answerWhat is remote usability

testing?

2.1 Comprehension Paper-and-PencilMatch/Drag and

drop

Match the type of test to its

correct characteristic.

2.2 Knowledge Paper-and-Pencil Short answer

Explain the advantages of

usability testing using

interviews over other

methods.


drop




Explain the advantages of

A/B usability testing over

other methods.


drop




Explain the advantages

usability testing using an eye

tracking software over other

methods.

5.1 Application Performance Simulation with a

checklist

Learners will identify

participants as a part of a


EDTECH 503 – Instructional Design Projectsimulation.

5.2 Application Performance

Simulation with a

checklist

Learners will identify an

observer as a part of the

simulation.


Simulation with a

checklist

Learners will create the

scenarios to test as a part of

the simulation.


Simulation with a

checklist

Learners will schedule the

interviews with participants

as the part of the simulation.


Simulation with a

checklist

Learners will prepare the test

room as a part of the

simulation.


Simulation with a

checklist

Learners will greet the

participant as a part of the

simulation.


Simulation with a

checklist

Learners will inform the

purpose of the usability test

to the participant as a part of

the simulation.


Simulation with a

checklist

Learners will collect answers

to the pretest questionnaire

as a part the simulation.


Simulation with a

checklist


scenarios to perform to the

participants as a part the

simulation.

5.10 Application PerformanceSimulation with a

checklist

Learners will collect data as

a part the simulation.

5.11 Application Performance Simulation with a Learners will collect answers Copyright © 2013, Dalia Allencherry 14

EDTECH 503 – Instructional Design Projectchecklist to the posttest questionnaire

as a part the simulation.


Simulation with a

checklist

Learners will thank the

participant as a part the

simulation.


checklist

Learners will collect data

from the observer as a part

the simulation.

6.1 Application PerformanceObservation with

a checklist


participants to conduct A/B

testing for the scenario

provided by the instructor.


a checklist

Learners will set two

wireframes or interfaces

ready for A/B usability test

for the scenario provided by

the instructor.


a checklist

Learners will develop

questions to conduct A/B




a checklist


participants to conduct A/B



6.5 Application Performance Observation with

a checklist


participant of the questions

to conduct A/B testing for

the scenario provided by the


EDTECH 503 – Instructional Design Projectinstructor.


a checklist

Learners will schedule

participants to take A/B

usability testing for the

scenario provided by the

instructor.


a checklist

Learners will collect the

answers to the questions

from the A/B testing for the

scenario provided by the

instructor.


checklist

Learners will identify the

software to use as a part of

the simulation.


checklist

Learners will develop the

scenarios for the participants

to perform as a part of the

simulation.


checklist

Learners will identify the

participants as a part of the

simulation.


checklist

Learners will set the

interface to test as a part of

the simulation.


checklist

Learners will schedule the

participants to take the test as

a part of the simulation.

7.6 Application Performance Simulation with a

checklist

Learners will collect the

results as a part of the


EDTECH 503 – Instructional Design Projectsimulation.

8.1 Evaluation PerformanceSimulation with

checklist of skills

Learners will make

conclusions for each test goal

as a part of a simulation

8.2 Synthesis PerformanceSimulation with

checklist of skills

Learners will summarize

biographic information of

participants and other

metadata.

Part 3c: ARCS table

ATTENTION

A.1 Perceptual Arousal

The instructor will show a funny video on usability testing. The video is included in the

instructor guide.

A2. Inquiry Arousal

The instructor will ask what usability is and collect responses from all the learners. He or she

will inform the objective of the course to the learners.

A3. Variability

The instructor will ask questions about the learners’ job responsibilities and connect each feature

of usability testing to their work.

RELEVANCE

R1. Goal orientation

The instructor will use the learner analysis and need analysis results to decide the student’s needs

R2. Motive matching

The instructor will allow learners to choose the participants for their mock usability testing

sessions. The learners will have the freedom to choose a website of their choice to perform a

usability test.


EDTECH 503 – Instructional Design ProjectR3. Familiarity

The learners will be asked to tie the features of usability testing to their work. The instructor will

ask questions regarding this and collect responses from the learners.

CONFIDENCE

C1. Learning requirements

The instructor will email the requirements of the course to the learners a week before the training

day. The instructor can assist in building a positive expectation for success by checking the

progress of the learners and guiding them throughout the course.

C2. Success opportunities

The learners will have the opportunity to conduct usability tests as a group and by themselves

and receive immediate feedback from the instructor and the peers.

C3. Personal control

The instructor will provide examples of the increase in revenue some websites earned after

updating the website based on usability testing results.

SATISFACTION

S1. Natural consequences

The instructor will inform the learners that usability testing will be added to their regular work

schedule.

S2. Positive consequences

The instructor will provide feedback during mock sessions and other activities during the course.

Also, the instructor will encourage the learners and provide them support to subscribe to feeds

and emails related to usability testing.

S3. Equity

The instructor will congratulate the learners when they complete the activities and mock sessions

as a part of the course. The instructor will also talk about the importance of usability testing in

relation to their job responsibilities. He or she can explain how usability tests lead to effective

websites, which further lead to more revenue from sales through the websites and eventually lead

to good pay raise for the learners.



Part 4: Instructor Guide

Active Attention or Gain Attention

The instructor will start with a video on usability testing in order to gain attention. Provide

examples of usability testing: images of websites before and after usability testing (example) and

the increase in revenue though websites after usability testing (example). Continue to connect

each of the features of usability testing to their job responsibilities, throughout the course. Also,

talk about the importance of usability testing and its potential effect on the company website and

the business.

Establish Purpose or Inform Learners of Purpose

The instructor will inform learners that the purpose of the course is for them to acquire

knowledge on usability testing and conduct tests as a part of their job.

Arouse Interest and Motivation or Stimulate Learners’ Attention/Motivation

The instructor will give opportunities for learners to conduct mock sessions and other usability

testing activities during the course and provide feedback. Also, motivate the learners by tying

increase in website revenue due to usability testing to their performance evaluation and pay raise.

Preview the Learning Activity or Provide Overview

The instructor will provide an introduction to the course. He or she will inform the goal of the

course to the learners. They will learn about usability testing for websites: definition of usability

testing, different methods of the testing, the importance of testing, and the advantages of various

methods of usability testing. They will also get a chance to practice usability testing as a group as

well individually. Finally, they will subscribe to feeds and newsletters related to usability testing.

Body


https://en.wikipedia.org/wiki/Web_feed

http://www.usertesting.com/blog/2013/06/26/stubhub-increases-conversions-rates-and-revenue-with-usertesting/

http://www.howto.gov/web-content/usability/first-fridays/gallery

https://www.youtube.com/watch?v=3Qg80qTfzgU

EDTECH 503 – Instructional Design ProjectRecall relevant prior knowledge or Stimulate recall of prior knowledge

The learners have very low prior knowledge on usability testing. The instructor will ask

questions in order to recall their prior knowledge. The target audience is web designers and

developers; therefore, they have a good knowledge of the various parts or features of a website

when it comes to testing a website’s features.

Process information and examples or Present information and examples

The instructor will introduce the definition of usability testing to the learners. The learners will

be presented with not only the explanation of usability testing and its various methods, but also

the opportunity to watch the instructor perform usability testing. The instructor will provide real-

time examples of Sargent Manufacturing website usability test and Sheraton website usability

test.

Focus Attention or Gain & Direct Attention

The instructor will continuously interact with the learners during the course. He or she will ask

questions and collect responses from the learners. There will be mock usability testing sessions

and other activities when learners participate and receive immediate feedback from the instructor

and the peers. There will be group discussions regarding how the learners can apply certain

characteristics (card sorting and tree testing) of usability testing in their jobs.

Employ Learning Strategies or Guide or Prompt Use of Learning Strategies

The instructor will concentrate on procedural learning in the course. The exercise of usability

tests include following a procedure. Strategies for declarative and conceptual learning are also

part of this course. The learners have very low prior knowledge in the field of usability testing.

The declarative learning strategies will help learners to acquire the knowledge of usability testing

in order to recall it at the stage of application. The strategies for concept learning will help the

participants learn the concept of usability testing. They will be able to identify the case and the

right method when a website requires usability testing.


http://www.optimalworkshop.com/treejack.htm

http://en.wikipedia.org/wiki/Card_sorting

http://www.aims.co.il/NewYorkHotel-recomendations.pdf


http://www.wpi.edu/Pubs/E-project/Available/E-project-052710-070710/unrestricted/Usability_and_Usability_Testing_of_Websites.pdf

EDTECH 503 – Instructional Design ProjectPractice or Provide for and Guide Practice

The learners will have the opportunity to practice usability testing during the course. The

instructor will answer any questions they have and guide them through the process. He or she

will provide constructive feedback to the learners on their performance on mock usability

sessions and other activities. The learners will make revisions to their work based on the

feedback.

Evaluate Feedback or Provide Feedback

The instructor will observe the performance of the participants during the course. The learners

will receive immediate feedback on mock sessions from the instructor and the peers. The

learners will also have the opportunity to conduct personal usability tests and share the feedback

with the class to receive feedback regarding the method of usability test chose and the

characteristics of the website tested. The instructor will also provide feedback regarding the tone

of voice used during testing, testing room set-up, and eye tracking software used. There will be a

test at the end of the course and the learners will receive the result and the feedback.

Conclusion

Summarize and review or Provide summary and review

The instructor will summarize what the participants have learned. They learned about usability

testing, different methods of usability testing, and the importance and advantages of various

methods of usability testing. They also got a chance to practice usability testing as a group as

well as individually and received immediate feedback from the instructor and the peers. Finally,

they subscribed to feeds and newsletters related to usability testing in order to retain and transfer

the learning.

Transfer learning or Enhance transfer

The instructor will remind the learner that usability testing will be added to their regular work

schedule. He or she will encourage the learners to stay connected with the field of usability. The Copyright © 2013, Dalia Allencherry 21

EDTECH 503 – Instructional Design Projectinstructor will suggest practicing usability testing of websites for training purpose and encourage

learners to make use of the opportunity. In the future, he learners will also get chances to

contribute to white papers and do ROI calculations regarding usability testing of the company

websites.

Remotivate and Close or Provide Remediation and Closure

The instructor will emphasize the importance of usability testing and its connection to the

learners’ job responsibilities. Show an example of a website that has gone through usability

testing. Stress the connection between increased revenue through usability-tested websites and

the learners’ pay raise. Compare the motivation levels of the learners before the course (learner

analysis survey) and after the course (course feedback questionnaire). Conduct mock usability

test sessions and provide immediate feedback.

Assess Learning or Conduct Assessment Evaluate

The instructor will assess the learners on each of the learning objectives. He or she will also

monitor the participation of the learners in group discussions and mock sessions. The learners

will receive the result of the end of course evaluation.

Feedback and Seek Remediation or Provide Feedback and Remediation

The instructor will provide resources for learners to stay in connect with the field of usability.

Introduce them to options like subscription to feeds (UX Matters and usability.gov) and

newsletters (Human Factors). Congratulate them on completing the course and encourage them

to earn a certification on usability analysis and testing. Suggest the organizer or management to

create a usability club with scheduled and continuous meetings every two months to talk about

usability or present team members’ experience with usability. The instructor will provide the end

of course result and feedback to the learners.


http://www.humanfactors.com/home/usability.asp

http://www.usability.gov/

http://uxmatters.com/


EDTECH 503 – Instructional Design ProjectPart 5: Learner Content

5a: Learning materials

The learners will be provided the following learning materials to use during the instructional

time.

Learning materials PurposeTest start handout (see

appendix A)

• Learners can use as a reference to perform activities.

• Learners can use as a quick reference guide for the future

Pretest questionnaire handout

(see appendix B)



Posttest questionnaire

handout (see appendix C)



Learner guide

• Learners can follow along with the instructor

• Learners can use as a reference guide for the future

5b: Assessment materials

The following assessment materials will be used as a part of the instruction.

Assessment methods PurposeEvaluation of objectives (see

Appendix D) • The achievement of objectives can be measured

Performance checklist (see

Appendix E)

• Instructor will evaluate learners based on the checklist

• Learners can self-evaluate based on the checklist

• Learners are aware of the evaluation methods

5c: Technology tool rationale

The following technology tools will be used as a part of the instruction.



Technology tool Purpose

Computers

• Learners can complete activities that includes accessing websites

• Learners can access the online test at the end of the course

Projector

• Instructor can project the power point presentation on a screen for

learners to view

Websites

• Human Factors

• Usability.gov

• UX Matters

• Sears

• Hertz

• Cheapoair

• YouTube

• Learners can subscribe to feeds and newsletters

• Learners can subscribe to feeds

• Learners can subscribe to feeds

• Learners can conduct mock usability test



• Learners can watch an example of usability test

Tobii Studio • Learners can practice usability testing using this eye-tracking software

Part 6: Formative Evaluation Plan

6a: Expert review plan

The SME is my colleague Bincy Philip, who is a certified usability analyst (CUA from HFI). She

is an expert in user-centered analysis, conceptual design, practical usability testing, effective web

and application design, and usability research. She regularly conducts usability tests for as a part

of her job. The instructional materials were submitted to her on Friday, July 26. She completed

the review during the weekend and got the package back to me on Monday, July 29. The

following list of questions was sent to her along with the review package.

1. Does the course cover all the required content based on the course goal?

2. Are the objectives achievable based on the content covered?

3. Are the examples and other scenarios given in the course realistic?

4. Does the instruction flow in a logical order?


https://www.youtube.com/watch?v=3Qg80qTfzgU

http://www.cheapoair.com/

http://www.hertz.com/

http://www.sears.com/en_us.html?s_tnt=43913:7:0&

http://uxmatters.com/

http://www.usability.gov/

http://www.humanfactors.com/home/usability.asp

EDTECH 503 – Instructional Design Project5. Do you suggest any changes to the course content?

6. Do you consider any part of the course as unnecessary?

7. What are the strengths of the course?

8. What are the weaknesses of the course?

9. Would you add anything to engage the learners more? Please explain.

6b: One-to-one review plan

The goal of the one-to-one evaluation is to review the course on a small scale. I will identify two

people who represent the target audience. I will have one-to-one meetings with each of them. I

will make it clear that the goal of the meeting is to evaluate the course material, not them. The

participants will evaluate the instructional materials. I will ask specific questions and collect

feedback.

1. Does the instructor guide follow a logical order?

2. Are the objectives clear and appropriate considering the project goal?

3. Are the objectives attainable by web designers and developers?

4. Do you think the graphics and videos are appealing?

5. Can you relate the examples to your work situations?

6. Are there any subtopics that need to be explained further?

7. Will you be able to understand the course materials without instructor support?

8. If you were the instructor, what changes would you make?

6c: Small group evaluation plan

Small group evaluation is another type of evaluation. I will identify four people who represent

the target audience for this evaluation. I will meet with the participants as a group. This is a good

opportunity to check the validity of the instructional materials in the context of a group setting.

All the group activities will be tested at this time. I will ask the following questions to the

participants.Copyright © 2013, Dalia Allencherry 25

EDTECH 503 – Instructional Design Project1. Are the instructions for the group activities clear?

2. Were you able to complete the group activities within the given time?

3. Were you able to conduct usability tests with the instructions given in the learner guide

and without instructor support?

4. Did you find any information that is not relevant based on the course goal?

5. Did you feel engaged throughout the course?

6. What is your level of satisfaction regarding the course delivery? Rate 1-5

7. What do you think can be improved about the instruction and the materials?

6d: Field trial plan

Field trial is the opportunity to test the changes made based on the feedback received during the

one-to-one and the small group evaluations. I will select a sample of six people representing the

target audience for this evaluation. The room will be set to mimic the real-time training room.

There will be a feedback session at the end where I will ask specific questions and collect

feedback.

1. Does the instruction flow in a logical order? If not, please explain.

2. Are the objectives clear and appropriate considering the course goal? If not, please

explain.

3. Is the instruction easy to follow? If not, please explain.

4. Were you engaged throughout the course? If not, please explain.

5. Do you think any additional information needs to be added to the course? Please explain.

6. Is the location appropriate for the course delivery? Please explain.

7. Was there an ample supply of materials (pens, sticky notes, highlighters, etc.) required?

Please explain

8. Do you suggest any changes to the course? Please explain

9. Would you add anything to engage the learners more? Please explain.


EDTECH 503 – Instructional Design ProjectPart 7: Formative Evaluation Report

7a: Evaluation survey

The following expert evaluation form was provided to the subject matter expert.

Source: http://freeonlinesurveys.com/s.asp?sid=7rv8a7uncbvu9co307668

7b: Report of expert review

My SME provided valuable insights and suggestions after reviewing the course materials. She

thinks that the course covers all the content required in order to achieve the course goal. The

objectives are connected to the course goals and are achievable based on the course. Even though

the scenarios used in the course are realistic, the SME suggests changing some of the scenarios

with the ones related to the target audience’s work. They can be the from the company website

the target audience uses as a part of their work or any other e-commerce website. The SME

thinks the course follows a logical sequence. She recommends making the steps to perform a

usability test clearer. She suggests this because the target audience has very little prior

knowledge of the procedure and none of them have done this before. She couldn’t identify any

unnecessary content in the course. She identifies the group as well as the individual activities or

in other words, mock sessions or simulations as the biggest strength of the course. She also likes

the idea of helping learners to subscribe to feeds and newsletters in order to stay connected with

the subject. She thinks the idea of “Lunch and Learn” sessions to discuss usability and present

team members’ experience may not do well because people may choose not to attend these

sessions. “Lunch and Learn” sessions have not been very successful in the company. She

considers the course as very engaging due to the presence of the group and individual activities.

Overall, the feedback was positive.

7c: Designer’s response to review

As a result of the expert review, I made a few changes to the course. I broke down some of the

steps of the usability testing procedure into two and added notes or tips about conducting some


http://freeonlinesurveys.com/s.asp?sid=7rv8a7uncbvu9co307668

EDTECH 503 – Instructional Design Projectof the steps. For e.g. In the case of the step “Schedule interviews with the participants”, I added

the note “Decide if the interview is direct or virtual and monitored or unmonitored”. Even though

I talked about conducting virtual interviews in the course, I forgot to add it as a part of the

procedure. I replaced two of the websites selected for usability testing (BBC news and

Dictionary.com) with e-commerce websites (Hertz and Cheapoair) according to the SME’s

feedback. Based on the SME’s concerns regarding team members’ attendance to “Lunch and

Learn” sessions, I chose to recommend forming a usability club and have scheduled and

continuous meetings every month or two to discuss usability and also present team members’

experience with usability.

Part 8: Standards Grid

Professional Standards Addressed (AECT)

The following standards, developed by the Association for Educational Communications and

Technology (AECT), and used in the accreditation process established by the National Council

for Accreditation of Teacher Education (NCATE), are addressed to some degree in this course.

The numbers of the standards correspond to the numbers next to the course tasks show on the list

of assignments. Not all standards are addressed explicitly through student work.

Assignments meeting standard in whole or part

Standard 1: DESIGN

1.1 Instructional Systems Design

(ISD) X

ID Project

1.1.1 Analyzing X ID Project

1.1.2 Designing X ID Project

1.1.3 Developing X ID Project

1.1.4 Implementing X ID Project

1.1.5 Evaluating X Selected Discussion Forums; ID Project

1.2 Message Design

1.3 Instructional Strategies X ID Project

1.4 Learner Characteristics X ID Project


http://www.cheapoair.com/

http://www.hertz.com/

http://dictionary.reference.com/

http://www.bbc.com/news/

EDTECH 503 – Instructional Design ProjectStandard 2: DEVELOPMENT

2.0 (includes 2.0.1 to 2.0.8) X ID Project

2.1 Print Technologies X Reading Quiz; ID Projects

2.2 Audiovisual Technologies

2.3 Computer-Based Technologies X (all assignments)

2.4 Integrated Technologies

Standard 3: UTILIZATION

3.0 (includes 3.0.1 & 3.0.2)

3.1 Media Utilization X (all assignments)

3.2 Diffusion of Innovations

3.3 Implementation and

Institutionalization X

ID Project

3.4 Policies and Regulations

Standard 4: MANAGEMENT

4.0 (includes 4.0.1 & 4.0.3)

4.1 Project Management

4.2 Resource Management

4.3 Delivery System Management

4.4 Information Management

Standard 5: EVALUATION

5.1 Problem Analysis X

5.2 Criterion-Referenced

Measurement X

ID Project

5.3 Formative and Summative

Evaluation X

ID Project

5.4 Long-Range Planning

COURSE GOALS & OBJECTIVES


EDTECH 503 – Instructional Design ProjectThe overall goal for the course is for each student to consider and use the systematic process of

instructional design to create an instructional product. To achieve this goal, students will engage

in activities that promote reflective practice, emphasize realistic contexts, and employ a number

of communications technologies. Following the course, students will be able to:

1. Discuss the historical development of the practice of instructional design with regard to

factors that led to its development and the rationale for its use

2. Describe at least two reasons why instructional design models are useful

3. Identify at least six instructional design models and classify them according to their use

4. Compare and contrast the major elements of three theories of learning as they relate to

instructional design

5. Define “instructional design.”

6. Define the word “systematic” as it relates to instructional design

7. Define “learning” and synthesize its definition with the practice of instructional design

8. Relate the design of instruction to the term “educational (or “instructional”) technology”

9. Describe the major components of the instructional design process and the functions of

models in the design process

10. Provide a succinct summary of various learning contexts (declarative knowledge,

conceptual, declarative, principle, problem-solving, cognitive, attitudinal, and

psychomotor)

11. Build an instructional design product that integrates major aspects of the systematic

process and make this available on the web.

a. Describe the rationale for and processes associated with needs, learner, context,

goal, and task analyses

i. Create and conduct various aspects of a front-end analysis

ii. Identify methods and materials for communicating subject matter that are

contextually relevant

b. Describe the rationale for and processes associated with creating design

documents (objectives, motivation, etc.)

i. Construct clear instructional goals and objectives


EDTECH 503 – Instructional Design Projectii. Develop a motivational design for a specific instructional task

iii. Develop assessments that accurately measure performance objectives

c. Select and implement instructional strategies for selected learning tasks

i. Select appropriate media tools that support instructional design decisions

d. Describe the rationale and processes associated with the formative evaluation of

instructional products

i. Create a plan for formative evaluation

12. Identify and use technology resources to enable and empower learners with diverse

backgrounds, characteristics, and abilities.

13. Apply state and national content standards to the development of instructional products

14. Meet selected professional standards developed by the Association for Educational

Communications and Technology

15. Use various technological tools for instructional and professional communication

AECT STANDARDS (Applicable to EDTECH 503)

1.0 Design

1.1 Instructional Systems Design

1.1.a Utilize and implement design principles which specify optimal conditions for

learning.

1.1.b Identify a variety of instructional systems design models and apply at least one

model.

1.1.1 Analyzing

1.1.1.a Write appropriate objectives for specific content and outcome levels.

1.1.1.b Analyze instructional tasks, content, and context.

1.1.2 Designing


EDTECH 503 – Instructional Design Project1.1.2.a Create a plan for a topic of a content area (e.g., a thematic unit, a text chapter, an

interdisciplinary unit) to demonstrate application of the principles of macro-level design.

1.1.2.b Create instructional plans (micro-level design) that address the needs of all

learners, including appropriate accommodations for learners with special needs.

1.1.2.d Incorporate contemporary instructional technology processes in the development

of interactive lessons that promote student learning.

1.1.3 Developing

1.1.3.a Produce instructional materials which require the use of multiple media (e.g.,

computers, video, projection).

1.1.3.b Demonstrate personal skill development with at least one: computer authoring

application, video tool, or electronic communication application.

1.1.4 Implementing

1.1.4.a Use instructional plans and materials which they have produced in contextualized

instructional settings (e.g., practice, field experiences, and training) that address the needs

of all learners, including appropriate accommodations for learners with special needs.

1.1.5 Evaluating

1.1.5.a Utilize a variety of assessment measures to determine the adequacy of learning

and instruction.

1.1.5.b Demonstrate the use of formative and summative evaluation within practice and

contextualized field experiences.

1.1.5.c Demonstrate congruency among goals/objectives, instructional strategies, and

assessment measures.

1.3 Instructional Strategies

1.3.a Select instructional strategies appropriate for a variety of learner characteristics and

learning situations.


EDTECH 503 – Instructional Design Project1.3.b Identify at least one instructional model and demonstrate appropriate

contextualized application within practice and field experiences.

1.3.c Analyze their selection of instructional strategies and/or models as influenced by the

learning situation, nature of the specific content, and type of learner objective.

1.3.d Select motivational strategies appropriate for the target learners, task, and learning

situation.

1.4 Learner Characteristics

1.4.a Identify a broad range of observed and hypothetical learner characteristics for their

particular area(s) of preparation.

1.4.b Describe and/or document specific learner characteristics which influence the

selection of instructional strategies.

1.4.c Describe and/or document specific learner characteristics which influence the

implementation of instructional strategies.

2.0 Development

2.0.1 Select appropriate media to produce effective learning environments using

technology resources.

2.0.2 Use appropriate analog and digital productivity tools to develop instructional and

professional products.

2.0.3 Apply instructional design principles to select appropriate technological tools for

the development of instructional and professional products.

2.0.4 Apply appropriate learning and psychological theories to the selection of

appropriate technological tools and to the development of instructional and professional

products.

2.0.5 Apply appropriate evaluation strategies and techniques for assessing effectiveness

of instructional and professional products.


EDTECH 503 – Instructional Design Project2.0.6 Use the results of evaluation methods and techniques to revise and update

instructional and professional products.

2.0.7 Contribute to a professional portfolio by developing and selecting a variety of

productions for inclusion in the portfolio.

2.1 Print Technologies

2.1.3 Use presentation application software to produce presentations and supplementary

materials for instructional and professional purposes.

2.1.4 Produce instructional and professional products using various aspects of integrated

application programs.

2.3 Computer-Based Technologies

2.3.2 Design, produce, and use digital information with computer-based technologies.

3.0 Utilization

3.1 Media Utilization

3.1.1 Identify key factors in selecting and using technologies appropriate for learning

situations specified in the instructional design process.

3.1.2 Use educational communications and instructional technology (SMETS) resources

in a variety of learning contexts.

3.3 Implementation and Institutionalization

3.3.1 Use appropriate instructional materials and strategies in various learning contexts.

3.3.2 Identify and apply techniques for integrating SMETS innovations in various

learning contexts.

3.3.3 Identify strategies to maintain use after initial adoption.

4.0 Management

(none specifically addressed in 503)

5.0 Evaluation


EDTECH 503 – Instructional Design Project5.1 Problem Analysis

5.1.1 Identify and apply problem analysis skills in appropriate school media and

educational technology (SMET) contexts (e.g., conduct needs assessments, identify and

define problems, identify constraints, identify resources, define learner characteristics,

define goals and objectives in instructional systems design, media development and

utilization, program management, and evaluation).

5.2 Criterion-referenced Measurement

5.2.1 Develop and apply criterion-referenced measures in a variety of SMET contexts.

5.3 Formative and Summative Evaluation

5.3.1 Develop and apply formative and summative evaluation strategies in a variety of

SMET contexts.

SMET = School Media & Educational Technologies



Appendices

Appendix A

Test Start Manual

Welcome and Purpose

Thank you for agreeing to participate

Our goal is to evaluate the website (you are testing) from a user’s perspective.

Test Facilitator’s and Observer’s Role

As facilitator, I will ask a series of questions while the observer’s role is to record your

responses

During this session, I may not be able to offer any suggestions or hints.

There may be times, when I will ask you to explain why you said or did something.

Test Participant’s Role

Simply perform tasks on this site as you would if no one was around. Please “think out

loud” when performing tasks.

You will also be asked a series of questions about your experience at the end of this

session.

Things to Keep in Mind

This is not a test of you; we are testing the site. So, don not worry about making

mistakes, there is no right or wrong answer.

We just want to know if we designed the site well for you.

If you ever feel that you are lost or cannot complete a scenario with the information that

you have been given, please let me know. I will ask you what you might do in a real-

world setting and then either put you on the right track or move you to the next scenario.


EDTECH 503 – Instructional Design Project Finally, as you use the site, please do so as you would be on your own.

Appendix B

Pretest Questionnaire

1. What is your current position and title?

2. How long have you held this position and been with the company?

________ in position __________ with the company

Computer Expertise

1. What would you consider as your level of computer expertise? (Create a scale)

Low 1 2 3 4 5 High

2. Besides reading email, what are typical activities you do on the computer?

Gaming/entertainment Reading news Shopping/banking

Graphic design/digital pictures

Programming/word processing

Social Media

Other:

3. Share 2-3 of your favorite web sites that you frequently visit. Why are they your favorite?

Website Reasons

4. What computer platform do you usually use? [e.g. Mac, Windows XP, etc.]

5. What Internet browser(s) do you usually use? [e.g., Firefox, IE v.6, AOL, Chrome, etc.]


EDTECH 503 – Instructional Design Project6. Did you use the website (you are testing) in the last year? How often do you visit the

website?

Appendix C

Post Test Questionnaire

1. What are your overall impressions of the website (you are testing)?

2. If you had to give the site a grade, from A to F, where A as exemplary and F as failing,

what grade would you give the website, and why?

A B C D F

3. Name three words or characteristics that describe the website.

4. What are the three things you like best about the website?

5. What are the three things you like least about the website?

6. If you could make one significant change to the website, what change would you make?

7. Can you describe the site’s purpose?

8. Why would someone go to the website?

9. Would you return to the website in the future? Why/why not?

10. What would entice you to return?

11. Do you have any other questions or comments about the website or your experiences with

it?



Appendix D

Objectives End of course assessment1.1 Learners will define usability testing in their own words Define usability testing.

1.2 Learners will explain the purpose of usability testing

Which of the following is a purpose of usability testing? Select all that apply.A. Identify technical errorsB. Identify users' mental modelC. Identify organization of tasksD. Identify issues of a site after build and before release

1.3 Learners will explain the importance of usability testing Explain the importance of usability testing.

1.4 Learners will list three methods of usability testing

Which of the following are methods of usability testing?A. Usability testing using interviews.B. Usability testing using an eye tracking software.C. A/B usability testingD. Peer testing

1.5 Learners will list the characteristics of a website that need to be evaluated

Which of the following characteristics of a website need to be evaluated during usability testing?A. Information architectureB. Color combinationC. Company logoD. Website technical issues

1.6 Learners will explain hallway usability testing What is hallway usability testing?1.7 Learners will explain remote usability testing What is remote usability testing?

2.1 Learners will explain usability testing using interviews in their own words

Match the following.Column 1: (Usability testing using interviews, A/B usability testing, and Usability testing using an eye tracking software)Column 2: Larger sample size enables statistically reliable metrics, Most suitable to measure data such as sales percentages, subscription rates, etc., and Can automatically decide where the user’s interest is concentrated


EDTECH 503 – Instructional Design Project2.1 Learners will recall the advantages of usability testing using interviews compared to other methods

Explain the advantages of usability testing using interviews over other methods.

3.1 Learners will explain A/B testing in their own words

Match the following.Column 1: (Usability testing using interviews, A/B usability testing, and Usability testing using an eye tracking software)Column 2: Larger sample size enables statistically reliable metrics, Most suitable to measure data such as sales percentages, subscription rates, etc., and Can automatically decide where the user’s interest is concentrated

3.2 Learners will recall the advantages of A/B usability testing compared to other methods

Explain the advantages of A/B usability testing over other methods.

4.1 Learners will explain usability testing using an eye tracking software

Match the following.Column 1: (Usability testing using interviews, A/B usability testing, Usability testing using an eye tracking software)Column 2: Larger sample size enables statistically reliable metrics, Most suitable to measure data such as sales percentages, subscription rates, etc., Can automatically decide where the user’s interest is concentrated

4.2 Learners will recall the advantages of usability testing using an eye tracking software compared to other methods

Explain the advantages usability testing using an eye tracking software over other methods.

Note: The rest of the objectives are measured by observation or simulation using a checklist (see appendix E).



Appendix E

Required Tasks Completed? Y/NEarn at least 80% marks in the end of the course assessment

Usability testing using interviews - Group activity

Analyze test case and goalsIdentify participantsIdentify an observerIdentify interviewerCreate scenarios to test

Schedule interviews with the participantsGreet the participantCollect answers to the questions in the pretest questionnaireInform scenarios for the participants to perform and collect data

Collect answers to the posttest questionnaire

Thank the participantCollect data from the observer

Make conclusions for each test goal

Summarize biographic information of participants and other metadata

A/B usability testing - Individual activity

Analyze test case and goalsSet two wireframes or interfaces ready for A/B usability test

Write questions to ask participants

Identify participantsInform or present the questions to the participants and collect dataMake conclusions for each test goalSummarize biographic information of participants and other metadata


EDTECH 503 – Instructional Design ProjectUsability testing using an eye tracking software - Individual activity

Analyze the test case and goals

Identify software to use (Tobii Studio as decided by the company)Develop scenarios to testIdentify participantsSet interface to test

Schedule the participants to take the test.

Collect data from participant responses

Make conclusions for each test goalSummarize biographic information of participants and other metadata


Date post:	10-Feb-2018
Category:	Documents
Upload:	dinhdat
View:	217 times
Download:	5 times

Web viewThe instructional design process parallels other processes of development when observing the...

Documents