Post on 10-Feb-2018
transcript
Dalia Allencherry
EDTECH 503 (Summer 2013)
Instructional Design Project
Submitted to: Dr. Jennifer Freed
August 4, 2013
Project Title: Usability Testing for Websites
Copyright © 2013, Dalia Allencherry
EDTECH 503 – Instructional Design Project
Table of Contents Page
Reflection Synthesis paper 3Part 1. Topic 4 Part 1a. Goal statement 4 Part 1b. Audience description 4 Part 1c. Rationale 4Part 2. Analysis Report 6 Part 2a1: Needs assessment survey 6 Part 2a2: Needs assessment data 7 Part 2b1: Learning context description 8 Part 2b2: Transfer context description 8 Part 2c: Description of Learners 9 Part 2d: Task Analysis Flow Chart 10Part 3. Planning 10 Part 3a: List of instructional objectives 10 Part 3b: Objectives matrix table 12 Part 3c: ARCS table 17Part 4. Instructor Guide 19Part 5. Learner Content 23 Part 5a: Learning materials 23 Part 5b: Assessment materials 23 Part 5c: Technology tool rationale 23Part 6. Formative Evaluation Plan 24 Part 6a: Expert review plan 24 Part 6b: One-to-one review plan 25 Part 6c: Small group evaluation plan 25 Part 6d: Field trial plan 26Part 7. Formative Evaluation Report 26 Part 7a: Evaluation survey 26 Part 7b: Report of expert review 27 Part 7c: Designer’s response to review 27Part 8. Standards Grid 28Appendices 36
Copyright © 2013, Dalia Allencherry 2
EDTECH 503 – Instructional Design Project
Reflection Synthesis Paper
The instructional design process parallels other processes of development when observing the
broad stages of project inception, planning, and completion. For example, one can illustrate the
instructional design process via the lens of an architect. Akin to an instructional designer, an
architect commences with an inspiration: the desire to create a functional and aesthetic physical
structure to be used and enjoyed by society. Likewise, an instructional designer beckons her
faculties to generate novel methods for conveying knowledge. Such knowledge should also
serve a practical purpose and be appealing and beneficial to the learners that will eventually use
the materials. Next, an architect spends long hours at the drafting table--the blueprints should be
just right, they should be scalable to real space, and should comply with local building codes.
Similarly, an instructional designer maps out her plans and creates design documents. The
architect then fashions a miniature model of the building, presents it to the client, and makes
changes based on the feedback. Similarly, an instructional designer presents the training to a
sample audience, collects feedback, and makes the appropriate changes. These stages of
development will soon produce real platforms in order to engage actual students. However, such
instruments must fulfill agreed upon standards and survive the scrutiny of subject matter experts.
Finally, the day comes when the structure is erected, the ribbon is cut, and the space is utilized
for its intended purpose. The building is enjoyed by society and the architect is pleased having
journeyed with the idea from inception to completion. Likewise, the instructional designer is
gratified by harnessing an idea, guiding it through the planning stages while satisfying strict
standards, and ultimately releasing the product to the intended audience.
Copyright © 2013, Dalia Allencherry 3
EDTECH 503 – Instructional Design ProjectI have worked as an instructional designer in the past. However, in this course I learned many
things that I should have used on the job, but was not aware of. This course was eye opening for
me, especially the ID project. I never paid attention to the motivational factor of learners prior to
this course. Therefore, I made no attempts to include motivational strategies in my design
documents and storyboards. My biggest takeaway from the course was learning about
motivational strategies. Sometimes I struggled with generating strategies to motivate learners
during the project. The course activities were extremely useful and I learned a lot from my
classmates’ posts and feedback. Even though I have submitted my work for SME reviews, I
never conducted one-to-one evaluations, group evaluations, or field trials prior to this course.
Now, I understand the importance of these evaluations. Extensive learner analysis and its
significance is another prime takeaway. In the past, I collected data about the target audience
from the client for most of my projects (i.e., the company or employer of the target audience paid
me to create the training materials and provided relevant data). I now grasp the importance of
learner analysis and how it should be conducted properly. Overall, this course was a great
opportunity for me to ‘unlearn’ activities that I performed poorly and to learn the appropriate
methods of instructional design processes.
There will be many occasions to incorporate knowledge from the course in my job. I review test
items and manage a team of content specialists that creates test items. I can apply my newfound
skills in learner analysis to effectively analyze the target audience and design assessments that
test the objectives. I am responsible for providing training to my team members based on their
needs. This arrangement will enable me to transfer my knowledge on the instructional design
process to design training for my team members.
Part 1: Topic
Part 1a: Goal statement
Learners will acquire the knowledge and conduct usability tests of a website.
Copyright © 2013, Dalia Allencherry 4
EDTECH 503 – Instructional Design ProjectPart 1b: Audience description
The learners are web designers and developers of the company ALT. They have either an
associate degree or a bachelor’s degree in web design or a related field with an experience
ranging from three to eleven years.
Part 1c: Rationale
The sales through the company website are not improving as expected even after rigorous
marketing efforts. Therefore, the senior management team of ALT decided to take a look at the
factors that may stop potential customers from making a purchase through the website. They
identified one of the reasons as lack of ease to use the website. Usability analysis or testing was
never a part of the website development. In this case, the corporate decided to provide training to
the website designers and developers on usability testing and incorporate usability tests in the
regular work schedule. This course introduces the concept of usability to the learners and
provides them resources to conduct usability tests. It also gives them a chance to practice various
methods of usability tests in an instructor-monitored environment. This gives them the
opportunity to get feedback from the instructor and the peers during the mock sessions.
The overall project consists of about 80% supplantive and 20% generative strategy. The
instruction is mainly supplantive because of the procedures that need to be followed for usability
testing. Even though certain procedures need to be followed, the learners have opportunities to
use their interpretation of the information and detail to decide on the questions they want to ask
during interviews, website features they want to cover, type of method to choose, and the setting
in which they want to conduct the test. The primary type of learning that is concentrated on this
course is procedural. The instruction involves procedures to conduct various usability testing
methods.
The goal of the project is classified under three different strategies. They are strategies for
declarative knowledge instruction, strategies leading to concept learning, and strategies for
instruction to learning procedures, which is the major instructional strategy. The learners have
very minimal prior knowledge in usability testing. The instruction involves strategies of
Copyright © 2013, Dalia Allencherry 5
EDTECH 503 – Instructional Design Projectdeclarative knowledge for the learners to understand the definition, context, purpose, and
importance of usability tests. The strategies of concept learning are used for the learners to recall
the reasons behind usability testing and apply them in the context of a website to determine the
most suitable usability test method for the site. The instruction of concept at an abstract level is
suitable in the context of this project. The strategies for instruction to learning procedures are
used for learners to understand and perform the step-by-step procedures involved in conducting
various usability testing methods.
Part 2: Analysis Report
Part 2a1: Needs assessment survey
The following need assessment survey was sent out to 13 people to collect data for the need
analysis.
Name:
Job title:
Male or Female:
Age:
1. How many years of experience do you have in website design or development?
2. What degree(s) do you possess related to website design or development?
3. Define usability testing.
4. Have you ever conducted a usability test?
5. Have you ever conducted a user acceptance test?
6. What is the difference between a usability test and a user acceptance test?
7. What motivates you the most about conducting usability tests for websites?
8. How motivated are you about conducting usability test? Rate 1-5
9. How relevant do you think is the need to conduct usability analysis? Not relevant,
Relevant, Very relevant
10. How confident are you that you can successfully perform usability analysis? Rate 1-5
Copyright © 2013, Dalia Allencherry 6
EDTECH 503 – Instructional Design Project11. What characteristics of usability analysis interest you the most?
12. How important is usability testing when your job responsibilities are considered? Not
Important, Important, Very important
13. Do you expect any constraints regarding you performing usability tests?
14. What consequence do you expect if you perform an incorrect usability test?
15. What are the goals you plan to achieve by taking the usability test training?
16. How do you feel about the way the company offers trainings?
17. How seriously do you usually consider trainings offered by the company? Not serious,
Serious, Very serious
18. How often does the company follow-up with the training sessions provided? Never,
Sometimes, Always
Part 2a2: Needs assessment data
The need analysis survey revealed that the work-related experience of the learners range from
three to eleven years. They possess either an associate degree or bachelor’s degree in a field
related to their work. The learners have very low prior knowledge in usability testing. None of
them have conducted a usability test even though two of them correctly defined usability testing.
Some of the learners have conducted user acceptance testing, however, forty percent of the
learners think of usability testing a part of user acceptance testing.
The learners showed low motivation levels. None of the learners appeared to be highly motivated
(level 5). Eight-five percent of the learners think usability analysis is relevant to their work. The
company rarely follows-up on training sessions provided; therefore the measures to retain the
knowledge of usability testing need to be included in the course. Even though eight percent of
the examinees take trainings seriously, none of them seem to take it very serious. None of them
identified any constraints that could affect their performance in conducting usability test other
than proper training. As shown below, most of the learners are not sure about the importance of
usability testing when their job responsibilities are considered.
Copyright © 2013, Dalia Allencherry 7
EDTECH 503 – Instructional Design ProjectImportance of usability testing when your job responsibilities are considered
Not important
Important
Very important
Don't know
Part 2b1: Learning context description
The learning environment is a training room in the company compound. The room is well lit,
well ventilated, and has enough room for learners to move around freely. There are desks and
seating for fifteen people in the room. There are fifteen computers with internet connection for
the learners to use. A laptop, a projector, and a display screen are available in the room. Note
books, pens, and markers are available in the room. The instructor is a usability expert. Learners
will receive emails about the instructor of the course and basic information about usability, in
advance. Thirteen employees will attend the course. The course will be delivered from 1 p.m. to
4 p.m. Water, coffee, and ice tea will be available throughout the course. Some snacks will be
provided during a break at 2:30 p.m.
Part 2b2: Transfer context description
The learners will transfer the knowledge and skills learned to their daily work. As a part of a new
corporate-wide initiative to improve processes, usability will be added to the regular work
schedule of the learners. Usability testing is scheduled to be conducted every six months as a part
of the regular work schedule for the website development team. It will also be conducted in the
case of any major revisions to an existing website. Additionally, the learners will subscribe to
feeds and newsletters related to usability, which will enable them to build on their knowledge
Copyright © 2013, Dalia Allencherry 8
EDTECH 503 – Instructional Design Projectrelated to the topic and apply it in their usability analysis and testing. The result of the transfer of
knowledge and skills from the usability testing will be measured using Return on Investment
(ROI) calculations that involve the cost of the training and the change in sales through the
updated website after usability testing. Web analytics studies related to the website before and
after usability testing will also measure the transfer of knowledge and skills as a result of the
training.
Part 2c: Description of Learners
The leaners are web designers and developers of the company ALT. They possess either an
associate degree or a bachelor’s degree in a field related to their work. They have a work
experience of three to eleven years in the field. The experience related to website design and
development will be at the benefit of the learners as usability testing deals with various parts of a
website. The learners consist of 30% women and 70% men with an age range of twenty-eight to
fifty years. They show low motivation and very minimal prior knowledge of the training topic.
Define usability testing
Usability testing definition0
1
2
3
4
5
6
7
8
CorrectPartially correctWrong
Copyright © 2013, Dalia Allencherry 9
EDTECH 503 – Instructional Design Project
Part 2d: Task Analysis Flow Chart
Click here to open the graphic in a window.
Part 3: Planning
Part 3a: List of instructional objectives
1 Learners will explain usability testing
1.1 Learners will define usability testing in their own words
1.2 Learners will explain the purpose of usability testing
1.3 Learners will explain the importance of usability testing
1.4 Learners will list three methods of usability testing
1.5 Learners will list the characteristics of a website that need to be evaluated
1.6 Learners will explain hallway usability testing
1.7 Learners will explain remote usability testing
2 Learners will describe usability testing using interviews
2.1 Learners will explain usability testing using interviews in their own wordsCopyright © 2013, Dalia Allencherry 10
EDTECH 503 – Instructional Design Project2.2 Learners will recall the advantages of usability testing using interviews
compared to other methods
3 Learners will describe A/B usability testing
3.1 Learners will explain A/B testing in their own words
3.2 Learners will recall the advantages of A/B usability testing compared to other
methods
4 Learners will describe usability testing using an eye tracking software
4.1 Learners will explain usability testing using an eye tracking software
4.2 Learners will recall the advantages of usability testing using an eye tracking
software compared to other methods
5 Learners will apply usability testing using interviews
5.1 Learners will identify participants
5.2 Learners will identify the observer
5.3 Learners will create the scenarios to test
5.4 Learners will schedule the interviews with participants
5.5 Learners will prepare the test room
5.6 Learners will greet the guest
5.7 Learners will inform the purpose of the usability test to the participant
5.8 Learners will collect answers to the pretest questionnaire
5.9 Learners will inform the scenarios to perform to the participants
5.10 Learners will collect the data
5.11 Learners will collect answers to the posttest questionnaire
5.12 Learners will thank the participant
5.13 Learners will collect data from the observer
6 Learners will apply A/B usability testing
6.1 Learners will identify participants
6.2 Learners will set two wireframes or interfaces ready for testing
6.3 Learners will develop questions for the participants to answer
6.4 Learners will identify participants to conduct the test
6.5 Learners will inform the participants of the questions
Copyright © 2013, Dalia Allencherry 11
EDTECH 503 – Instructional Design Project6.6 Learners will schedule the participants to take the test
6.7 Learners will collect the answers to the questions
7 Learners will apply usability testing using an eye tracking software
7.1 Learners will identify the software to use
7.2 Learners will develop the scenarios for the participants to perform
7.3 Learners will identify the participants
7.4 Learners will set the interface to test
7.5 Learners will schedule the participants to take the test
7.6 Learners will collect the data
8 Learners will summarize the results of usability testing
8.1 Learners will make conclusions for each test goal
8.2 Learners will summarize biographic information of the participants and other
metadata
Part 3b: Objectives matrix table
Learning
Objectives
(a)
Bloom’s
Taxonomy
Classification (b)
Format of
Assessment (c)
Description of
test form (d)
Sample items
(e)
1.1 Knowledge Paper-and-Pencil Short Answer Define usability testing.
1.2 Comprehension Paper-and-PencilMultiple-choice
question
Choose the correct responses
of a multiple-choice multiple
response (MCMR) question
on the purpose of usability
testing.
1.3 Comprehension Paper-and-Pencil Short AnswerExplain the importance of
usability testing.
1.4 Knowledge Paper-and-Pencil Multiple-choice
question
Choose the correct responses
of a multiple-choice multiple
responses (MCMR) question
Copyright © 2013, Dalia Allencherry 12
EDTECH 503 – Instructional Design Projectto select the three methods of
usability testing.
1.5 Knowledge Paper-and-PencilMultiple-choice
question
Choose the correct response
of a MCSR question on the
characteristics of a website
that need to be evaluated.
1.6 Comprehension Paper-and-Pencil Short answerWhat is hallway usability
testing?
1.7 Comprehension Paper-and-Pencil Short answerWhat is remote usability
testing?
2.1 Comprehension Paper-and-PencilMatch/Drag and
drop
Match the type of test to its
correct characteristic.
2.2 Knowledge Paper-and-Pencil Short answer
Explain the advantages of
usability testing using
interviews over other
methods.
3.1 Comprehension Paper-and-PencilMatch/Drag and
drop
Match the type of test to its
correct characteristic.
3.2 Knowledge Paper-and-Pencil Short answer
Explain the advantages of
A/B usability testing over
other methods.
4.1 Comprehension Paper-and-PencilMatch/Drag and
drop
Match the type of test to its
correct characteristic.
4.2 Knowledge Paper-and-Pencil Short answer
Explain the advantages
usability testing using an eye
tracking software over other
methods.
5.1 Application Performance Simulation with a
checklist
Learners will identify
participants as a part of a
Copyright © 2013, Dalia Allencherry 13
EDTECH 503 – Instructional Design Projectsimulation.
5.2 Application Performance
Simulation with a
checklist
Learners will identify an
observer as a part of the
simulation.
5.3 Application Performance
Simulation with a
checklist
Learners will create the
scenarios to test as a part of
the simulation.
5.4 Application Performance
Simulation with a
checklist
Learners will schedule the
interviews with participants
as the part of the simulation.
5.5 Application Performance
Simulation with a
checklist
Learners will prepare the test
room as a part of the
simulation.
5.6 Application Performance
Simulation with a
checklist
Learners will greet the
participant as a part of the
simulation.
5.7 Application Performance
Simulation with a
checklist
Learners will inform the
purpose of the usability test
to the participant as a part of
the simulation.
5.8 Application Performance
Simulation with a
checklist
Learners will collect answers
to the pretest questionnaire
as a part the simulation.
5.9 Application Performance
Simulation with a
checklist
Learners will inform the
scenarios to perform to the
participants as a part the
simulation.
5.10 Application PerformanceSimulation with a
checklist
Learners will collect data as
a part the simulation.
5.11 Application Performance Simulation with a Learners will collect answers Copyright © 2013, Dalia Allencherry 14
EDTECH 503 – Instructional Design Projectchecklist to the posttest questionnaire
as a part the simulation.
5.12 Application Performance
Simulation with a
checklist
Learners will thank the
participant as a part the
simulation.
5.13 Application PerformanceSimulation with a
checklist
Learners will collect data
from the observer as a part
the simulation.
6.1 Application PerformanceObservation with
a checklist
Learners will identify
participants to conduct A/B
testing for the scenario
provided by the instructor.
6.2 Application PerformanceObservation with
a checklist
Learners will set two
wireframes or interfaces
ready for A/B usability test
for the scenario provided by
the instructor.
6.3 Application PerformanceObservation with
a checklist
Learners will develop
questions to conduct A/B
testing for the scenario
provided by the instructor.
6.4 Application PerformanceObservation with
a checklist
Learners will identify
participants to conduct A/B
testing for the scenario
provided by the instructor.
6.5 Application Performance Observation with
a checklist
Learners will inform the
participant of the questions
to conduct A/B testing for
the scenario provided by the
Copyright © 2013, Dalia Allencherry 15
EDTECH 503 – Instructional Design Projectinstructor.
6.6 Application PerformanceObservation with
a checklist
Learners will schedule
participants to take A/B
usability testing for the
scenario provided by the
instructor.
6.7 Application PerformanceObservation with
a checklist
Learners will collect the
answers to the questions
from the A/B testing for the
scenario provided by the
instructor.
7.1 Application PerformanceSimulation with a
checklist
Learners will identify the
software to use as a part of
the simulation.
7.2 Application PerformanceSimulation with a
checklist
Learners will develop the
scenarios for the participants
to perform as a part of the
simulation.
7.3 Application PerformanceSimulation with a
checklist
Learners will identify the
participants as a part of the
simulation.
7.4 Application PerformanceSimulation with a
checklist
Learners will set the
interface to test as a part of
the simulation.
7.5 Application PerformanceSimulation with a
checklist
Learners will schedule the
participants to take the test as
a part of the simulation.
7.6 Application Performance Simulation with a
checklist
Learners will collect the
results as a part of the
Copyright © 2013, Dalia Allencherry 16
EDTECH 503 – Instructional Design Projectsimulation.
8.1 Evaluation PerformanceSimulation with
checklist of skills
Learners will make
conclusions for each test goal
as a part of a simulation
8.2 Synthesis PerformanceSimulation with
checklist of skills
Learners will summarize
biographic information of
participants and other
metadata.
Part 3c: ARCS table
ATTENTION
A.1 Perceptual Arousal
The instructor will show a funny video on usability testing. The video is included in the
instructor guide.
A2. Inquiry Arousal
The instructor will ask what usability is and collect responses from all the learners. He or she
will inform the objective of the course to the learners.
A3. Variability
The instructor will ask questions about the learners’ job responsibilities and connect each feature
of usability testing to their work.
RELEVANCE
R1. Goal orientation
The instructor will use the learner analysis and need analysis results to decide the student’s needs
R2. Motive matching
The instructor will allow learners to choose the participants for their mock usability testing
sessions. The learners will have the freedom to choose a website of their choice to perform a
usability test.
Copyright © 2013, Dalia Allencherry 17
EDTECH 503 – Instructional Design ProjectR3. Familiarity
The learners will be asked to tie the features of usability testing to their work. The instructor will
ask questions regarding this and collect responses from the learners.
CONFIDENCE
C1. Learning requirements
The instructor will email the requirements of the course to the learners a week before the training
day. The instructor can assist in building a positive expectation for success by checking the
progress of the learners and guiding them throughout the course.
C2. Success opportunities
The learners will have the opportunity to conduct usability tests as a group and by themselves
and receive immediate feedback from the instructor and the peers.
C3. Personal control
The instructor will provide examples of the increase in revenue some websites earned after
updating the website based on usability testing results.
SATISFACTION
S1. Natural consequences
The instructor will inform the learners that usability testing will be added to their regular work
schedule.
S2. Positive consequences
The instructor will provide feedback during mock sessions and other activities during the course.
Also, the instructor will encourage the learners and provide them support to subscribe to feeds
and emails related to usability testing.
S3. Equity
The instructor will congratulate the learners when they complete the activities and mock sessions
as a part of the course. The instructor will also talk about the importance of usability testing in
relation to their job responsibilities. He or she can explain how usability tests lead to effective
websites, which further lead to more revenue from sales through the websites and eventually lead
to good pay raise for the learners.
Copyright © 2013, Dalia Allencherry 18
EDTECH 503 – Instructional Design Project
Part 4: Instructor Guide
Active Attention or Gain Attention
The instructor will start with a video on usability testing in order to gain attention. Provide
examples of usability testing: images of websites before and after usability testing (example) and
the increase in revenue though websites after usability testing (example). Continue to connect
each of the features of usability testing to their job responsibilities, throughout the course. Also,
talk about the importance of usability testing and its potential effect on the company website and
the business.
Establish Purpose or Inform Learners of Purpose
The instructor will inform learners that the purpose of the course is for them to acquire
knowledge on usability testing and conduct tests as a part of their job.
Arouse Interest and Motivation or Stimulate Learners’ Attention/Motivation
The instructor will give opportunities for learners to conduct mock sessions and other usability
testing activities during the course and provide feedback. Also, motivate the learners by tying
increase in website revenue due to usability testing to their performance evaluation and pay raise.
Preview the Learning Activity or Provide Overview
The instructor will provide an introduction to the course. He or she will inform the goal of the
course to the learners. They will learn about usability testing for websites: definition of usability
testing, different methods of the testing, the importance of testing, and the advantages of various
methods of usability testing. They will also get a chance to practice usability testing as a group as
well individually. Finally, they will subscribe to feeds and newsletters related to usability testing.
Body
Copyright © 2013, Dalia Allencherry 19
EDTECH 503 – Instructional Design ProjectRecall relevant prior knowledge or Stimulate recall of prior knowledge
The learners have very low prior knowledge on usability testing. The instructor will ask
questions in order to recall their prior knowledge. The target audience is web designers and
developers; therefore, they have a good knowledge of the various parts or features of a website
when it comes to testing a website’s features.
Process information and examples or Present information and examples
The instructor will introduce the definition of usability testing to the learners. The learners will
be presented with not only the explanation of usability testing and its various methods, but also
the opportunity to watch the instructor perform usability testing. The instructor will provide real-
time examples of Sargent Manufacturing website usability test and Sheraton website usability
test.
Focus Attention or Gain & Direct Attention
The instructor will continuously interact with the learners during the course. He or she will ask
questions and collect responses from the learners. There will be mock usability testing sessions
and other activities when learners participate and receive immediate feedback from the instructor
and the peers. There will be group discussions regarding how the learners can apply certain
characteristics (card sorting and tree testing) of usability testing in their jobs.
Employ Learning Strategies or Guide or Prompt Use of Learning Strategies
The instructor will concentrate on procedural learning in the course. The exercise of usability
tests include following a procedure. Strategies for declarative and conceptual learning are also
part of this course. The learners have very low prior knowledge in the field of usability testing.
The declarative learning strategies will help learners to acquire the knowledge of usability testing
in order to recall it at the stage of application. The strategies for concept learning will help the
participants learn the concept of usability testing. They will be able to identify the case and the
right method when a website requires usability testing.
Copyright © 2013, Dalia Allencherry 20
EDTECH 503 – Instructional Design ProjectPractice or Provide for and Guide Practice
The learners will have the opportunity to practice usability testing during the course. The
instructor will answer any questions they have and guide them through the process. He or she
will provide constructive feedback to the learners on their performance on mock usability
sessions and other activities. The learners will make revisions to their work based on the
feedback.
Evaluate Feedback or Provide Feedback
The instructor will observe the performance of the participants during the course. The learners
will receive immediate feedback on mock sessions from the instructor and the peers. The
learners will also have the opportunity to conduct personal usability tests and share the feedback
with the class to receive feedback regarding the method of usability test chose and the
characteristics of the website tested. The instructor will also provide feedback regarding the tone
of voice used during testing, testing room set-up, and eye tracking software used. There will be a
test at the end of the course and the learners will receive the result and the feedback.
Conclusion
Summarize and review or Provide summary and review
The instructor will summarize what the participants have learned. They learned about usability
testing, different methods of usability testing, and the importance and advantages of various
methods of usability testing. They also got a chance to practice usability testing as a group as
well as individually and received immediate feedback from the instructor and the peers. Finally,
they subscribed to feeds and newsletters related to usability testing in order to retain and transfer
the learning.
Transfer learning or Enhance transfer
The instructor will remind the learner that usability testing will be added to their regular work
schedule. He or she will encourage the learners to stay connected with the field of usability. The Copyright © 2013, Dalia Allencherry 21
EDTECH 503 – Instructional Design Projectinstructor will suggest practicing usability testing of websites for training purpose and encourage
learners to make use of the opportunity. In the future, he learners will also get chances to
contribute to white papers and do ROI calculations regarding usability testing of the company
websites.
Remotivate and Close or Provide Remediation and Closure
The instructor will emphasize the importance of usability testing and its connection to the
learners’ job responsibilities. Show an example of a website that has gone through usability
testing. Stress the connection between increased revenue through usability-tested websites and
the learners’ pay raise. Compare the motivation levels of the learners before the course (learner
analysis survey) and after the course (course feedback questionnaire). Conduct mock usability
test sessions and provide immediate feedback.
Assess Learning or Conduct Assessment Evaluate
The instructor will assess the learners on each of the learning objectives. He or she will also
monitor the participation of the learners in group discussions and mock sessions. The learners
will receive the result of the end of course evaluation.
Feedback and Seek Remediation or Provide Feedback and Remediation
The instructor will provide resources for learners to stay in connect with the field of usability.
Introduce them to options like subscription to feeds (UX Matters and usability.gov) and
newsletters (Human Factors). Congratulate them on completing the course and encourage them
to earn a certification on usability analysis and testing. Suggest the organizer or management to
create a usability club with scheduled and continuous meetings every two months to talk about
usability or present team members’ experience with usability. The instructor will provide the end
of course result and feedback to the learners.
Copyright © 2013, Dalia Allencherry 22
EDTECH 503 – Instructional Design ProjectPart 5: Learner Content
5a: Learning materials
The learners will be provided the following learning materials to use during the instructional
time.
Learning materials PurposeTest start handout (see
appendix A)
• Learners can use as a reference to perform activities.
• Learners can use as a quick reference guide for the future
Pretest questionnaire handout
(see appendix B)
• Learners can use as a reference to perform activities.
• Learners can use as a quick reference guide for the future
Posttest questionnaire
handout (see appendix C)
• Learners can use as a reference to perform activities.
• Learners can use as a quick reference guide for the future
Learner guide
• Learners can follow along with the instructor
• Learners can use as a reference guide for the future
5b: Assessment materials
The following assessment materials will be used as a part of the instruction.
Assessment methods PurposeEvaluation of objectives (see
Appendix D) • The achievement of objectives can be measured
Performance checklist (see
Appendix E)
• Instructor will evaluate learners based on the checklist
• Learners can self-evaluate based on the checklist
• Learners are aware of the evaluation methods
5c: Technology tool rationale
The following technology tools will be used as a part of the instruction.
Copyright © 2013, Dalia Allencherry 23
EDTECH 503 – Instructional Design Project
Technology tool Purpose
Computers
• Learners can complete activities that includes accessing websites
• Learners can access the online test at the end of the course
Projector
• Instructor can project the power point presentation on a screen for
learners to view
Websites
• Human Factors
• Usability.gov
• UX Matters
• Sears
• Hertz
• Cheapoair
• YouTube
• Learners can subscribe to feeds and newsletters
• Learners can subscribe to feeds
• Learners can subscribe to feeds
• Learners can conduct mock usability test
• Learners can conduct mock usability test
• Learners can conduct mock usability test
• Learners can watch an example of usability test
Tobii Studio • Learners can practice usability testing using this eye-tracking software
Part 6: Formative Evaluation Plan
6a: Expert review plan
The SME is my colleague Bincy Philip, who is a certified usability analyst (CUA from HFI). She
is an expert in user-centered analysis, conceptual design, practical usability testing, effective web
and application design, and usability research. She regularly conducts usability tests for as a part
of her job. The instructional materials were submitted to her on Friday, July 26. She completed
the review during the weekend and got the package back to me on Monday, July 29. The
following list of questions was sent to her along with the review package.
1. Does the course cover all the required content based on the course goal?
2. Are the objectives achievable based on the content covered?
3. Are the examples and other scenarios given in the course realistic?
4. Does the instruction flow in a logical order?
Copyright © 2013, Dalia Allencherry 24
EDTECH 503 – Instructional Design Project5. Do you suggest any changes to the course content?
6. Do you consider any part of the course as unnecessary?
7. What are the strengths of the course?
8. What are the weaknesses of the course?
9. Would you add anything to engage the learners more? Please explain.
6b: One-to-one review plan
The goal of the one-to-one evaluation is to review the course on a small scale. I will identify two
people who represent the target audience. I will have one-to-one meetings with each of them. I
will make it clear that the goal of the meeting is to evaluate the course material, not them. The
participants will evaluate the instructional materials. I will ask specific questions and collect
feedback.
1. Does the instructor guide follow a logical order?
2. Are the objectives clear and appropriate considering the project goal?
3. Are the objectives attainable by web designers and developers?
4. Do you think the graphics and videos are appealing?
5. Can you relate the examples to your work situations?
6. Are there any subtopics that need to be explained further?
7. Will you be able to understand the course materials without instructor support?
8. If you were the instructor, what changes would you make?
6c: Small group evaluation plan
Small group evaluation is another type of evaluation. I will identify four people who represent
the target audience for this evaluation. I will meet with the participants as a group. This is a good
opportunity to check the validity of the instructional materials in the context of a group setting.
All the group activities will be tested at this time. I will ask the following questions to the
participants.Copyright © 2013, Dalia Allencherry 25
EDTECH 503 – Instructional Design Project1. Are the instructions for the group activities clear?
2. Were you able to complete the group activities within the given time?
3. Were you able to conduct usability tests with the instructions given in the learner guide
and without instructor support?
4. Did you find any information that is not relevant based on the course goal?
5. Did you feel engaged throughout the course?
6. What is your level of satisfaction regarding the course delivery? Rate 1-5
7. What do you think can be improved about the instruction and the materials?
6d: Field trial plan
Field trial is the opportunity to test the changes made based on the feedback received during the
one-to-one and the small group evaluations. I will select a sample of six people representing the
target audience for this evaluation. The room will be set to mimic the real-time training room.
There will be a feedback session at the end where I will ask specific questions and collect
feedback.
1. Does the instruction flow in a logical order? If not, please explain.
2. Are the objectives clear and appropriate considering the course goal? If not, please
explain.
3. Is the instruction easy to follow? If not, please explain.
4. Were you engaged throughout the course? If not, please explain.
5. Do you think any additional information needs to be added to the course? Please explain.
6. Is the location appropriate for the course delivery? Please explain.
7. Was there an ample supply of materials (pens, sticky notes, highlighters, etc.) required?
Please explain
8. Do you suggest any changes to the course? Please explain
9. Would you add anything to engage the learners more? Please explain.
Copyright © 2013, Dalia Allencherry 26
EDTECH 503 – Instructional Design ProjectPart 7: Formative Evaluation Report
7a: Evaluation survey
The following expert evaluation form was provided to the subject matter expert.
Source: http://freeonlinesurveys.com/s.asp?sid=7rv8a7uncbvu9co307668
7b: Report of expert review
My SME provided valuable insights and suggestions after reviewing the course materials. She
thinks that the course covers all the content required in order to achieve the course goal. The
objectives are connected to the course goals and are achievable based on the course. Even though
the scenarios used in the course are realistic, the SME suggests changing some of the scenarios
with the ones related to the target audience’s work. They can be the from the company website
the target audience uses as a part of their work or any other e-commerce website. The SME
thinks the course follows a logical sequence. She recommends making the steps to perform a
usability test clearer. She suggests this because the target audience has very little prior
knowledge of the procedure and none of them have done this before. She couldn’t identify any
unnecessary content in the course. She identifies the group as well as the individual activities or
in other words, mock sessions or simulations as the biggest strength of the course. She also likes
the idea of helping learners to subscribe to feeds and newsletters in order to stay connected with
the subject. She thinks the idea of “Lunch and Learn” sessions to discuss usability and present
team members’ experience may not do well because people may choose not to attend these
sessions. “Lunch and Learn” sessions have not been very successful in the company. She
considers the course as very engaging due to the presence of the group and individual activities.
Overall, the feedback was positive.
7c: Designer’s response to review
As a result of the expert review, I made a few changes to the course. I broke down some of the
steps of the usability testing procedure into two and added notes or tips about conducting some
Copyright © 2013, Dalia Allencherry 27
EDTECH 503 – Instructional Design Projectof the steps. For e.g. In the case of the step “Schedule interviews with the participants”, I added
the note “Decide if the interview is direct or virtual and monitored or unmonitored”. Even though
I talked about conducting virtual interviews in the course, I forgot to add it as a part of the
procedure. I replaced two of the websites selected for usability testing (BBC news and
Dictionary.com) with e-commerce websites (Hertz and Cheapoair) according to the SME’s
feedback. Based on the SME’s concerns regarding team members’ attendance to “Lunch and
Learn” sessions, I chose to recommend forming a usability club and have scheduled and
continuous meetings every month or two to discuss usability and also present team members’
experience with usability.
Part 8: Standards Grid
Professional Standards Addressed (AECT)
The following standards, developed by the Association for Educational Communications and
Technology (AECT), and used in the accreditation process established by the National Council
for Accreditation of Teacher Education (NCATE), are addressed to some degree in this course.
The numbers of the standards correspond to the numbers next to the course tasks show on the list
of assignments. Not all standards are addressed explicitly through student work.
Assignments meeting standard in whole or part
Standard 1: DESIGN
1.1 Instructional Systems Design
(ISD) X
ID Project
1.1.1 Analyzing X ID Project
1.1.2 Designing X ID Project
1.1.3 Developing X ID Project
1.1.4 Implementing X ID Project
1.1.5 Evaluating X Selected Discussion Forums; ID Project
1.2 Message Design
1.3 Instructional Strategies X ID Project
1.4 Learner Characteristics X ID Project
Copyright © 2013, Dalia Allencherry 28
EDTECH 503 – Instructional Design ProjectStandard 2: DEVELOPMENT
2.0 (includes 2.0.1 to 2.0.8) X ID Project
2.1 Print Technologies X Reading Quiz; ID Projects
2.2 Audiovisual Technologies
2.3 Computer-Based Technologies X (all assignments)
2.4 Integrated Technologies
Standard 3: UTILIZATION
3.0 (includes 3.0.1 & 3.0.2)
3.1 Media Utilization X (all assignments)
3.2 Diffusion of Innovations
3.3 Implementation and
Institutionalization X
ID Project
3.4 Policies and Regulations
Standard 4: MANAGEMENT
4.0 (includes 4.0.1 & 4.0.3)
4.1 Project Management
4.2 Resource Management
4.3 Delivery System Management
4.4 Information Management
Standard 5: EVALUATION
5.1 Problem Analysis X
5.2 Criterion-Referenced
Measurement X
ID Project
5.3 Formative and Summative
Evaluation X
ID Project
5.4 Long-Range Planning
COURSE GOALS & OBJECTIVES
Copyright © 2013, Dalia Allencherry 29
EDTECH 503 – Instructional Design ProjectThe overall goal for the course is for each student to consider and use the systematic process of
instructional design to create an instructional product. To achieve this goal, students will engage
in activities that promote reflective practice, emphasize realistic contexts, and employ a number
of communications technologies. Following the course, students will be able to:
1. Discuss the historical development of the practice of instructional design with regard to
factors that led to its development and the rationale for its use
2. Describe at least two reasons why instructional design models are useful
3. Identify at least six instructional design models and classify them according to their use
4. Compare and contrast the major elements of three theories of learning as they relate to
instructional design
5. Define “instructional design.”
6. Define the word “systematic” as it relates to instructional design
7. Define “learning” and synthesize its definition with the practice of instructional design
8. Relate the design of instruction to the term “educational (or “instructional”) technology”
9. Describe the major components of the instructional design process and the functions of
models in the design process
10. Provide a succinct summary of various learning contexts (declarative knowledge,
conceptual, declarative, principle, problem-solving, cognitive, attitudinal, and
psychomotor)
11. Build an instructional design product that integrates major aspects of the systematic
process and make this available on the web.
a. Describe the rationale for and processes associated with needs, learner, context,
goal, and task analyses
i. Create and conduct various aspects of a front-end analysis
ii. Identify methods and materials for communicating subject matter that are
contextually relevant
b. Describe the rationale for and processes associated with creating design
documents (objectives, motivation, etc.)
i. Construct clear instructional goals and objectives
Copyright © 2013, Dalia Allencherry 30
EDTECH 503 – Instructional Design Projectii. Develop a motivational design for a specific instructional task
iii. Develop assessments that accurately measure performance objectives
c. Select and implement instructional strategies for selected learning tasks
i. Select appropriate media tools that support instructional design decisions
d. Describe the rationale and processes associated with the formative evaluation of
instructional products
i. Create a plan for formative evaluation
12. Identify and use technology resources to enable and empower learners with diverse
backgrounds, characteristics, and abilities.
13. Apply state and national content standards to the development of instructional products
14. Meet selected professional standards developed by the Association for Educational
Communications and Technology
15. Use various technological tools for instructional and professional communication
AECT STANDARDS (Applicable to EDTECH 503)
1.0 Design
1.1 Instructional Systems Design
1.1.a Utilize and implement design principles which specify optimal conditions for
learning.
1.1.b Identify a variety of instructional systems design models and apply at least one
model.
1.1.1 Analyzing
1.1.1.a Write appropriate objectives for specific content and outcome levels.
1.1.1.b Analyze instructional tasks, content, and context.
1.1.2 Designing
Copyright © 2013, Dalia Allencherry 31
EDTECH 503 – Instructional Design Project1.1.2.a Create a plan for a topic of a content area (e.g., a thematic unit, a text chapter, an
interdisciplinary unit) to demonstrate application of the principles of macro-level design.
1.1.2.b Create instructional plans (micro-level design) that address the needs of all
learners, including appropriate accommodations for learners with special needs.
1.1.2.d Incorporate contemporary instructional technology processes in the development
of interactive lessons that promote student learning.
1.1.3 Developing
1.1.3.a Produce instructional materials which require the use of multiple media (e.g.,
computers, video, projection).
1.1.3.b Demonstrate personal skill development with at least one: computer authoring
application, video tool, or electronic communication application.
1.1.4 Implementing
1.1.4.a Use instructional plans and materials which they have produced in contextualized
instructional settings (e.g., practice, field experiences, and training) that address the needs
of all learners, including appropriate accommodations for learners with special needs.
1.1.5 Evaluating
1.1.5.a Utilize a variety of assessment measures to determine the adequacy of learning
and instruction.
1.1.5.b Demonstrate the use of formative and summative evaluation within practice and
contextualized field experiences.
1.1.5.c Demonstrate congruency among goals/objectives, instructional strategies, and
assessment measures.
1.3 Instructional Strategies
1.3.a Select instructional strategies appropriate for a variety of learner characteristics and
learning situations.
Copyright © 2013, Dalia Allencherry 32
EDTECH 503 – Instructional Design Project1.3.b Identify at least one instructional model and demonstrate appropriate
contextualized application within practice and field experiences.
1.3.c Analyze their selection of instructional strategies and/or models as influenced by the
learning situation, nature of the specific content, and type of learner objective.
1.3.d Select motivational strategies appropriate for the target learners, task, and learning
situation.
1.4 Learner Characteristics
1.4.a Identify a broad range of observed and hypothetical learner characteristics for their
particular area(s) of preparation.
1.4.b Describe and/or document specific learner characteristics which influence the
selection of instructional strategies.
1.4.c Describe and/or document specific learner characteristics which influence the
implementation of instructional strategies.
2.0 Development
2.0.1 Select appropriate media to produce effective learning environments using
technology resources.
2.0.2 Use appropriate analog and digital productivity tools to develop instructional and
professional products.
2.0.3 Apply instructional design principles to select appropriate technological tools for
the development of instructional and professional products.
2.0.4 Apply appropriate learning and psychological theories to the selection of
appropriate technological tools and to the development of instructional and professional
products.
2.0.5 Apply appropriate evaluation strategies and techniques for assessing effectiveness
of instructional and professional products.
Copyright © 2013, Dalia Allencherry 33
EDTECH 503 – Instructional Design Project2.0.6 Use the results of evaluation methods and techniques to revise and update
instructional and professional products.
2.0.7 Contribute to a professional portfolio by developing and selecting a variety of
productions for inclusion in the portfolio.
2.1 Print Technologies
2.1.3 Use presentation application software to produce presentations and supplementary
materials for instructional and professional purposes.
2.1.4 Produce instructional and professional products using various aspects of integrated
application programs.
2.3 Computer-Based Technologies
2.3.2 Design, produce, and use digital information with computer-based technologies.
3.0 Utilization
3.1 Media Utilization
3.1.1 Identify key factors in selecting and using technologies appropriate for learning
situations specified in the instructional design process.
3.1.2 Use educational communications and instructional technology (SMETS) resources
in a variety of learning contexts.
3.3 Implementation and Institutionalization
3.3.1 Use appropriate instructional materials and strategies in various learning contexts.
3.3.2 Identify and apply techniques for integrating SMETS innovations in various
learning contexts.
3.3.3 Identify strategies to maintain use after initial adoption.
4.0 Management
(none specifically addressed in 503)
5.0 Evaluation
Copyright © 2013, Dalia Allencherry 34
EDTECH 503 – Instructional Design Project5.1 Problem Analysis
5.1.1 Identify and apply problem analysis skills in appropriate school media and
educational technology (SMET) contexts (e.g., conduct needs assessments, identify and
define problems, identify constraints, identify resources, define learner characteristics,
define goals and objectives in instructional systems design, media development and
utilization, program management, and evaluation).
5.2 Criterion-referenced Measurement
5.2.1 Develop and apply criterion-referenced measures in a variety of SMET contexts.
5.3 Formative and Summative Evaluation
5.3.1 Develop and apply formative and summative evaluation strategies in a variety of
SMET contexts.
SMET = School Media & Educational Technologies
Copyright © 2013, Dalia Allencherry 35
EDTECH 503 – Instructional Design Project
Appendices
Appendix A
Test Start Manual
Welcome and Purpose
Thank you for agreeing to participate
Our goal is to evaluate the website (you are testing) from a user’s perspective.
Test Facilitator’s and Observer’s Role
As facilitator, I will ask a series of questions while the observer’s role is to record your
responses
During this session, I may not be able to offer any suggestions or hints.
There may be times, when I will ask you to explain why you said or did something.
Test Participant’s Role
Simply perform tasks on this site as you would if no one was around. Please “think out
loud” when performing tasks.
You will also be asked a series of questions about your experience at the end of this
session.
Things to Keep in Mind
This is not a test of you; we are testing the site. So, don not worry about making
mistakes, there is no right or wrong answer.
We just want to know if we designed the site well for you.
If you ever feel that you are lost or cannot complete a scenario with the information that
you have been given, please let me know. I will ask you what you might do in a real-
world setting and then either put you on the right track or move you to the next scenario.
Copyright © 2013, Dalia Allencherry 36
EDTECH 503 – Instructional Design Project Finally, as you use the site, please do so as you would be on your own.
Appendix B
Pretest Questionnaire
1. What is your current position and title?
2. How long have you held this position and been with the company?
________ in position __________ with the company
Computer Expertise
1. What would you consider as your level of computer expertise? (Create a scale)
Low 1 2 3 4 5 High
2. Besides reading email, what are typical activities you do on the computer?
Gaming/entertainment Reading news Shopping/banking
Graphic design/digital pictures
Programming/word processing
Social Media
Other:
3. Share 2-3 of your favorite web sites that you frequently visit. Why are they your favorite?
Website Reasons
4. What computer platform do you usually use? [e.g. Mac, Windows XP, etc.]
5. What Internet browser(s) do you usually use? [e.g., Firefox, IE v.6, AOL, Chrome, etc.]
Copyright © 2013, Dalia Allencherry 37
EDTECH 503 – Instructional Design Project6. Did you use the website (you are testing) in the last year? How often do you visit the
website?
Appendix C
Post Test Questionnaire
1. What are your overall impressions of the website (you are testing)?
2. If you had to give the site a grade, from A to F, where A as exemplary and F as failing,
what grade would you give the website, and why?
A B C D F
3. Name three words or characteristics that describe the website.
4. What are the three things you like best about the website?
5. What are the three things you like least about the website?
6. If you could make one significant change to the website, what change would you make?
7. Can you describe the site’s purpose?
8. Why would someone go to the website?
9. Would you return to the website in the future? Why/why not?
10. What would entice you to return?
11. Do you have any other questions or comments about the website or your experiences with
it?
Copyright © 2013, Dalia Allencherry 38
EDTECH 503 – Instructional Design Project
Appendix D
Objectives End of course assessment1.1 Learners will define usability testing in their own words Define usability testing.
1.2 Learners will explain the purpose of usability testing
Which of the following is a purpose of usability testing? Select all that apply.A. Identify technical errorsB. Identify users' mental modelC. Identify organization of tasksD. Identify issues of a site after build and before release
1.3 Learners will explain the importance of usability testing Explain the importance of usability testing.
1.4 Learners will list three methods of usability testing
Which of the following are methods of usability testing?A. Usability testing using interviews.B. Usability testing using an eye tracking software.C. A/B usability testingD. Peer testing
1.5 Learners will list the characteristics of a website that need to be evaluated
Which of the following characteristics of a website need to be evaluated during usability testing?A. Information architectureB. Color combinationC. Company logoD. Website technical issues
1.6 Learners will explain hallway usability testing What is hallway usability testing?1.7 Learners will explain remote usability testing What is remote usability testing?
2.1 Learners will explain usability testing using interviews in their own words
Match the following.Column 1: (Usability testing using interviews, A/B usability testing, and Usability testing using an eye tracking software)Column 2: Larger sample size enables statistically reliable metrics, Most suitable to measure data such as sales percentages, subscription rates, etc., and Can automatically decide where the user’s interest is concentrated
Copyright © 2013, Dalia Allencherry 39
EDTECH 503 – Instructional Design Project2.1 Learners will recall the advantages of usability testing using interviews compared to other methods
Explain the advantages of usability testing using interviews over other methods.
3.1 Learners will explain A/B testing in their own words
Match the following.Column 1: (Usability testing using interviews, A/B usability testing, and Usability testing using an eye tracking software)Column 2: Larger sample size enables statistically reliable metrics, Most suitable to measure data such as sales percentages, subscription rates, etc., and Can automatically decide where the user’s interest is concentrated
3.2 Learners will recall the advantages of A/B usability testing compared to other methods
Explain the advantages of A/B usability testing over other methods.
4.1 Learners will explain usability testing using an eye tracking software
Match the following.Column 1: (Usability testing using interviews, A/B usability testing, Usability testing using an eye tracking software)Column 2: Larger sample size enables statistically reliable metrics, Most suitable to measure data such as sales percentages, subscription rates, etc., Can automatically decide where the user’s interest is concentrated
4.2 Learners will recall the advantages of usability testing using an eye tracking software compared to other methods
Explain the advantages usability testing using an eye tracking software over other methods.
Note: The rest of the objectives are measured by observation or simulation using a checklist (see appendix E).
Copyright © 2013, Dalia Allencherry 40
EDTECH 503 – Instructional Design Project
Appendix E
Required Tasks Completed? Y/NEarn at least 80% marks in the end of the course assessment
Usability testing using interviews - Group activity
Analyze test case and goalsIdentify participantsIdentify an observerIdentify interviewerCreate scenarios to test
Schedule interviews with the participantsGreet the participantCollect answers to the questions in the pretest questionnaireInform scenarios for the participants to perform and collect data
Collect answers to the posttest questionnaire
Thank the participantCollect data from the observer
Make conclusions for each test goal
Summarize biographic information of participants and other metadata
A/B usability testing - Individual activity
Analyze test case and goalsSet two wireframes or interfaces ready for A/B usability test
Write questions to ask participants
Identify participantsInform or present the questions to the participants and collect dataMake conclusions for each test goalSummarize biographic information of participants and other metadata
Copyright © 2013, Dalia Allencherry 41
EDTECH 503 – Instructional Design ProjectUsability testing using an eye tracking software - Individual activity
Analyze the test case and goals
Identify software to use (Tobii Studio as decided by the company)Develop scenarios to testIdentify participantsSet interface to test
Schedule the participants to take the test.
Collect data from participant responses
Make conclusions for each test goalSummarize biographic information of participants and other metadata
Copyright © 2013, Dalia Allencherry 42