Design Research for a quality experience
National Software Testing Conference 2016Carmen Brion
Tech Reliability and Usability are basic requirements in a digital product for customers
To find competitive advantage we need to understand the specific Jobs To Be Done that solve problems for our core audience
Lack of Design Research, or leaving it too late, increases the risk of failure
IMVU, Eric Ries’ first startup, was initially a failure because the team never undertook any research to assess the idea and strategy.
“Listen old man, you don’t understand. What is the deal with this crazy business of inviting friends before I know if it is cool”
Source: Lean Startup by Eric Ries
Continuous Design Research increases likelihood of success, even in difficult markets
Intuit became the leader in the difficult market of personal finance by investing in understanding the jobs to be done for their core audience.
Quicken was the result of:
● early generative research
● recurrent usability testing while building
● public beta versions before launch
Source: The Lean Product Playbook by Dan Olsen
Postponing or not doing Design Research will affect the product metrics
Acquisition
Activation
Retention
Revenue
Referral
How do customers find you?
Do they have a great first experience?
Do they come back?
How do you make money?
Do they tell others?
Becomes harder and more expensive
Reduces brand loyalty and likelihood of retention
“If you make a customer unhappy in the physical world they might tell 6 friends, if you do so on the internet, they can tell 6,000” Jeff Bezos
Unhappy customers are less likely to come back
Increase of costs by rework and customer support
Effects of bad experiences
Source: Lean Startup Pirate Metrics (AARRR)
“Your job isn’t to build more software faster: it’s to maximize the outcome and impact you get from what you choose to build” Jeff Patton
● QA and Usability Testing just guarantee usable software outputs● To meet customer outcomes we need continuous and adequate
Design Research
Customers buy and use a product because it does a job for them
Job To Be Done (JTBD)
Context of use
Desired OUTCOME
Product/Service
● What outcome is the customer trying to achieve?
● What are the problems and pain points getting in the way of the outcomes?
● How can they achieve outcomes in a way that also allows the business to reach its goals?
DigitalAbility
Expectations from other solutions
JTBDAbility
Who is the audience?What are their JTBD?What are they currently using?
How can we do the job better?Which solutions do the job?
Which solution does the job better?
Is the solution easy to use?How effective is the solution at doing the job?
Which is the most efficient variant to get the job done?Why might one variant work better than the other(s)?
Evaluative research
Generative research
Product Exploration
To get outcomes we need Continuous Design Research
Product Development Product Validation
To get outcomes we need Continuous Design Research
EthnographyDiary studiesContextual inquiry
InterviewsCo-creation workshop
Tree testCard sorts Click testsPrototype testing
MVT, A/B testingPath analysis, HeatmapsRecorded sessionsFeedback surveysUsability testing
Generative research
Evaluative research
Usability testing
SurveysBenchmarkDesk researchAnalytics
Evidence-based Output Outcome
Product Development Product ValidationProduct Exploration
Research/Design Framework
DEFINESOLUTION
ASSESSMENTSOLUTION
EXPLORATIONPROBLEM
DEFINITIONPROBLEM
EXPLORATION
HYPOTHESES AND
JOBS TO BE DONE
DEFINITION
“Few ideas work on the first try. Iteration is key to innovation” ― Sebastian Thrun
“Design Research is a team sport” GDS
1 32 4 5 6 7 8 9
Kick off meetingDetermine the assumptions and questions
Our research process for prototype testing
Prototype definitionDetermine and agree what to prototype
Recruitment specAgree participants profiles and start recruitment
Sessions planDefine all hypotheses to test
Prototype reviewsThe core team assesses progress and discusses any changes
Discussion guideDetermine and triangulate tasks and questions for participants
Run the sessionsTeam and stakeholders practice ‘Active Observation’
Sessions washupReview and agree the main themes and insights with the core team
Insights presentationPresent insights and hypotheses outcomes to the core team and stakeholders
2 weeks 1 week
“[Both QA and UAT] follow repetitive and set user paths that assume a level of rational predictability which is rarely present in human behaviour” Jamie Mayers, Made by Many
How can we cater for the user paths we may not have predicted?
QA that goes beyond checking that technical requirements have been met
Product Development
USABILITY TESTING and QA
UAT
● Work with UX and Development to prioritise bug fixes taking into account potential benefits to the personas and the impact to their JTBD
● Get ideas from the following in order to define test cases:○ using the product personas to understand the context of use, JTBD,
ability and level of expertise○ observe usability sessions to gather inspiration on how customers
use the product and potential unusual paths/usage
Product ValidationProduct Exploration
UAT that goes beyond an acceptable system
USABILITY TESTING and QA
UAT
Business Analysts can collaborate with UX to:● Define key User Journeys to assess● Define an experience rating that includes how well the product meets:
○ the experience principles ○ the JTBD for each User Journey
Product Development Product ValidationProduct Exploration
But it is really unlikely that we can predict all unusual paths that people might take
Give the product to your customers, let them play, see where it breaks for them
Photo by Storem
Product Development
USABILITY TESTING and QA
UAT
Product Validation
Evidence-based Output Outcome
Let your customers be the product testers
MVT, A/B testingPath analysis, HeatmapsRecorded sessionsUsability testing
Run usability sessions and complement with a Beta test or a product validation phase.
“Alone we can do so little; together we can do so much”- Helen Keller
Photo by Easa Shamih
Thank you!You can find me @Tea_monster
References
Irrational realism: on QA vs Testing, by Jamie Mayeshttps://medium.com/the-many/irrational-realism-on-qa-vs-testing-7fd8618f7fdb#.q9b3wxosm
It’s not just semantics: managing outcomes vs outputs, by Deborah Mills-Scofieldhttps://hbr.org/2012/11/its-not-just-semantics-managing-outcomes
The neglected practice of iteration, by Jeff Pattonhttps://www.agileconnection.com/article/neglected-practice-iteration
Early usability testing decides the product’s destiny, by Tamara Rendulichttp://www.paulolyslager.com/early-usability-testing-decides-product-destiny/?utm_content=buffer4482d&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer