Design Research for a quality experience

National Software Testing Conference 2016Carmen Brion

Tech Reliability and Usability are basic requirements in a digital product for customers

To find competitive advantage we need to understand the specific Jobs To Be Done that solve problems for our core audience

Lack of Design Research, or leaving it too late, increases the risk of failure

IMVU, Eric Ries’ first startup, was initially a failure because the team never undertook any research to assess the idea and strategy.

“Listen old man, you don’t understand. What is the deal with this crazy business of inviting friends before I know if it is cool”

Source: Lean Startup by Eric Ries

Continuous Design Research increases likelihood of success, even in difficult markets

Intuit became the leader in the difficult market of personal finance by investing in understanding the jobs to be done for their core audience.

Quicken was the result of:

● early generative research

● recurrent usability testing while building

● public beta versions before launch

Source: The Lean Product Playbook by Dan Olsen

Postponing or not doing Design Research will affect the product metrics

Acquisition

Activation

Retention

Revenue

Referral

How do customers find you?

Do they have a great first experience?

Do they come back?

How do you make money?

Do they tell others?

Becomes harder and more expensive

Reduces brand loyalty and likelihood of retention

“If you make a customer unhappy in the physical world they might tell 6 friends, if you do so on the internet, they can tell 6,000” Jeff Bezos

Unhappy customers are less likely to come back

Increase of costs by rework and customer support

Effects of bad experiences

Source: Lean Startup Pirate Metrics (AARRR)

“Your job isn’t to build more software faster: it’s to maximize the outcome and impact you get from what you choose to build” Jeff Patton

● QA and Usability Testing just guarantee usable software outputs● To meet customer outcomes we need continuous and adequate

Design Research

Customers buy and use a product because it does a job for them

Job To Be Done (JTBD)

Context of use

Desired OUTCOME

Product/Service

● What outcome is the customer trying to achieve?

● What are the problems and pain points getting in the way of the outcomes?

● How can they achieve outcomes in a way that also allows the business to reach its goals?

DigitalAbility

Expectations from other solutions

JTBDAbility

Who is the audience?What are their JTBD?What are they currently using?

How can we do the job better?Which solutions do the job?

Which solution does the job better?

Is the solution easy to use?How effective is the solution at doing the job?

Which is the most efficient variant to get the job done?Why might one variant work better than the other(s)?

Evaluative research

Generative research

Product Exploration

To get outcomes we need Continuous Design Research

Product Development Product Validation

To get outcomes we need Continuous Design Research

EthnographyDiary studiesContextual inquiry

InterviewsCo-creation workshop

Tree testCard sorts Click testsPrototype testing

MVT, A/B testingPath analysis, HeatmapsRecorded sessionsFeedback surveysUsability testing

Generative research

Evaluative research

Usability testing

SurveysBenchmarkDesk researchAnalytics

Evidence-based Output Outcome

Product Development Product ValidationProduct Exploration

Research/Design Framework

DEFINESOLUTION

ASSESSMENTSOLUTION

EXPLORATIONPROBLEM

DEFINITIONPROBLEM

EXPLORATION

HYPOTHESES AND

JOBS TO BE DONE

DEFINITION

“Few ideas work on the first try. Iteration is key to innovation” ― Sebastian Thrun

“Design Research is a team sport” GDS

1 32 4 5 6 7 8 9

Kick off meetingDetermine the assumptions and questions

Our research process for prototype testing

Prototype definitionDetermine and agree what to prototype

Recruitment specAgree participants profiles and start recruitment

Sessions planDefine all hypotheses to test

Prototype reviewsThe core team assesses progress and discusses any changes

Discussion guideDetermine and triangulate tasks and questions for participants

Run the sessionsTeam and stakeholders practice ‘Active Observation’

Sessions washupReview and agree the main themes and insights with the core team

Insights presentationPresent insights and hypotheses outcomes to the core team and stakeholders

2 weeks 1 week

“[Both QA and UAT] follow repetitive and set user paths that assume a level of rational predictability which is rarely present in human behaviour” Jamie Mayers, Made by Many

How can we cater for the user paths we may not have predicted?

QA that goes beyond checking that technical requirements have been met

Product Development

USABILITY TESTING and QA

UAT

● Work with UX and Development to prioritise bug fixes taking into account potential benefits to the personas and the impact to their JTBD

● Get ideas from the following in order to define test cases:○ using the product personas to understand the context of use, JTBD,

ability and level of expertise○ observe usability sessions to gather inspiration on how customers

use the product and potential unusual paths/usage

Product ValidationProduct Exploration

UAT that goes beyond an acceptable system

USABILITY TESTING and QA

UAT

Business Analysts can collaborate with UX to:● Define key User Journeys to assess● Define an experience rating that includes how well the product meets:

○ the experience principles ○ the JTBD for each User Journey

Product Development Product ValidationProduct Exploration

But it is really unlikely that we can predict all unusual paths that people might take

Give the product to your customers, let them play, see where it breaks for them

Photo by Storem

Product Development

USABILITY TESTING and QA

UAT

Product Validation

Evidence-based Output Outcome

Let your customers be the product testers

MVT, A/B testingPath analysis, HeatmapsRecorded sessionsUsability testing

Run usability sessions and complement with a Beta test or a product validation phase.

“Alone we can do so little; together we can do so much”- Helen Keller

Photo by Easa Shamih

Thank you!You can find me @Tea_monster

References

Irrational realism: on QA vs Testing, by Jamie Mayeshttps://medium.com/the-many/irrational-realism-on-qa-vs-testing-7fd8618f7fdb#.q9b3wxosm

It’s not just semantics: managing outcomes vs outputs, by Deborah Mills-Scofieldhttps://hbr.org/2012/11/its-not-just-semantics-managing-outcomes

The neglected practice of iteration, by Jeff Pattonhttps://www.agileconnection.com/article/neglected-practice-iteration

Early usability testing decides the product’s destiny, by Tamara Rendulichttp://www.paulolyslager.com/early-usability-testing-decides-product-destiny/?utm_content=buffer4482d&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer