e    

 

Presented by: 

Ray  rell 

Brought to you by: 

  

340 Corporate Way, Suite   Orange Park, FL 32073 888‐2

K3 Keynot4/9/20148:30 AM     

“The Art of Complex System Testing”  

 A

Intel      

    

300,68‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com 

Ray Arell Intel  

As director of Intel Emergent Systems and Coaching, Ray Arell is a transformative leader in the adoption of agile, lean, and complex system methods inside Intel. Ray’s group is currently coaching a community of practice of more than ten-thousand people who are moving to a continuous value delivery culture. Prior to this role, he spent several decades as both an engineer and engineering manager of teams focused on CPU, chipset, graphics, wireless, and software development. Ray is a popular speaker at events worldwide and coauthor of Change-Based Test Management: Improving the Software Validation Process.  

2/26/2014

1

THE ART OF COMPLEX SYSTEM THE ART OF COMPLEX SYSTEM TESTINGTESTINGRay Arell, Director 

Intel Emergent Systems and Coaching 

Copyright © 2014 Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.  

2/26/2014

2

Source: http://en.wikipedia.o

2/26/2014

3

2/26/2014

4

Source: Flickr.com 

2/26/2014

5

nts

Many

Chaotic

Requ

ireme

Fe

Complex

SimpleComplicated

Technology Far fromcertainty

Close tocertainty

Few

KnowledgeHigh Low

Simple

Source: Flickr.com 

2/26/2014

6

2/26/2014

7

Source: http://www.calresco.org/lucas/cas.htm

You can't connect the dots looking forward; you can only connect them looking backwards—Steve Jobs

2/26/2014

8

We manage the emergence of beneficial coherence within attractors, within boundaries—Dave Snowden

Volta

ge

Frequency 

2/26/2014

9

Source: http://www.mdofpc.com

2/26/2014

10

CYNEFIN FRAMEWORK

Complicated

C E

Complex

C E Dave Snowden

Simple

C E

good practiceemergent practice

Chaotic

probe-sense-respond sense-analyze-respond

C E

Disorder

Dave Snowden

Disorder

best practice

Cause Effect

novel practiceact-sense-respond sense-categorize-respond

C E

Source: cognitive‐edge.com 

2/26/2014

11

Source: www.thephonetown.com; with a fish m

Complex Complicatedc

a

Disorder

b

c

SimpleChaos

Source: Cynefin by David Snowdenhttp://cognitive‐edge.com/

2/26/2014

12

D i d St tdge 

Desired State

Tyranny of the expert

Know

le

PROBEPROBE--SENSESENSE--RESPONDRESPONDemergent practiceemergent practice

2/26/2014

13

2/26/2014

14

Source: Ylvis; The Fox (What Does The Fox Say?)

“It takes a thousand voices to tell a single story”- Native American proverb

Source: Competition among memes in a world with limited attention; L. Weng, A. Flammini, A. Vespignani & F. Menczer

2/26/2014

15

Prompting Question

Story Capture 

Act on Signals

Stimulate/Dampen Patterns

Self‐Signification

Discussion Visualization

Deepen InsightSource: cognitive‐edge.com 

SELF‐SIGNIFICATION IS THE KEY

Poor Performance Exciting 

This story is about:  The product talked about is: 

l b l b l

g

d bl

N/AN/A

Poor Reliability Poor Usability

Quality

Expected Desirable

Qualities 

More info: cognitive‐edge.com 

2/26/2014

16

Poor Performance

N/A

Poor Reliability Poor Usability

Quality

More info: cognitive‐edge.com 

2/26/2014

17

EXPANDING OUR TEST METHODS TO ADDRESS THE COMPLEX SPACE 

Sources: right http://www.ece.lsu.edu; left Paul Mathews 

Complicated  Complexto

C E C E

FINAL THOUGHTS

• We need to start to develop new test methods to address complex products

• Testing the rules is fine but it needs to be with real world complex stimulus 

• High satisfaction can only be understood if we listen closer to the stories our customers are telling

2/26/2014

18

ABSTRACT

It’s one week after your product’s launch, and everyone is happy. After all, for the first time in years your product development exceeded expectations Coding wasfirst time in years, your product development exceeded expectations. Coding was completed on time with very few defects. Suddenly, the report of a major usability 

and security flaw destroys the euphoria and sends everything into chaos. Unfortunately, this is not uncommon in our industry. So, how can we mitigate such things from happening? As he shares stories about the complex domain of product delivery, Ray Arell introduces a framework with associated emergent practices that enable you to better guide your product to success. He presents an overview of the Cynefin model, a description of complicated and complex systems, and 

discusses how to use it to establish an effective testing strategy. Ray describes how to identify key patterns of product usage to establish a robust defect‐prevention system that reduces product development costs. Lastly, Ray describes how to 

interview customers to identify key quality expectations, ensuring that your testing focuses on producing the highest value for your customers.

2/26/2014

19

As director of Intel Emergent Systems and Coaching, Ray Arell is a g y g, ytransformative leader in the adoption of agile, lean, and complex system methods inside Intel. Ray’s group is currently coaching a community of practice of more than ten‐thousand people who are moving to a continuous value delivery culture. Prior to this role, he spent several decades as both an engineer and engineering manager of teams focused on CPU, chipset, graphics, wireless, and software development. Ray is a popular speaker at events worldwide and coauthor of Change‐Based Test Management: Improving the Software Validation Process.