6 Secrets of Exploratory Testing in

Embedded Systems

Shmuel Gershon

Tel Aviv, July 2013

Who are you people?Testers

Explorers

Programmers

ManagersAficionados

In the wrong conference?

Curious

All of the others in one

Web

Embbeded

2

What am I doing here

3

Is this the tester you hire?

• PERSONAL FREEDOM

• RESPONSIBILITY of the individual tester

• OPTIMIZE THE QUALITY OF THE WORK

• LEARNING + DESIGN + EXECUTION + RESULT

INTERPRETATION...

• THROUGHOUT THE PROJECT

4

Remember our Super Tester?"A style of software testing that emphasizes

the PERSONAL FREEDOM

and RESPONSIBILITY of the individual tester

to OPTIMIZE THE QUALITY OF THE WORK by

treating test-related

LEARNING, test DESIGN, test EXECUTION, and

test RESULT INTERPRETATION as mutually

supportive activities that run in parallel

THROUGHOUT THE PROJECT" (Cem Kaner)

5

Exploratory Testing by Cem:"A style of software testing that emphasizes

the PERSONAL FREEDOM

and RESPONSIBILITY of the individual tester

to OPTIMIZE THE QUALITY OF THE WORK by

treating test-related

LEARNING, test DESIGN, test EXECUTION, and

test RESULT INTERPRETATION as mutually

supportive activities that run in parallel

THROUGHOUT THE PROJECT" (Cem Kaner)

6

Embedded Systems:• An embedded system is a computer system

designed for specific control functions within

a larger system, often with real-time

computing constraints. It is embedded as

part of a complete device often including

hardware and mechanical parts.== It is “hidden”, “untouchable”,

inside other systems!

7

Test This:

8

Now Test This:

9

Now Test This One:

10

And Finally Test This One:

11

Common Direction

Automated Tests

SDK

API

Automation Suite

Handless Operation

Seems as if reality forces you into it…

12

Why Breaking the Pattern?

• Stiffness

• Long Feedback Loop

• Borderless infrastructure

• Premature learning

• ...

• ...

• ...

13

Exploring the UntouchableRovers on celestial bodies (suchas the Mars Rovers) cannot be

easily remote-controlled since the

radio signal speed is slow (asignal from Mars to Earth takesbetween 3 and 21 mins). A rovers

is capable of operating

autonomously with little

assistance from ground control

for data acquisition, but requires

human input for identifying

promising targets, and

determining how to position

itself.14

Exploring the Hidden

15

16

How to Break the Pattern?

17

6 Principles

Principles

Probes

Verbs

Cut the Middleman

Judge Not

Stay in Control

• Simple Autonomy

Traduttore, Tradittore

• Caveat Testor

18

1) Probes

Have a system that gives you information, all

the information, unfiltered, from all the system.

You can filter it elsewhere.

- Signals

- Logs / States

- Results

- Timers / Watchdogs

Probes

19

2) Verbs

Name your actions: Put your hands inside the system.

- If it can do something, you can do it too.

- Not only direct inputs – controls the environment

- Sometimes the way to control it is to know how the system deals with environments and inputs- Your simulation + the system rules = state

Verbs

20

Cut the Middleman

3) Cut the Middleman

The system should allow tests to flow, without

stopping to re-automate.

- Short action and reaction cycles increase

engagement and cognition

21

Judge Not

4) Be cautious in judgment

Computers help get results, testers interpret

results.

- You can’t possibly teach a computer how an

unknown bug looks like

22

5) Stay in Control

If you are exploring, you need the steering

wheel.

- Cruise Control may be for known areas

- Automatic Pilot may be for getting started

Stay in Control

• Simple Autonomy

23

Traduttore, Tradittore

• Caveat Testor 6) Traduttore, Tradittore

We are building an abstraction, and all

abstractions are leaky.

“All models are false but some models are

useful.”George E. P. Box

24

The only summary slide you need to remember:

“It’s all about building an automated system for manual tests”

25

(Free!) Bonus Benefits!

• Find Limits

• When Automated Tests are needed they

can use the same Manual infrastructure

• Debug, Troubleshooting, Bug Isolation

26

More Bonus!Automation Layers

Test Structure

System Structure

Test Cases Automation

27

Credits, Questions, Answers

• Michael Stahl

• Gary Draishpits

• Cem Kaner

• Michael Bolton

• Doron Blazer

• Doron Maliach

• Yonatan Tidhar

• Bat-Sheva Honigstein

• Intel Sensors Teamo My Manager too!

• QA & Test 28