20140925 fistb keynote

Beyond Agile Testing to Lean Development

James O. Coplien Gertrud & Cope

BEYOND AGILE TESTING TO LEAN DEVELOPMENT - Jim Coplien • Testing is a business function that should be focused on risk • Lean defines testing as waste — why do it? • Automated tests should reflect requirements, but testers shouldn't test requirements • Testers are the team's conscience 50 Minutes

1 20140925FiSTBKeynote.key - 27 Sep 2014

2-1 20140925FiSTBKeynote.key - 27 Sep 2014








http://www.ambysoft.com/essays/agileTesting.html


What we Wish Agile Testing Were

(Big thanks to Michael Bolton and James Bach)


Agile Testing means “Test-first”

… and almost always at the unit level

Specification by example. “Context-driven testing is sometimes considered a ‘flavor' of agile software development.” — http://searchsoftwarequality.techtarget.com/definition/context-driven-testing




Specificatio

n by Examp

le Contextual testingAutomationJUn

it

TDD

"At the basis of the test automation pyramid is unit testing. Unit testing

should be the foundation of a solid test automation strategy and as such represents the largest part of the

pyramid.” — Mike Cohn





Specificatio

n by Examp

le Contextual testingAutomationJUn

it

TDD

Processes and tools over

Individuals and Interactions



Part I: (Unit) Testing is Muda


Unit Testing is One of the Least Efficient Ways we Know to Remove DefectsRemoval Step Efficiency

Formal Inspections 93%Static Analysis 55%

System Test 36%Functional Test 35%

Component Test 32%

Unit Test 32%

Desk Check 27%Prototyping 20%

Acceptance Tests 17%

Regression Tests 14%

(“Software Defect Origins and Removal Methods,” Draft 5, 28 December 2012, http://www.ifpug.org/Documents/Jones-SoftwareDefectOriginsAndRemovalMethodsDraft5.pdf


Why doesn’t unit testing work?

We can’t test things of business value

Not knowing what to test, we test many things that don’t need testing

Automated crap is still crap


Unit tests can’t test things of business value

In an OO world you can’t tell, from the code, whether a method will ever be executed — or in what sequence

There is rarely any explicit link from unit tests to anything of business value

What’s amazing is that we continue to test… Paths that will never be exercised Methods that are never used Ungrounded method invocation sequences


Not knowing what to test, we test things that don’t

need testing

We test scenarios that will never be exercised in the real world

We test methods that will never be invoked


Automated crap is still crap

Automation for its sake is waste

Design requires intelligence: intelligence cannot be automated

Autonomation instead of automation

You still need exploratory and experience-based testing

Get manual procedures working first and then automate the mundane parts

Design for Test like they do in hardware


Automated testing is not testing

http://qz.com/196200/toyota-is-becoming-more-efficient-by-replacing-robots-with-humans/ Max Niesen, April 7, 2014 James Bach: “Testing Requires Inference.” “Exploratory inference may sound like a strange idea. It means one idea leading to another in ways you can’t predict in advance” [and, therefore, can’t script].


The software is trained to pass the tests

"In software testing, it gets progressively harder and harder to find defects. If we run the same test or use the same technique over and over, the effort becomes less effective each time we do it, because we find only the easy-to-find bugs. The software is trained to pass the test. …” — Ross Smith, Microsoft


That kind of test thinking, leads to a bottom-up

procedural architecture

Citation: Siniaalto and Abrahamsson, Does Test-Driven Development Improve the Program Code? Alarming results from a Comparative Case Study. Proceedings of Cee-Set 2007, 10 - 12 October, 2007, Poznan, Poland.



Part II: Why do we unit test?

Because we can

It doesn’t require teamwork

We don’t have to talk with the business

We get to interact with tools instead of people

Our decisions are emotional, not rational

Last point: Most tests are irrational: i.e., you can never presume software correctness from the tests. But we do feel better by having checked, so we test anyhow.


Nokia Test Score 2: Testing

No dedicated QA - 0

Unit tested - 1

Feature tested - 5

Feature tested as soon as completed - 7

Software passes acceptance testing - 8

Software is deployed - 10


Part III: You Will Have Bugs!

Scenarios we test in the lab

!

!

! !

!

!

Scenarios exhibiting

faults

!

Scenarios invoked in the field


Testing waste


!

!

! !

!

!


faults

!


Type 2: Weinberg’s Law of Decomposition Type 4: Keeping tests that never fail


Testing waste


!

!

! !

!

!


faults

!


Type 1 Muda

Type 1: Defects



Testing waste


!

!

! !

!

!


faults

!


Type 2 Muda

Type 1 Muda

Type 1: Defects

Type 2: Stuff customers

don’t ask for



Testing waste


!

!

! !

!

!


faults

!


Type 2 Muda

Type 1 Muda

Type 4 Muda

Type 1: Defects


don’t ask for

Type 4: Overprocessing



Testing waste

!

!

!


faults

!


Type 1 Muda

Type 4 Muda

Type 1: Defects


don’t ask for

Type 4: OverprocessingType

1 Muda (field)



Testing waste

!

!

!


faults

!


Type 1 Muda

Type 4 Muda

Type 1: Defects


don’t ask for

Type 4: OverprocessingType

1 Muda (field)

Removing tests leaves field faults

undetected!



Reduction of WasteType 1: Better testing (e.g., going beyond contextual testing to exploratory & experience-based testing) (and better yet, better analysis and design, where the real payoffs are)

Type 2: Just get rid of the tests

Type 4: Remove redundant and tautalogical tests

Type 1 in the field: Ship your tests!


Don’t take the tests out when you ship!Remember, you will have bugs in the field!

Deployment is your largest test driver

Ship your tests as part of the code

IN case of a failure, it saves you from Type 9 muda (unsafe execution)

Be customer-focused


Defensive Programming

… is largely the incorporation of tests in the code

Long known for high-reliability systems

The value of asserting the obvious

Design by Contract


Assertions make code readable!

- (void) prepareForSegue: (UIStoryboardSegue*) segue sender: (id) sender { assert(segue != nil); if ([[segue identifier] isEqualToString: @"SegueAvatarPickerToPickOneLoginPlayer"]) { assert((UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPhone)); // The field parentViewController_ (instance variable) is // maybe null if we are in iPhone mode and doing a real // segue, instead of a popover as we do on the iPad. So // get the right object from the segue record. KnowsyPlayerPickerCommonViewController *parentViewController = [segue destinationViewController]; [parentViewController setKnowsyData: knowsyData_]; } else { // iPad - no special processing - should be no popover assert(parentViewController_ == nil); } }


Agile testing isn’t about tools!

It’s about the test effort being customer-focused

It’s about partners instead of customers

It’s about “stopping the line” when discovering a defect — even in deployment

It’s about quality at every step

These are all Lean ideas


Part IV: Beyond agile testing, Lean testingWhen a test finds a bug, it’s about fixing the process — not the bug

The long-term payoff is higher!

No bug-tracking systems: we fix problems NOW.

The tester should be the team’s conscience

Conscience… Curmudgeonly and like Emperor’s New Clothes — not in the sense of standing in the way. That’s business’s job.


Other Lean-isms

Deming: Quality is inversely proportional to the number of testers

GM kept around its cars for 1 - 6 weeks for test and repair: Toyota drove them off the end of the assembly line onto the car carriers.

No unit - subsystem - system - integration - acceptance test leveling.


Conclusion

Be humble about what tests can accomplish

It’s not about testing, but value

Design tests to assess properties of business value

Go beyond unit testing to defensive programming and design-by-contract

As agile testers, focus on process first and product second





Date post:	27-Nov-2014
Category:	Software
Upload:	james-coplien
View:	630 times
Download:	1 times