QQRExperience 2015SQTM Talk For Presentation

Experience with Quantitative Quality Requirements (QQRS)

David Mack EndresTechnical Lead, Bending Ray LLC

[email protected]

www.qualitymanagementconference.com1

2www.qualitymanagementconference.com

• Independent consultant with extensive experience in large systems:– Requirements– Architecture– Design & Implementation

• Previously software architect and designer in oilfield services, telecomm and office systems.

• Reviewer SWEBOK• Certified CSDP and CSM

Roadmap• Why quantify product quality?• Quality workshops• Quantified Quality Requirements • Test cases and measurements


Quantified Quality

Quality != 0 defectsQuality == Stakeholder Value


Why Quantify?Recent Project charter: [confidential material elided]

“…targeted at effecting a step-change improvement in the usability and efficiency of […] workflows for […] clients. This will be achieved through significant streamlining of our […] model building workflows, refactoring of our structural modeling subsystem […] for performance and scalability, and through the introduction of new software capabilities for QC and workflow automation.”

• Traditional testing only captures the last objective: new software capabilities.

• Stakeholder value depends on how well we have done the other stuff, the “ilities”

• But how do we measure that stuff? 6www.qualitymanagementconference.com

Targets & Measurements• Performance: which is measurable?

– “I want it darn fast”. “What is darn fast?” “When the user runs it, I want them to say ‘Darn, that’s fast’”, or

– “Operation x must be 10x faster in the new system than the predecessor system on the same data and hardware”

• Scalability– “Handle big models”, or– “Must be able to model 1000 survey blocks at

seismic resolution on the standard geophysical modelling workstation” 7www.qualitymanagementconference.com

Usability Targets & Measurements

• Usability– “Improve usability”– “Reduce wall clock time for the QC workflow

by 75% compared to benchmark run on the same data set in previous version.”

• Efficiency– “Improve efficiency”– “Reduce the number of steps for a new

project plan by 90% when a previous project plan can be applied”.


Quality Requirement Flow

•Business drivers•Stakeholder input•Brainstorming

Project Goals

•Measurable quality•Numeric targetsQQRs

•Usability tests• Instrumented software

Test Cases Repeat…

Usability and other

quality results


Quality Attribute Workshops Test Case Definition Implement and Test

QAWs“Quality Attribute Workshops”


Quality Attribute Workshops• We simplify the Software Engineering

Institute (CMU SEI) process:• Review business drivers• Brainstorm qualities• Build a “utility tree” of desired qualities. • Prioritize

• Requires a good facilitator: Stakeholders don’t care about measurable quality until their [unmeasured] quality goals are not achieved.


QAW Experience Report• Major software and business process

project, “C office” visibility• 5 QAWs from October 2013 – Jan 2014

– Each workshop covered one workflow– Responsible subject matter experts (SMEs)

and architects participated– Started with generic utility tree:

– SMEs focus on external qualities– Architects expand the view to internal qualities


Eliciting Quality Requirements• Drill into vague quality statements

– Good Performance: “What do you mean by performance?”

– Scalablity: “Can you characterize a ‘large’ problem?”

– Usability: “Who is the target for ease of use?”• If stuck, ask some why or what if

questions– Why is the software to be easy to learn?– What if there is a security breach and client

data is stolen? 13www.qualitymanagementconference.com

QAW Output• Detailed Utility Tree example

– Project load/save – Goal: < 30 s – Current: > 2 h, worst case – Model load interactive

» Goal: < 2s » Current: > 30 m, worst case

– Model save, dirty model » Goal: < 2s» Acknowledge that there will be a tradeoff with

computation time to create/change model – Model save, un-modified model

» Instantaneous, e.g. don't write it out again


QQRs“Quantified Quality Requirements”


Anatomy of a QQR

• Descriptive information– Name, identifier, gist, rationale and additional

description as needed• Scale: Identified dimension of goodness

stakeholders wish to control• Meter: How to measure performance

along that dimension• Goal: Defines the target value for this

release.


Improved QQRs

• Quantify tracking data– Current and past measurements: industry trends,

record value, competitor quality, etc.• Quantify goal ranges

– Survival interval (catastrophic failure outside this range)

– Failure level (pain or loss) within survival interval– Wish, stretch, other goal measures– Future target values (quality roadmap)


Example: User experience


40 30 20 10

Mean hours to complete work flow X

Mean # mouse clicks to complete workflow X

?< << > >?15 5

100 000

10 000

1 000

<<

<?<

90 000

3 000

600

Mean distance mouse moves to

complete workflow X

< >>

Testing & Tracking Quality


Write Test Plans for the QQRs• Be specific

– test machine capacity, load– test data set– detailed steps (must be repeatable!)

• Record benchmark– previous versions of the system

• Instrument the software for important properties

• Track the trends21www.qualitymanagementconference.com

Data Editing Example: Median filter

• Median filter test case– Load project: <specific test project>– Open Property Editing process and setup the GUI

• Click “Filtering” tab ,select option Median sampling method , filtering length selection -> Depth variant length, with Reference surface named “wbxxx”, Velocity option.

• Press [Apply]– Once the process has finished exit WITHOUT saving changes

to the project.– Navigate to <specific URL on intranet> and open the file named:

“ApplicationUsage”. • Recording of time of when the [Apply] was pressed and when the process

finished is recorded in that file (hint: Search by machine name, starting at the bottom for last entries).


Track and report results

Reported Results (2013.1)

2014.1 Results 2014.4 Results

3.1 Filtering operation: Median filter

00:41:11 00:10:01 00:03:48


Usability Example• Usability targets were quite vague: reduce

number of steps

• But a test case was defined and instrumented• Usability Test Case: Model QC

– Synchronize all intersections (model and seismic)– Quick Slice display– Camera linking– Cursor tracking


Model QC Usability Test Script0 Start recording with productivity software1 Choose 'Interpretation 2 Imaging' Perspective

2Create structural framework using the 3 surfaces and the following parameters:Boundary Definition: Z extents: 0 to -12000; Coarsening of 2Horizon Modeling: use horizon types: water bottom = erosional; salt top = geobody top; salt base = geobody basethe resulting structural model should have 3 zones

3populate with the following: water = constant (1500m/s) sediment = use zone "sediment " from model "salt_body_ISO_2013model" salt = constant (4500m/s)

4 insert renderers and display inline in 3D window5 display annotation on inline6 Bring up Tool Palette Model QC tools

7Use Property Reporter to QC velocity values on inline, making sure values are as expected

Part 21 Display inline from the seismic cube in separate 3D window2 Tile windows vertically3 camera link and cursor track for both windows4 Step/drag through inlines from seismic volume5 Sychronize all slices6 Step/drag through inlines from model7 Bring up Quick Slice View tools on Tool Palette8 Display inline, xline and Z slice from both seismic volume and model9 Display xline only from both the seismic volume and model


Instrument and test usability• Used Morae(see ref) and internal

performance tools to instrument and measure time, mouse clicks and mouse movement

• Specific measurements

Tracking


Workflows

Time Mouse Clicks Mouse Distance Improved Quality

Reduction

2014.1 2014.4 2014.1 2014.4 2014.1 2014.4 Time Clicks DistanceModel QC 0:04:12 0:02:20 101 42 60509 30415 Yes 44% 58% 50%

Usability Results


Workflows

Time Mouse Clicks Mouse Distance Improved Quality

Reduction

2014.1 2014.4 2014.1 2014.4 2014.1 2014.4 Time Clicks DistanceQC 0:12:34 0:07:11 203 135 141219 120106 0.43 0.33 0.15 Model Building 0:15:13 0:14:02 355 211 228699 177907 0.08 0.41 0.22 Model Manipulation 0:42:20 0:30:41 128 69 86225 60520 0.28 0.46 0.30 Property editing 1:45:27 0:57:00 280 189 119039 75295 0.46 0.33 0.37 Scenario testing 0:16:11 0:14:30 215 226 124633 111215 0.10 (0.05) 0.11

2014.1 Overall Project Run 88:57:58 0.49 2014.4 Overall Project Run 45:27:18

QC

Model Building

Model ManipulationProperty editing

Scenario testing

0:00:00

1:12:00

2:24:00

Time Performance2014.1 2014.4

QC

Model Building


Scenario testing

0

200

400

Mouse Clicks

QC

Model Building


Scenario testing

0

200000

400000

Mouse Distance (pixels)

Summary• Quality is defined by stakeholder value• Use workshops to elicit quality requirements• Define measurable quality targets• Track and report quality attribute

measurements


Contact and References• Contact me: [email protected], LinkedIn:

BendingRayLLC• Tom Gilb: “Competitive Engineering” 2005

• Scalar requirements (QQRs)

• Carnegie-Mellon University, Software Engineering Institute (www.sei.cmu.edu)– M. Barbacci, et al. Quality Attribute Workshops (3rd ed.), 2003

• Morae User Experience and Market Research– http://www.techsmith.com/morae.html

• Douglas Hubbard: “How to Measure Anything”, 3rd ed. 2014 29www.qualitymanagementconference.com

mailto:[email protected]

http://www.techsmith.com/morae.html

http://www.techsmith.com/morae.html

Date post:	22-Mar-2017
Category:	Documents
Upload:	david-mack-endres
View:	13 times
Download:	0 times

QQRExperience 2015SQTM Talk For Presentation

Documents