Test Status Reporting: Focus Your Message for Executives

T7 Concurrent Class

10/3/2013 11:15:00 AM

"Test Status Reporting: Focus

Your Message for Executives"

Presented by:

Stephan Obbeck

KROLL Consulting AG

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Stephan Obbeck

KROLL Consulting AG

Senior consultant and managing director of the test center at KROLL Consulting AG,

Wiesbaden, Germany, Stephan Obbeck has more than fifteen years of IT experience with

hands-on software testing and is a certified project and test manager. After starting his career

as a tester, Stephan now invents test organizations for leading telecommunication and media

companies in Europe and manages test projects. As an architect for test automation, Stephan

supports organizations in moving from manual testing to automation.

1

03. October 2013

Stephan Obbeck

KROLL Consulting AG

Wiesbaden, Germany

Test Status Reporting: Focus Your Message for Executives

Agenda

Firmenvorstellung 2KROLL Consulting AG

View this presentation as animated Prezi at

http://prezi.com/j0ufgev9cukz

2

Management - and Test Reports

• Too Many Information� Management members pretend to have not much time -

keep them thinking like this!

� Your status report is not the only one a manager has to read

• Too Technical Details

� Most management members do not understand the

technical details of a test project

� If they do not understand, they rarely ask for more

information

� Decisions are made on data which is not understood

• Contradictory Information

� Data is tricky - it might fight back if you have contradictory

information

� Irregular reporting mechanism and content during a project

(this week reporting about progress, next week about

defects)

3KROLL Consulting AG

Status Reports - Layers of Communication


• Information Transfer FACTS� Obvious Status of the project

� Transfer of Information

� Actual Situation and next steps

• Non-Information Transfer EMOTIONS� Project advertisement

� Underline Requests you might have by data

� Influence Management decisions regarding your test project

• Reduced Complexity COMPACTYour recipient does not have much time

Focus your message - bring it to the point

� key facts on one page

� statistics understandable without additional information

� Make sure the right decision is taken from the report

3

Management Summary Report


• Status of your project� Colored status

• Progress of your work� Tasks during reporting period

� Tasks for the next reporting period

• Maximum two statistics that

describe:

� Progress of the test project

� Situation of your main project risks

Detailed Report


• Status of your project� Additional statistics and features

� Overview of bugs

� Statistics for specific stakeholders

� =

4

Status Reports - Structure of Test Reports


Keep report as

short as possible

Use graphics and

charts whenever possible

Avoid technical details

needing explanation

Focus on what you really

want to say

Prepare Your Test Report: Recipients

During Test Planning –

Plan Your Reporting

• To whom do you report?� What is his / her position?

� Which information does he / she need?

� How deep is the technical understanding?

� To whom do you HAVE to report?

� To whom do you WANT to report?


5

Prepare Your Test Report: Content

During Test Planning –

Plan Your Reporting

• What do you want to report?� There are tons of data information you can report.

� Which numbers really give value to management?

� How can you influence management?

• How do you want to report?� Plan your project and keep your reporting in mind.

� Reporting the progress?

� Reporting the risks or better the tasks?


Stakeholder Analysis (WHO?) - Questions


Stakeholder are all persons /

organizations who might be affected by your project.

• This analysis helps you to find out, who might have interest /

or requests reports from your test project:� What is the position of the stakeholder?

� How is the stakeholder affected by your project?

� What is the stakeholder's opinion to your project?

� How does the stakeholder see the project?

6

Stakeholder Analysis (WHO?) - Rating


• Rate each stakeholder by� Interest

� Power

• Use a matrix to evaluate your results.

• Visualize your stakeholders and

their relation to the project.

Stakeholder Analysis (WHO?) - Matrix


7

Risk Analysis (WHAT?) – for Benefit

A risk analysis identifies areas of danger to

your project and how they can be defeated.

• Report your success. (What have you already achieved?)

• Report your areas of risk. (What can make your project fail?)

• How can you measure your risk?

• How can numbers and diagrams

illustrate your risk?


Risk Analysis (WHAT?) - Matrix

• Analyse your risks and evaluate� Likelyhood (How likely is it that this risk occurs?)

� Consequences (How big are the consequences of this risk?)

• Use a matrix to group your risks

• Visualize the most important risks for your project


8

Risk Analysis (WHAT?) - Visualization


Example

Risk: High number of defects might be found� Your team can handle 20 defects per week. If more occure, you need

additional testers.

� Report number of defects from the beginning on with a bar chart.

Plan Your Work Packages Right (HOW?)


Try to build work packages

that have the same size by either

• Testcases with equal number of steps.

• Group Testcases to bundles of same

estimated working time.

• Use parameters for level of completeness

for each working package (see later slide).

• If your work packages have a comparable size� You ease your test planning.

� You have better control on your project status.

� Your status report can use linear progress information.

9

• How to present the overall

project status?

• Some of the most common used

status methods:� “Signal Status” with red, yellow, green

� Project tendencies

� Subjective project progress

General Status Reporting


Status Reports often use a “signal status“

• Green:

Project on track

•• Yellow: Yellow:

Slight delays - project re-planning necessary

• Red:

Main project targets can not be reached within

project parameters

• Problem:� Yellow (middle) is a safe harbor.

� I have seen projects being on yellow for weeks and months.

The Yellow Comfort Zone


10

Signal Status with Even Number

BETTER:

Use a reporting system with an

even number of status fields (e.g. 4 or 6)

• Green: Project on track

•• Yellow:Yellow: Slight delays - no management attention

necessary (e.g. setup daily defect meetings)

• Orange: Significant problems - management

attention necessary (e.g. additional resources)

• Red: Main project targets can not be reached

within project parameters


Avoid Project Tendency

Often a tendency is added to the status:

Which direction the project heads to.

"Status will go up next week, when we work over the weekend“

"Status will go further down, if we do not get another tester“

• Do not report trends - manage your risks!� Every manager should know that a risk coming to reality

will have a negative impact on the project.

� Report the risk with the actions need to be taken.

� If a risk is resolved by actions, your project will be

in scope again!

� Add a risk management matrix to your report.

� Ensure management can see the risks and their actions

on a first look at the report / risk matrix.


11

Risk Management

Example of a Risk Management Matrix:

A simple way how to manage a risk (Risk #01):

• Risk: Time schedule tight. If many bugs will be found during

testing and need to be documented, time for testing might

be too short.

• Action taken: We will work next weekend.

• Result: At the weekend we can test 25 test cases and the

project will be back on track on Monday.


90% Syndrom

Joe has to write a test concept.

At the end of every day you ask him about the status.

What answer will you get as the days went by? 90%

Longer tasks often stick in an incomplete status of 80 - 95%.

There are always some remaining things to be done...


12

Level of Completeness

• Split longer tasks ( > 1 day) � Sub tasks (<= 1 day)

� Assign level of completeness

to each sub task.

• Add a constant

"time elapsed" fraction

for each reporting period

during the project:� If your project is 4 days long (=100%) and

you report every day, your time fraction is 25%

• Report elapsed time and level of completeness

• At least one of these two should have a constant progress to

manage your test report by progress status reports.


Manage your Test with Status Reports


13

Improve your Status Reports


By using different Charts

• Excel - and other tools - offer dozens of different chart types.

• Which one to choose?� Support the message you want to transfer to management.

� One Eye Catcher for your message.

� Use colors to emphasize your main message.

• Green = OK

• Red = Attention

� I prefer basic chart layouts,

do not be too creative.

Pie Chart and Bar Chart


• Pie Chart� Basic chart to show different status information

� Low expressiveness

� Just a snapshot status

• Bar Chart� Same information available as in pie chart for each bar

� History information available (PAST)

� History information to follow a trend (FUTURE)

14

Test Progress Evaluation


If your goal is to finish 2000 test cases in calendar week 32 the history in

the bar chart can help you to justice if you are on track.

Green arrow: Your project is on track.

Red arrow: Your project is not on track and you need to

take corrections (new target date)

Yellow / orange arrow: You look fine but you should be careful...

Test Progress


In general your test progress

• does not follow a straight line

• follows an S-Curve:

� Starting flat

� Increases in the middle

� Flats down at the end

15

Test Progress Analysis Methods


• This analysis method works if you have a

constant work progress for each of your

reporting periods.

• If this is not the case (different number of

testers, etc.):� Use a time elapsed progress report.

� Add an expected target line to your diagram to identify your

planned progress during the project.

Test Automation


Three reports should be created out

of your test automation

16

Management Summary

• Purpose:� Total test results of the test run

� Basis for decisions

• Adressee:� Managers

� Decision Makers

• Structure:� Not more than one or two pages

� Use graphics to evaluate the result quickly

� Use colors to illustrate important messages

� Just a passed / not passed information

� No (or very short) explaining texts for errors


Error Report

• Purpose:� List of errors found during the test run.

� Checklist for analysts to work through all

anomalies and review it.

• Addressee:� Developers, Testers and Software Analysts

� Project Manager

• Structure:� Show all results that are not OK.

� Show the test case and the object under test.

� State a text explaining the anomaly: The IS

state and the expected SHOULD result.


17

Inspection Report

• Purpose:� Complete list of all test results for each test

cases and objects under test.

� Complete overview of the test run - all test

cases have been executed.

• Addressee:� Project Manager

� QA Inspector

� Auditing Department

• Structure:� Show ALL results of the test run

(OK and not OK).

� Show all test cases and all objects under test.

� Explaining text might be used as well.

� Might be created as a data file for a test

database.


Summary - You‘re almost there…


Don’t forget

• Keep your report compact.

• Plan your reporting before kicking off

the project:� WHO? (Stakeholder Analysis)

� WHAT? (Risk Analysis)

� HOW? (Project Planning)

• Avoid reporting mechanism needing explanation.

• Use your project control mechanism in your report.

• Organize your reporting from test automation.

18

I appreciate your feedback


THANK YOU

• The presented methods are taken from various management

certification workshops and are my own experiences in

different projects.

• If you pick up one of these ideas and try it in your

organization, feel free to drop me an email and share your

experiences with me:

[email protected]

Thank You

STEPHAN OBBECK

KROLL Consulting AG

Schenkendorfstraße 5

D-65187 Wiesbaden

www.kroll-consulting.de [email protected]

Date post:	07-Nov-2014
Category:	Technology
Upload:	techwellpresentations
View:	186 times
Download:	1 times

Test Status Reporting: Focus Your Message for Executives

Technology