How to test Level-of-Service Requirements

University of Southern California

Center for Systems and Software Engineering

How to test Level-of-Service Requirements

CS 577b Software Engineering II

Supannika Koolmanojwong



Level of Service Requirements

2

NON-FUNCTIONAL

REQUIREMENTS

Quality Attributes-ilities

Quality goals

Non-behavioralrequirements

Performance

Requirements



Examples of LOS requirements• Security• Usability• Testability• Maintainability• Scalability• Backup• Compliance• Reliability• Resource constraints

(speed, memory, bandwidth)

• Extensibilities• Fault Tolerance• Interoperability• Portability• Stability• Response Time• Disaster Recovery• Escrow (ensure maintenance of SW)

• Privacy

3



Non-functional requirements

• Can be categorized into two main categories– Execution qualities - observable at run time

– Evolution qualities – embodied in the static structure of the software system

4

Security, usability

Maintainability, scalability

http://en.wikipedia.org/wiki/Non-functional_requirement



How to test your LOS requirements

• Start with how to write your LOS requirements

• Make it SMART– Specific – clear, consistent and concise– Measurable – Attainable – achievable given what they know– Realizable – can be done given the constraints– Traceable – need to understand why we need

this

5http://www.win.tue.nl/~wstomv/edu/2ip30/references/smart-requirements.pdf



Specific

• “The mission planning system shall support several planning environments for generating the mission plan”

• Specific ?

6

• What do you mean by several?• Have you defined “planning environment”

and “mission plan”?



Specific• Avoid obviously, clearly, certainly• Avoid some, several, many• Avoid etc., so on, such as• When numbers are specified, identify the units• Ensure pronouns are clearly referenced

– When module A calls B, its message history file is updated

• Use glossary

7



Measurable• “The system shall produce a plan optimized for

time.”– The only way to measure is to compare with an absolute

optimum, which may not be available.– Should have a fixed performance against a predefined set

of test cases for which the absolute optimum is known. • Think about

– Can this requirement be verified? – Can the test be conducted on one site? – Can this requirement be tested in isolation?

8



Attainable• Possible physically for the system to

exhibit that requirements • “The system shall be 100% reliable and 100% available”• “The system shall have a minimum response to a query of 1

second irrespective of system load”– Must be very expensive to meet these

requirements or will never be accepted • Think

– Is there a theoretical solution to the problem?– Done before? Any feasibility study? – Any physical constraints ? Memory size

9



Realizable

• Achievable given what is known about the constraints

• 99% reliability– But limited budget, short schedule

• Think– Sufficient resources / time / budget?– Afford to manage them? – Use status, “desirable” at the beginning, after

prototyping / has evidence, then update the status

10



Traceable

• Traceable to/from concepts / spec / design / implementation / test

• Need to know why we need this; business justification

• Verify that it is implemented

11



Testing LOS requirements

12http://scaledagileframework.com/nonfunctional-requirements/



Check list for LOS requirements• Security

• Login requirements - access levels, CRUD levels • Password requirements - length, special characters, expiry, recycling policies • Inactivity timeouts – durations, actions

• Audit • Audited elements – what business elements will be audited? • Audited fields – which data fields will be audited? • Audit file characteristics - before image, after image, user and time stamp, etc

• Performance • Response times - application loading, screen open and refresh times, etc • Processing times – functions, calculations, imports, exports • Query and Reporting times – initial loads and subsequent loads

13http://leadinganswers.typepad.com/leading_answers/2009/03/nonfunctional-requirements-minimal-checklist.html



Check list for LOS requirements• Capacity

• Throughput – how many transactions per hour does the system need to be able to handle? • Storage – how much data does the system need to be able to store? • Year-on-year growth requirements

• Availability • Hours of operation – when is it available? Consider weekends, holidays, maintenance times, etc • Locations of operation – where should it be available from, what are the connection requirements?

• Reliability • Mean Time Between Failures – What is the acceptable threshold for down-time? e.g. one a year, 4,000 hours • Mean Time To Recovery – if broken, how much time is available to get the system back up again?




• Integrity • Fault trapping (I/O) – how to handle electronic interface failures, etc • Bad data trapping - data imports, flag-and-continue or stop the import policies, etc • Data integrity – referential integrity in database tables and interfaces • Image compression and decompression standards

• Recovery • Recovery process – how do recoveries work, what is the process? • Recovery time scales – how quickly should a recovery take to perform? • Backup frequencies – how often is the transaction data, set-up data, and system (code) backed-up? • Backup generations - what are the requirements for restoring to previous instance(s)?

• Compatibility • Compatibility with shared applications – What other systems does it need to talk to? • Compatibility with 3rd party applications – What other systems does it have to live with amicably? • Compatibility on different operating systems – What does it have to be able to run on? • Compatibility on different platforms – What are the hardware platforms it needs to work on?




Check list for LOS requirements• Maintainability

• Conformance to architecture standards – What are the standards it needs to conform to or have exclusions from? • Conformance to design standards – What design standards must be adhered to or exclusions created? • Conformance to coding standards – What coding standards must be adhered to or exclusions created?

• Usability • Look and feel standards - screen element density, layout and flow, colours, UI metaphors, keyboard shortcuts • Internationalization / localization requirements – languages, spellings, keyboards, paper sizes, etc

• Documentation • Required documentation items and audiences for each item

16



17http://www.ibm.com/developerworks/webservices/library/ws-soa-nonfunctional/index.html



Compatibility Testing

• Hardware platform (IBM 360, HP 9000)• Peripherals (printer, DVD)• OS (Linux, Windows)• DB (Oracle, SQL server, MySQL)• Browser (Chrome, Firefox)• Carrier (Verizon, Sprint)• Hardware (different phones)

18



Compliance / Conformance Testing

• Conform to standards• Often performed by external organization• Local / global standards

19



Soak / Endurance / Load / Stress Testing

• Test beyond the maximum ratings for a longer period of time

• Test for robustness, availability, error handling, memory leaks, denial of service, response time, throughput

• Spike test – suddenly increase the number of load generated

20



Recovery Testing

• How well an application is able to recover from crashes, hardware failures

• Forced failure of the software – suddenly restart the computer, unplug the

network cable

21



Security Testing• 6 basic security concepts that need to be covered

– Confidentiality• Protecting the disclosure of information

– Integrity• Correctness, as intended

– Authentication• Confirming the identify, correct label

– Availability• Ready when expected

– Authorization• Access control, right service to right person

– Non-repudiation• Ensure that the transferred mesg has been sent and

received by the parties claiming to have sent and received. Can not deny that the party was the one who sent.

22http://en.wikipedia.org/wiki/Security_testing



Monkey Test

• Unit test with no specific test • Random strings / actions

23



Usability Testing• Measure ease of use• Goals

– Learnability – how easy to accomplish a task first time– Efficiency – how much time, how many steps– Memorability – remember afterwards– Errors – how many mistakes– Satisfaction – confident, stressed ? Recommend to a

friend ? • Task first, design second

24



How to improve usability

• Use representative users• Ask users to perform representative tasks• Observe and listen

• Testing 5 users is typically enough

25



Usability testing for the website• What your visitors think the first time they see your site.• Whether your visitors "get" what your site is about.• What visitors think about your site's look and feel.• Whether your visitors can read the content easily and understand

what they see.• Whether your visitors can perform your set "tasks" easily, and if

not, why not.• Where visitors get "stuck"... and click away.

• What visitors think about the product, pricing and offer.

26http://www.usabilitytesting.tv/articles/9-what-is-usability-testing



More usability checklists for website

• http://www.usereffect.com/topic/25-point-website-usability-checklist

• http://www.pantos.org/atw/35317.html

27

http://www.usereffect.com/topic/25-point-website-usability-checklist



http://www.pantos.org/atw/35317.html

http://www.pantos.org/atw/35317.html



Other Methods• Cognitive Walkthrough

– Evaluating the user interaction of a prototype / final product.

– Core-Capability Drivethrough• Benchmarking

– Create/ Use standardized test materials for a specific type of design.

– E.g. time to do core task, time to fix errors, time to learn applications

28http://en.wikipedia.org/wiki/Usability_requirements



Persona• Need to know exactly who your customers are• Help product developers and product designers

understand how to optimize the product around the ways the user will want to use the product

• As detail as possible in all aspects that are related to your system

• Generally 10 persona

29



User Stories vs Persona

• “As a mom, I want to take and share videos of the kids so that I can share important moments with grandparents, aunts, uncles, and friends”

• User Stories – Left brain, focus on functionality

• Persona – Right brain, focus on outcomes

30http://www.agilemarketing.net/user-stories-agile-marketing-part-2/



User Stories vs Persona Role Description

Evaluation LeadLeads the evaluation; does all the initial research; works with the group to make a recommendation to the decision maker

Decision Maker Makes the final decision

Financial decision maker

Owns the budget and makes the final decision on the financial terms.May be the same person as the decision maker.

SQL admin Responsible for the administration of the SQL database.

Developer Develops applications that use the SQL database

User The eventual end user of the application using the SQL database

31http://www.agilemarketing.net/agile-marketing-user-stories/



32http://www.agilemarketing.net/agile-marketing-user-stories/



33http://becubed.me/2007/06/08/download-an-example-persona-used-in-the-design-of-a-web-application/http://blog.ocad.ca/wordpress/gdes1b26-fw2010-19/2011/02/18/exercise-eight-ocad-student-personas/

http://becubed.me/2007/06/08/download-an-example-persona-used-in-the-design-of-a-web-application/

http://becubed.me/2007/06/08/download-an-example-persona-used-in-the-design-of-a-web-application/



34



35http://www.dylanux.com/case-study-1.shtml#.URVOcaV9Jbo



1. Finding the users2. Building a hypothesis3. Verification4. Finding Patterns5. Constructing personas

– Body– Psyche– Background– Emotions and attitudes, – Personal traits

6. Defining Situations7. Validation and Buy-in8. Dissemination of Knowledge9. Creating Scenarios10. Ongoing Development

36http://www.hceye.org/HCInsight-Nielsen.htm



Workshop• Identify possible roles & their description• Pick one role and define the persona

– Picture– Basic Demographic

• Age, occupation, hometown, marital status– Attributes– Description– User Scenario– Goals & Aspirations– One wild card (anything that would be good for your

project, such as brands, device usage)• Present in front of the class• For off-campus student, submit as in-class quiz

37

Date post:	25-Feb-2016
Category:	Documents
Upload:	ross
View:	37 times
Download:	0 times

How to test Level-of-Service Requirements

Documents