Date post: | 18-Nov-2014 |
Category: |
Documents |
Upload: | softwarecentral |
View: | 530 times |
Download: | 0 times |
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
11
CES 592 Telecommunications CES 592 Telecommunications System Product VerificationSystem Product Verification
Sonoma State UniversitySonoma State University Fall 2004Fall 2004
Class Lecture 2: Class Lecture 2:
High-Availability Architectures,High-Availability Architectures,
Testing Constraints, Network Testing Constraints, Network Failures, and Test StrategiesFailures, and Test Strategies
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
22
Telecom System ArchitectureTelecom System Architecture History – Some telecom network History – Some telecom network
failuresfailures High Availability definedHigh Availability defined HW / SW ArchitecturesHW / SW Architectures Unique problems/constraints/goalsUnique problems/constraints/goals
– ISO 9001 & TL 9000ISO 9001 & TL 9000– Other constraintsOther constraints
Testing for High-AvailabilityTesting for High-Availability
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
33
Communications Network FailuresCommunications Network Failures October 27, 1980 ARPAnet collapse – October 27, 1980 ARPAnet collapse –
self-propagating error required all self-propagating error required all systems to be restarted. 4 hr outagesystems to be restarted. 4 hr outage
January 15, 1990 AT&T nationwide January 15, 1990 AT&T nationwide outage – error in C code caused self-outage – error in C code caused self-propagating 9 hour outage, 5 million propagating 9 hour outage, 5 million blocked callsblocked calls
June 27, 1991, 8 million line outage – June 27, 1991, 8 million line outage – self-propagating error in untested code self-propagating error in untested code patchpatch
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
44
Communications Network FailuresCommunications Network Failures Famous Fiber Cuts (“Backhoe Fade”)Famous Fiber Cuts (“Backhoe Fade”)
– 11/19/1990, 150,000 phone lines, 11/19/1990, 150,000 phone lines, outage lasted several hoursoutage lasted several hours
– 12/4/1991, 100,000 phone lines, several 12/4/1991, 100,000 phone lines, several hours, interrupted FAA flight control and hours, interrupted FAA flight control and NY Mercantile ExchangeNY Mercantile Exchange
From From Computer Related RisksComputer Related Risks, , by Peter G. Neumannby Peter G. Neumann
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
55
Telecom System ArchitectureTelecom System Architecture
History – Some telecom network History – Some telecom network failuresfailures
High Availability definedHigh Availability defined HW / SW ArchitecturesHW / SW Architectures Unique problems/constraints/goalsUnique problems/constraints/goals
– ISO 9001 & TL 9000ISO 9001 & TL 9000– Other constraintsOther constraints
Testing for High-AvailabilityTesting for High-Availability
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
66
High Availability DefinedHigh Availability Defined 99% Uptime = 3 days downtime / year99% Uptime = 3 days downtime / year 99.9% Uptime = 8 hours downtime / year99.9% Uptime = 8 hours downtime / year 99.95% = 4 hours, 23 minutes / year99.95% = 4 hours, 23 minutes / year 99.99% = 53 minutes / year99.99% = 53 minutes / year 99.999% = 5 minutes / year (‘five nines’)99.999% = 5 minutes / year (‘five nines’) 99.9999% = 30 seconds / year99.9999% = 30 seconds / year
Reference Telcordia GR-1110, TR-332Reference Telcordia GR-1110, TR-332
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
77
High Availability DefinedHigh Availability Defined
Expectation for reliable communications, Expectation for reliable communications, even in the event of major disasters – even in the event of major disasters – when communications are needed mostwhen communications are needed most
Service Level Agreements (SLA)Service Level Agreements (SLA) Automatic and instantaneous recovery Automatic and instantaneous recovery
from internal & external faultsfrom internal & external faults Very high Mean Time Between Failure Very high Mean Time Between Failure
(MTBF) 100,000+ hours(MTBF) 100,000+ hours
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
88
Telecom System ArchitectureTelecom System Architecture
History – Some telecom network History – Some telecom network failuresfailures
High Availability definedHigh Availability defined HW / SW ArchitecturesHW / SW Architectures Unique problems/constraints/goalsUnique problems/constraints/goals
– ISO 9001 & TL 9000ISO 9001 & TL 9000– Other constraintsOther constraints
Testing for High-AvailabilityTesting for High-Availability
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
99
High Availability Telecom SystemHigh Availability Telecom System HW & SW Architectures HW & SW Architectures
Designed for High AvailabilityDesigned for High Availability– Single fault tolerantSingle fault tolerant– Low probability of double faultLow probability of double fault
Passive backplane (higher reliability – MTBF 1,000,000+ hrs)Passive backplane (higher reliability – MTBF 1,000,000+ hrs) Modular Design - Fault in one card won’t impact other cardsModular Design - Fault in one card won’t impact other cards Separation of control plane and data planeSeparation of control plane and data plane Card Redundancy - “Hot Redundancy” (maintain sync)Card Redundancy - “Hot Redundancy” (maintain sync) On-line HW Replacement – “Hot Swappable”On-line HW Replacement – “Hot Swappable” In-Service errorless SW, FW, FPGA Upgrades/DowngradesIn-Service errorless SW, FW, FPGA Upgrades/Downgrades Alarm logs, audit logs, and provisioned settings preserved Alarm logs, audit logs, and provisioned settings preserved Microprocessor Watchdog Timers / HeartbeatMicroprocessor Watchdog Timers / Heartbeat User errors minimized (major source of outages)User errors minimized (major source of outages) Graceful shutdown / restorationGraceful shutdown / restoration ““Hardened” hardware for operation at extreme power, Hardened” hardware for operation at extreme power,
temperature, humidity, corrosion, ESD, & vibration levelstemperature, humidity, corrosion, ESD, & vibration levels Network designed for Link and Node protectionNetwork designed for Link and Node protection
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
1010
Telecom System ArchitectureTelecom System Architecture
History – Some telecom network History – Some telecom network failuresfailures
High Availability definedHigh Availability defined HW / SW ArchitecturesHW / SW Architectures Unique Unique
problems/constraints/goalsproblems/constraints/goals– ISO 9001 & TL 9000ISO 9001 & TL 9000– Other ConstraintsOther Constraints
Testing for High-AvailabilityTesting for High-Availability
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
1111
Unique Problems, Constraints, Goals: Unique Problems, Constraints, Goals: ISO 9001 & TL 9000ISO 9001 & TL 9000
ISO 9001 Covers Quality Assurance in ISO 9001 Covers Quality Assurance in Design/Development, Production, Installation and Design/Development, Production, Installation and ServicingServicing
Certification process:Certification process:– Document quality processes of your organizationDocument quality processes of your organization– Audit by RegistrarAudit by Registrar– Certification and follow-up inspectionsCertification and follow-up inspections
““With ISO 9000 you can still have terrible processes and With ISO 9000 you can still have terrible processes and products. You can certify a manufacturer that makes life products. You can certify a manufacturer that makes life jackets from concrete, as long as those jackets are made jackets from concrete, as long as those jackets are made according to the documented procedures” according to the documented procedures” Richard Buetow, Richard Buetow, Director of Corporate Quality, MotorolaDirector of Corporate Quality, Motorola
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
1212
Unique Problems, Constraints, Goals: Unique Problems, Constraints, Goals: ISO 9001 & TL 9000ISO 9001 & TL 9000
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
1313
Unique Problems, Constraints, Unique Problems, Constraints, Goals: ISO 9001 & TL 9000Goals: ISO 9001 & TL 9000
TL 9000 is a quality management TL 9000 is a quality management process for design, development, process for design, development, manufacturing, delivery, installation, manufacturing, delivery, installation, and maintenance of and maintenance of telecommunications hardware and telecommunications hardware and software.software.
““The organization shall establish and maintain The organization shall establish and maintain a method to trace documented a method to trace documented requirements through design and test.”requirements through design and test.”
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
1414
Unique problems, constraints, goals: Unique problems, constraints, goals: Other Constraints Other Constraints
Can not test on a live operational Can not test on a live operational network, therefore the testing network, therefore the testing configuration must be representativeconfiguration must be representative
Must comply to numerous national Must comply to numerous national and international standards from and international standards from multiple standards bodiesmultiple standards bodies
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
1515
Telecom System ArchitectureTelecom System Architecture
History – Some telecom network History – Some telecom network failuresfailures
High Availability definedHigh Availability defined HW / SW ArchitecturesHW / SW Architectures Unique problems/constraints/goalsUnique problems/constraints/goals
– ISO 9001 & TL 9000ISO 9001 & TL 9000– Other constraintsOther constraints
Testing for High-AvailabilityTesting for High-Availability
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
1616
The Product Development CycleThe Product Development Cycle
System Spec
Software
Spec
New Product
Idea
New Product
Idea
Software Development SW Unit Test
HW-SWIntegrationHardwareSpec Hardware
DevelopmentHW Unit T
est
Product Verification
Engineering Developmentfunctions
Product Verificationfunctions
Customer & marketDriven inputs
Product LineManagement &
Engineeringinputs
Release to
manufacture
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
1717
Product Verification PhaseProduct Verification Phase
Formal Product Verification Phase
Software Verification
HW-SWIntegration
Test
HW Compliance & Agency approvals
Release to Production
HW Stress Testing
HW Standards/Reqts testing
Software Verification
VolumeProduction
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
1818
Testing for High AvailabilityTesting for High Availability
““Networks are very complex systems Networks are very complex systems and the only way to test them is to and the only way to test them is to partition them into manageable partition them into manageable layers and functions. Doing this is layers and functions. Doing this is truly an art”truly an art”
- Robert Buchanan, Jr.- Robert Buchanan, Jr.
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
1919
Testing for High AvailabilityTesting for High Availability
Systematic, structured testingSystematic, structured testing ANSI / Telcordia / ITU-T Standards-based ANSI / Telcordia / ITU-T Standards-based
testingtesting Environmental testing for hardwareEnvironmental testing for hardware Stress / Load testing for softwareStress / Load testing for software HW & SW fault insertion testingHW & SW fault insertion testing Interoperability testingInteroperability testing Soak testing – continuous operationSoak testing – continuous operation Statistical sampling for manufacturingStatistical sampling for manufacturing
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
2020
Testing for High AvailabilityTesting for High Availability Systematic, structured testingSystematic, structured testing
– Planned test strategyPlanned test strategy– Thorough, well though out test plan:Thorough, well though out test plan:
Test cases traceable back to specificationsTest cases traceable back to specifications Trade-off decisions made for permutations not performedTrade-off decisions made for permutations not performed Test plan includes positive and negative test casesTest plan includes positive and negative test cases Test case pre-defines unambiguous pass/fail criteriaTest case pre-defines unambiguous pass/fail criteria Test environment is described in detailTest environment is described in detail Risks are anticipated and managed with contingency plansRisks are anticipated and managed with contingency plans
– And yet, controlled randomness in the test caseAnd yet, controlled randomness in the test case– Prioritization: run the important tests earlyPrioritization: run the important tests early– Focus on areas of greatest risk – system state transitionsFocus on areas of greatest risk – system state transitions– Learn from the bugs that you findLearn from the bugs that you find– Use of automation to increase coverage, reduce scheduleUse of automation to increase coverage, reduce schedule– Continuous refining, improving of test plan & test casesContinuous refining, improving of test plan & test cases
Portions from Portions from High Quality Software EngineeringHigh Quality Software Engineering, by Ross Collard, by Ross Collard
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
2121
Testing for High AvailabilityTesting for High Availability
Major Causes of Defects NOT Being FoundMajor Causes of Defects NOT Being Found 36% Scope: scenario was beyond the test strategy36% Scope: scenario was beyond the test strategy 21% Permutations: an untested combination failed21% Permutations: an untested combination failed 9% Stochastic: random failure that did not occur during 9% Stochastic: random failure that did not occur during
testingtesting 6% Process: non-compliance in the way the test was 6% Process: non-compliance in the way the test was
performedperformed 5% Oversight: the problem was missed by the tester5% Oversight: the problem was missed by the tester 3% Coverage: the test scenario was not included in the test 3% Coverage: the test scenario was not included in the test
strategystrategy 3% Incomplete test: the scenario was in the test plan, but 3% Incomplete test: the scenario was in the test plan, but
not included in the test casesnot included in the test cases(Percent of all defects which were not found)(Percent of all defects which were not found)
Study by Tellabs, 1998Study by Tellabs, 1998
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
2222
Testing for High AvailabilityTesting for High Availability
Systematic, structured testingSystematic, structured testing ANSI / Telcordia / ITU-T Standards-ANSI / Telcordia / ITU-T Standards-
based testingbased testing Environmental testing for hardwareEnvironmental testing for hardware Stress / Load testing for softwareStress / Load testing for software HW & SW fault insertion testingHW & SW fault insertion testing Interoperability testingInteroperability testing Soak testing – continuous operationSoak testing – continuous operation Statistical sampling for manufacturingStatistical sampling for manufacturing
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
2323
Testing for High AvailabilityTesting for High Availability
ANSI / Telcordia / ITU-T Standards-ANSI / Telcordia / ITU-T Standards-based testingbased testing– Verify that interfaces meet requirements Verify that interfaces meet requirements
of standards (GR-253…)of standards (GR-253…)– Verify fail-over performance meets Verify fail-over performance meets
standards (60 mS)standards (60 mS)
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
2424
Testing for High AvailabilityTesting for High Availability
Systematic, structured testingSystematic, structured testing ANSI / Telcordia / ITU-T Standards-based ANSI / Telcordia / ITU-T Standards-based
testingtesting Environmental testing for hardwareEnvironmental testing for hardware Stress / Load testing for softwareStress / Load testing for software HW & SW fault insertion testingHW & SW fault insertion testing Interoperability testingInteroperability testing Soak testing – continuous operationSoak testing – continuous operation Statistical sampling for manufacturingStatistical sampling for manufacturing
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
2525
Elements of Hardware VerificationElements of Hardware Verification
Hardware Verification
Compliance &
Agency Approvals
Stress TestingHALT/HASS
Standards basedTesting
Physical LayerLogical Layer
EMCSafetyNEBSTelecom
Design Stress TestingAccelerated Life-cycle Testing
(Beyond normal operating limits)Where does it break?
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
2626
Testing for High AvailabilityTesting for High Availability
Environmental testing for hardwareEnvironmental testing for hardware– Operation over temperature, supply Operation over temperature, supply
voltage, vibrationvoltage, vibration– Monitor software performance (traffic, Monitor software performance (traffic,
alarms) during environmental testingalarms) during environmental testing
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
2727
Testing for High AvailabilityTesting for High Availability
Systematic, structured testingSystematic, structured testing ANSI / Telcordia / ITU-T Standards-based ANSI / Telcordia / ITU-T Standards-based
testingtesting Environmental testing for hardwareEnvironmental testing for hardware Stress / Load testing for softwareStress / Load testing for software HW & SW fault insertion testingHW & SW fault insertion testing Interoperability testingInteroperability testing Soak testing – continuous operationSoak testing – continuous operation Statistical sampling for manufacturingStatistical sampling for manufacturing
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
2828
Testing for High AvailabilityTesting for High Availability Stress / Load testing for softwareStress / Load testing for software
– Multiple, simultaneous traffic typesMultiple, simultaneous traffic types– Errors on input interface (example: bit errors)Errors on input interface (example: bit errors)– Maximum user activity: db backup, multiple Maximum user activity: db backup, multiple
session launch, multiple data requestssession launch, multiple data requests– Bottleneck / over-subscription of data trafficBottleneck / over-subscription of data traffic– Fail-over testing: single failureFail-over testing: single failure– Fail-over testing: double failureFail-over testing: double failure– Startup under stress, load, & errorsStartup under stress, load, & errors– Alarm hysteresis, holdoff & alarm stormsAlarm hysteresis, holdoff & alarm storms– Goal: more stress/load than SW will ever see Goal: more stress/load than SW will ever see
operationally (find the breaking point)operationally (find the breaking point)
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
2929
Testing for High AvailabilityTesting for High Availability
Systematic, structured testingSystematic, structured testing ANSI / Telcordia / ITU-T Standards-based ANSI / Telcordia / ITU-T Standards-based
testingtesting Environmental testing for hardwareEnvironmental testing for hardware Stress / Load testing for softwareStress / Load testing for software HW & SW fault insertion testingHW & SW fault insertion testing Interoperability testingInteroperability testing Soak testing – continuous operationSoak testing – continuous operation Statistical sampling for manufacturingStatistical sampling for manufacturing
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
3030
Testing for High AvailabilityTesting for High Availability
HW & SW fault insertion testingHW & SW fault insertion testing– Any hardware subsystem / module failureAny hardware subsystem / module failure
CPU resetCPU reset Power supply failurePower supply failure Oscillator failureOscillator failure Data bus / Address bus line fail high/lowData bus / Address bus line fail high/low Memory corruptionMemory corruption
– File corruptionFile corruption– Resource exhaustion (memory, file handles, Resource exhaustion (memory, file handles,
sockets, semaphores…)sockets, semaphores…)– User error – software should protect againstUser error – software should protect against
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
3131
Testing for High AvailabilityTesting for High Availability
Systematic, structured testingSystematic, structured testing ANSI / Telcordia / ITU-T Standards-based ANSI / Telcordia / ITU-T Standards-based
testingtesting Environmental testing for hardwareEnvironmental testing for hardware Stress / Load testing for softwareStress / Load testing for software HW & SW fault insertion testingHW & SW fault insertion testing Interoperability testingInteroperability testing Soak testing – continuous operationSoak testing – continuous operation Statistical sampling for manufacturingStatistical sampling for manufacturing
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
3232
Testing for High AvailabilityTesting for High Availability Interoperability testingInteroperability testing
– Testing interfaces which pass data back and Testing interfaces which pass data back and forthforth
– Compatibility testing with other equipmentCompatibility testing with other equipment– Compatibility testing with other vendor’s Compatibility testing with other vendor’s
equipmentequipment– Consider both hardware and software versionsConsider both hardware and software versions– Compatibility with current, prior, and next Compatibility with current, prior, and next
version of:version of: Operating System (Unix, Solaris, Windows)Operating System (Unix, Solaris, Windows) Java / JREJava / JRE
– Configurations to be tested must be prioritized Configurations to be tested must be prioritized by their importance and riskby their importance and risk
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
3333
Testing for High AvailabilityTesting for High Availability
Systematic, structured testingSystematic, structured testing ANSI / Telcordia / ITU-T Standards-based ANSI / Telcordia / ITU-T Standards-based
testingtesting Environmental testing for hardwareEnvironmental testing for hardware Stress / Load testing for softwareStress / Load testing for software HW & SW fault insertion testingHW & SW fault insertion testing Interoperability testingInteroperability testing Soak testing – continuous operationSoak testing – continuous operation Statistical sampling for manufacturingStatistical sampling for manufacturing
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
3434
Testing for High AvailabilityTesting for High Availability
Soak testing / longevity testing – Soak testing / longevity testing – continuous operationcontinuous operation– Telecom equipment designed for continuous Telecom equipment designed for continuous
operation for months, years, even decadesoperation for months, years, even decades– Run in lab for 14 hours? 7 days? 3 weeks?Run in lab for 14 hours? 7 days? 3 weeks?– 1 system for 1 year = 8 systems for 45 days?1 system for 1 year = 8 systems for 45 days?– Mixture of clean and errored trafficMixture of clean and errored traffic– Monitor for traffic interruptions, alarmsMonitor for traffic interruptions, alarms– Monitor performance counters: Monitor performance counters:
Error-free seconds counterError-free seconds counter Bad packet counterBad packet counter
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
3535
Testing for High AvailabilityTesting for High Availability
Systematic, structured testingSystematic, structured testing ANSI / Telcordia / ITU-T Standards-based ANSI / Telcordia / ITU-T Standards-based
testingtesting Environmental testing for hardwareEnvironmental testing for hardware Stress / Load testing for softwareStress / Load testing for software HW & SW fault insertion testingHW & SW fault insertion testing Interoperability testingInteroperability testing Soak testing – continuous operationSoak testing – continuous operation Statistical sampling for Statistical sampling for
manufacturingmanufacturing
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
3636
Testing for High AvailabilityTesting for High Availability
Statistical sampling for Statistical sampling for manufacturingmanufacturing– System testing done with small number System testing done with small number
of prototype hardware cardsof prototype hardware cards– Reliability testing must be done on large Reliability testing must be done on large
sample of production hardware cardssample of production hardware cards– Must be done for a long period to be Must be done for a long period to be
statistically significant: thousands of statistically significant: thousands of hours of total run timehours of total run time
9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved
3737
ReferencesReferences““Computer Related Risks”, Computer Related Risks”, Peter G. NeumannPeter G. Neumann
““Code Complete”, Code Complete”, Steve McConnellSteve McConnell
““Software Testing and Quality Assurance”, Software Testing and Quality Assurance”, Ross CollardRoss Collard
True Random Numbers: True Random Numbers: http://www.random.org/nform.htmlhttp://www.random.org/nform.html
Testing Computer Software, Testing Computer Software, by C. Kaner, J. Falk, and H. Nguyenby C. Kaner, J. Falk, and H. Nguyen
IEEE Standard for Software Test Documentation, Std 829 - 1998IEEE Standard for Software Test Documentation, Std 829 - 1998
Black-Box Testing: Techniques for Functional Testing of Software and Systems, Black-Box Testing: Techniques for Functional Testing of Software and Systems, Boris Beizer, Wiley, 1995Boris Beizer, Wiley, 1995
Managing the Testing Process, Managing the Testing Process, Rex BlackRex Black
Classic Testing Mistakes, Classic Testing Mistakes, Brian MarickBrian Marickhttp://http://www.testing.com/writings/classic/mistakes.pdfwww.testing.com/writings/classic/mistakes.pdf
Software QA / Test Resource CenterSoftware QA / Test Resource Centerhttp://http://www.softwareqatest.com/index.htmlwww.softwareqatest.com/index.html