Date post: | 22-Dec-2015 |
Category: |
Documents |
Upload: | meryl-lyons |
View: | 217 times |
Download: | 1 times |
Enabling continuous quality assurance and optimization in future enterprise cloud service brokers
Call: FP7-ICT-2011-8 Grant agreement no: 318392 www.broker-cloud.eu
Verification & Testing: SAP HANA Case StudyAnthony J H Simons (University of Sheffield)
Wolfgang K R Schwach (SAP Karlsruhe)
Overview
• Verification and Testing Toolset• Service Specification• Validation and Verification• Test Generation and Grounding• Test Execution on SAP HANA Cloud App
– V&T tools: Anthony Simons, Uni. Sheffield– App & grounding: Wolfgang Schwach, SAP
19/04/23 Anthony J H Simons, University of Sheffield 2
Verification & Testing Tools
19/04/23 Anthony J H Simons, University of Sheffield 3
Specification
• Why?– Gentle pressure towards common standards– Provide basis for formal service certification
• What?– Describe service interfaces and behaviour– State-based model of service API– IOPE model of detailed behaviour– XML language Java metamodel
19/04/23 Anthony J H Simons, University of Sheffield 4
Holiday Booking Service
19/04/23 Anthony J H Simons, University of Sheffield 5
View Days
Book DaysDelete
Days
select/fail
bookDays/ok
select/highselect/low
back/oksave/ok
back/okdelete/ok
choose/onechoose/fourchoose/seven
choose/onechoose/fourchoose/seven
save/failsave/duplicatesave/overflow
XML Specification – FSM
19/04/23 Anthony J H Simons, University of Sheffield 6
XML Specification – IOPE
19/04/23 Anthony J H Simons, University of Sheffield 7
Validation & Verification
• Why?– Designers write incomplete specifications– Designers write incorrect specifications
• How?– Validation tool explores state-transition diagram
• Reports unhandled events, missing transitions
– Verification tool simulates all input/memory space• Reports blocking/nondeterministic operations• Blocking: input exists which triggers no operation• Nondeterminism: some input triggers many operations
19/04/23 Anthony J H Simons, University of Sheffield 8
Verification Fault Analysis
19/04/23 Anthony J H Simons, University of Sheffield 9
off-by-one indexing
error
Test Generation
• Why?– More thorough coverage than manual testing– Assures full compliance to the specification
• How?– Generation tool explores the whole specification– Creates all interleaved sequences up to depth k– Reduces test-suite size by optimising
• Removes infeasible sequences (blocked in the spec)• Removes redundant sequences (equivalence-classes)• Compresses executable suite (multi-objective tests)
19/04/23 Anthony J H Simons, University of Sheffield 10
Generated Test Suite
19/04/23 Anthony J H Simons, University of Sheffield 11
multi-objective
test
Test Grounding
• Why?– High-level test suites are technology-neutral– Platform-specific translations required
• How?– Platform providers offer bespoke groundings– Broker@Cloud framework provides hooks
• XML test suite parsed as a Java metamodel instance• Visitor Design Pattern is used to generate concrete tests• Bespoke grounding to e.g. Java, SOAP, REST, Selenium
19/04/23 Anthony J H Simons, University of Sheffield 12
Example Java Grounding
19/04/23 Anthony J H Simons, University of Sheffield 13
auto-gen JUnit test assertions
SAP HANA Case Study
• User may book up to 5 days vacation– blocks of days can be booked, selected, deleted– prevents overlapping blocks, more than 5 days
• App designed using the SAP UI5 toolkit– visually pleasing point-and-click interface– operations nominally match the earlier specification– but a rich-client web app API is hard to test
• Deployed on the SAP HANA cloud platform– temporarily available at URI: https://brokercloud-
d050960trial.dispatcher.hanatrial.ondemand.com/
19/04/23 Anthony J H Simons, University of Sheffield 14
App Running on HANA
19/04/23 Anthony J H Simons, University of Sheffield 15
Selenium Test Driver
• Selenium: a tool for web-testing– normally used to capture-and-record a user's point-
and-click gestures interactively
– can replay recorded test scripts, stored as HTML tables, to run web app regression tests
• Leverage to test a rich-client cloud app– given the Selenium DOM for the web app
– translate high-level test suite into Selenium test scripts
– use the Selenium engine to execute the test scripts
19/04/23 Anthony J H Simons, University of Sheffield 16
Selenium Grounding Rules
19/04/23 Anthony J H Simons, University of Sheffield 17
Executable Test Scripts
19/04/23 Anthony J H Simons, University of Sheffield 18
Executing the Tests
19/04/23 Anthony J H Simons, University of Sheffield 19
Subtle Faults Detected
• Functionality– service failed to notify duplicate holiday booking as
a separate error case– service failed to distinguish overflow and duplicate
holiday booking responses
• Security– service executed operations in extra states, when
the operations should be ignored– service revealed response information, when the
request should be ignored
19/04/23 Anthony J H Simons, University of Sheffield 20
Beyond State-of-the-Art
• Extends leading SXM theory for MBT– Adds reasoning about the full SXM model– Adds validation and full verification capability– Adds test optimisation capability
• Exceeds all industrial testing methods– Full coverage of the specification achieved– All positive and negative test cases created– Identifies top 10% uniquely discriminating tests
19/04/23 Anthony J H Simons, University of Sheffield 21
• AcknowledgementFP7-ICT-2011-8 grant agreement no: 318392
• Project websitewww.broker-cloud.eu
Questions
19/04/23 Anthony J H Simons, University of Sheffield 22
References
• Broker@Cloud D20.1 State of the art and research baseline, http:www.broker-cloud.eu, 2013.
• Broker@Cloud D20.3 Requirements for continuous quality assurance and optimisation in cloud brokerage, http:www.broker-cloud.eu, 2013.
• AJH Simons and RE Lefticaru, Specification, verification and testing of brokered software services in the cloud, IEEE ICST 2015, Graz (submitted, under review)
• AJH Simons and RE Lefticaru, Stream X-Machine based verification and testing of cloud services, 2nd Int. Workshop on Cloud Service Brokerage, ESOCC, Manchester, 2 Sept., CCIS, 2014.
• M Kiran, A Friesen, AJH Simons and WKR Schwach, Model-based testing in cloud brokerage scenarios, 1st Int. Workshop on Cloud Service Brokerage, ICSOC, Berlin, 2 Dec., LNCS, 2013.
19/04/23 Anthony J H Simons, University of Sheffield 23