Date post: | 11-Feb-2017 |
Category: |
Software |
Upload: | hugh-mccamphill |
View: | 121 times |
Download: | 1 times |
@hughleo01 Liberty Information Technology
Liberty IT, Belfast and Dublin
Test trend analysis: Towards robust, reliable and timely tests
Hugh McCamphill
@hughleo01 Liberty Information Technology2Liberty IT @hughleo01
Indianapolis Dover Belfast GS LIU DCs Europe RDC Russia DRDC
Redmond DC
Global DRDC
& Utilities
Kansas City DC
Latin America RDC
GS LIU DRDC
Portsmouth DC
GS LIU DC
Turkey DCs India DCs Asia RDC China DCs
Wausau
Fairfield
Boston
Seattle Blanchardstown
11 major IT offices
7 country data centers (DCs)DRDC = Disaster Recovery DC
3 US domestic data centers (DCs)
4 international regional data centersLIU = Liberty International Underwriters
More than 5,000+ IT
employees around the globe
Australia
About Us
@hughleo01 Liberty Information Technology3Liberty IT @hughleo01
Principal Software Engineer in Test
Belfast Selenium Meetup Organiser
Writing Automation Since 2004
WebDriver user since 2011
@hughleo01 Liberty Information Technology4Liberty IT @hughleo01
Which tests are slow
Which steps are slow
By Martin Lewison from Forest Hills, NY, U.S.A. - Cedar Point and Oberlin's Commencement Uploaded by Astros4477, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=27339553
@hughleo01 Liberty Information Technology5Liberty IT @hughleo01
Tumbling Dice : Decorated Anti-tank Blocks near Harkess Rocks, Bamburgh, NorthumberlandImage Copyright Richard West. This work is licensed under the Creative Commons Attribution-Share Alike 2.0 Generic Licence
Which tests are intermittently passing
What are common failures across tests
@hughleo01 Liberty Information Technology6Liberty IT @hughleo01
“Create commons cute robot mural, Toronto" by https://www.flickr.com/photos/margonaut/ is licensed under CC BY 2.0
Where are the issues?
@hughleo01 Liberty Information Technology12Liberty IT @hughleo01
RunTests ResultsPost
Results
A Simple Process Flow
@hughleo01 Liberty Information Technology13Liberty IT @hughleo01
A time series database is a
software system that is optimized
for handling time series data,
arrays of numbers indexed
by time
Elasticsearch is a tool for storing, searching, and
analyzing structured and unstructured data
@hughleo01 Liberty Information Technology14Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
@hughleo01 Liberty Information Technology15Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
@hughleo01 Liberty Information Technology16Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
@hughleo01 Liberty Information Technology17Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{"properties":{"project":{"type":"string"},"environment":{"type":"string"},"duration":{"type":"double"},"name":{"type":"string","index":"not_analyzed"},"classname":{"type":"string","index":"not_analyzed"},"message":{"type":"string","index":"not_analyzed"},"status":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
curl -XPUT “http://localhost:9200/test-result” –
d @mapping_schema_framework.json
@hughleo01 Liberty Information Technology18Liberty IT @hughleo01
{"index":{"_index":"test-result","_type":"functional" }}
{"project":“TestProject","environment":"QA","classname":“TestProject.Foo.Bar","name":"ShouldBeAbleToAddItem","duration":"47.12","created":"2016-09-14 21:35:53","status":"Passed"}
curl -XPOST “http://localhost:9200/test-
result/_bulk?pretty” --data-binary @testResults.json
Posting of results
@hughleo01 Liberty Information Technology19Liberty IT @hughleo01
A Simple Process Flow
RunTests ResultsPost
Results Visualize
@hughleo01 Liberty Information Technology20Liberty IT @hughleo01
Kibana is an open source data visualization plugin for ElasticSearch
@hughleo01 Liberty Information Technology22Liberty IT @hughleo01
New visualization results in a single bucket
@hughleo01 Liberty Information Technology23Liberty IT @hughleo01
Splitting error messages into buckets of top occurring errors
@hughleo01 Liberty Information Technology24Liberty IT @hughleo01
Showing top ten most occurring error messages
@hughleo01 Liberty Information Technology25Liberty IT @hughleo01
Splitting error messages into buckets of tests affected by that error
@hughleo01 Liberty Information Technology26Liberty IT @hughleo01
Error Messages
Individual Tests
Splitting error messages into buckets of tests affected by that error
@hughleo01 Liberty Information Technology30Liberty IT @hughleo01
Step Time Versus Number Of Times Executed
@hughleo01 Liberty Information Technology31Liberty IT @hughleo01
Step Time Versus Number Of Times Executed
@hughleo01 Liberty Information Technology32Liberty IT @hughleo01
Step Time Versus Number Of Times Executed
@hughleo01 Liberty Information Technology34Liberty IT @hughleo01
References
http://martinfowler.com/articles/nonDeterminism.htm
https://watirmelon.blog/2015/11/11/your-tests-arent-flaky/