SQA in a TestOps era

SQA in a TestOps era

TomaszJaniszewski

DawidPacia

THE TRADITIONAL MODEL OF

SOFTWARE TESTING

THE TRADITIONAL MODEL OF SOFTWARE TESTING

TESTS

QUALITY ASSESSMENT

RUN TESTS

SYSTEM UNDER TEST

ANALYSISPASS? FAIL?

“To improve is to change; to be perfect is to change often.”

Winston Churchill

➜ Facebook users send on average over 31 million messages and view 2.77 million videos every minute

➜ Twitter users generates 347,222 Tweets each minute – or 21 million Tweets per hour

➜ Google provides search results for over 1 billion search queries each day

➜ Amazon deploys new software to production every 11.6 seconds.

The times has changed...

Let’s start with the firstset of slides

1.TRANSITION

HEADLINE

EXPOSURE CONTROL

TEST CASES AND MONITORS

THE TESTOPS MODEL OF SOFTWARE TESTING

EXPOSED USERS AND SYSTEMS

SYSTEM UNDER TEST

PRODUCTION USERS AND SYSTEMS

ACTIVE MONITORS

ANALYSIS

QUALITY ASSESSMENT

PRODUCTION USAGE

TELEMETRY DATA

KPIs / PATTERNS

CONTINUOUS BUILD

CODE CHANGE / FIX

SCM(Git, Svn, Bazaar ;)

NEW BUILD(Ant, Maven, Gradle, etc.)

CI TOOLS(Jenkins, Bamboo, Travis CI, Ansible,

etc.)

CONTINUOUS VALIDATION

TEST ENVIRONMENTS

BUILD

DEPLOYMENT

TEST RESULTS

CONTINUOUS DELIVERY

PRODUCTIONENVIRONMENTS

CONTINUOUS MONITORING

MONITORING(Kibana, Grafana,

Scalyr, Cacti, Amazon

CloudWatch, etc.)

SYNTHETIC / SEMANTIC

MONITORING

➜ Synthetic monitoring (or semantic monitoring) runs a subset of an application's automated tests against the live production system on a regular basis.

Continuous Monitoring

➜ Integration with DevOps infrastructure

➜ What tests to run and when➜ No need to stand up the whole

system to test

TestOps: What Does This Mean?

WHY SHOULD WE CARE SO

MUCH?

FAKAPY● Zgłaszanie błędu w komentarzy w Jirze - 500 klientom nie działało menu

nawigacyjne po aplikacji● Dobrze zorganizowane glow w Gicie - pull req, praca miedzy zespolami● Szkiletet ● Jak pisac kod (python)● Nazwy obiektow!● Re-uzywalnosc kodu● Korzystanie z paczek● Pop● Pliki readme● Jak unikac zlych praktyk● Konwencja komitowanie (a nie ‘fixed small issue’)● Konsekwencja pracy przez 2-mc na wlasnym branchu● Dlaczego nie tylko produkt powinien byc testowany, ale również same

testy

BUT IN A WORLD OF FALSE...

False Negatives

False Positives

The test is marked as Failed, when the functionality is

working

The test is marked as Passed, when the functionality is not

working.

...YOU CAN ENCOUNTER SOMETHING WORSE

False Negatives

False Positives

Flaky Tests

TESTS FAILING INTERMITTENTLY

RIS

KV

ALU

ECONTINUOUS: VALUE vs RISK

PLAN CODE BUILD TEST RELEASE DEPLOY OPERATE

COLLABORATION

AGILE DEVELOPMENT

CONTINUOUS INTEGRATION

CONTINUOUS DELIVERY

DEVOPS

Dawid Pacia

“More automatically, more continuously,

more risky...”

TESTOPS TOOLBOX AND BESTPRACTICES

GIT PROPER MODEL

19

time / commits

master

development

feature 1

feature 2

release

The Great QA Wall

Pipelinespipeline { agent none stages { stage('error') { steps { parallel( "Wait": { sleep 1 }, "NodeExec": { node(label: 'master') { isUnix() }… } stage('ExternalJob') { steps { build(job: 'TestMulti1', propagate: true) } ...

Exam

ple

But here it comes… Blue Ocean!

➜ You can use compose to bring up any number of instances of the services in your stack, example:

> docker-compose up --scale web=10 database=3

This is great for testing database clusters or load balancers, or resiliency with Pumba

Docker - Scalability Testing

➜ Docker can be useful for testing different system configurations like for example different DB backends:

○ Bring up new stack (with MySQL backend), Seed with data, Run API tests, Tear down stack.

○ Bring up new stack (with Postgres backend), Seed with data, Run API tests, Tear down stack.

○ Report on any differences in test results between MySQL and Postgres backends.

○ Integrates easily with Jenkins !

Docker - Configuration Testing

Selenium GridRIP

Regression Tests

Jenkins CI

Real-time reporting of Selenium tests on Kibana

UITests

AcceptanceTests

Apache Kafka

Elasticsearch Logstash

Jobs

Logs

Logs

Transformed documents

Kibana

Query results

Final Results

➜ Setup overview○ Configure Apache JMeter (through user.properties file) to

Create a CSV File:

/opt/apache/jmeter/apache-jmeter-3.2/bin/jmeter -n -t Basic\ Elasticsearch\ Test.jmx -l ./results.csv

○ Configure and Run Logstash○ Visualize the Results

Real-time reporting of JMeter results on Kibana

Final Results

➜ TestOps provides a unique aspect for automated testing

➜ It works with the Dev team to set up CI/CD/CM

➜ It creates a cloud or virtual technology to set up a testing environment on demand

➜ It plays a vital role to integrate the automated testing with Dev team for CI

➜ Explore and investigate!

Takeaway

Thanks!Any questions?You can find us at:

janiszewski.tomasz@aeon.networkpacia.dawid@aeon.network

web:aeon.network