QAing INFRASTRUCTUREA QA's Role in DevOps World

byAroj P George

Harshad Wankhede

What is Infrastructure Testing?

Typical Development Environment - Single Server

LIDB

ServiceApp

WEB

SERVER

Mocked ThirdParty Services

Developer/QA Laptop

Production Infrastructure – Multi-Tier

Replica Set

DB

DB

DB

LOAD

BALANCER

Service

Service

Service

App

App

App

LOAD

BALANCER

CDN

Third Party Services

Application Ecosystem

Firewalls and VPN and Secure Access Rules

Network Configuration

Storage Configuration

Virtualization

Authentication and Authorization Rules

System Services

Network Services

Operational Processes

Libraries

Application

What if we don’t test the infrastructure?

What is DevOps?

It’s a mindsetBridges gap between Developers and Operations

What to Test?

Database Failure ScenariosDo we have proper error handling, logging and alerting in place for below scenarios?

if the application is unable to read data Network issues DB Primary is down

Does the DB Primary failover if it goes down?Does the application keep running

Do we have alerts for the below?Replication has failedDB backups are not happeningDB high resource utlilization DB queries are very slow

Application Performance Issues

Do we have monitoring for high resource utilizationAre we tracking the response times for application and dependent web servicesAre we tracking errors generated in the logs?Do we get alerts if a cron job fails to run or if it errors?Ex: Newrelic, Nagios, Ganglia metrics collection, Executing Performance Testing scripts regularly

Server or Services are Down

How does the system react to database failureHow does the system react to application failureHow does the system react to third party service failureWhat happens if the load balancer is downEx: Chaos Monkey

Logging and Archiving

Have you configured appropriate logging?Are the logs being properly indexed and rotated?Do you maintain log history using tools like Splunk?Are the DB backups being archived?

Application Errors

Do we show appropriate error pages for various errors500, 503, 404 Pages

What happens if there is a sudden increase in the rate of errors?

Do we get alerted regarding the same.

Resource Consumption Issues

Have you configured alerting for CPU usage?Have you configured alerting for Memory usage?Have you configured alerts for Disk Utilization?

Warning if utilization reaches 80%Critical if utilization reaches 95%

CPU Usage

Memory Usage

Network Usage

Third Party Service Failure

Have you configured alerting when third party services are down?

Security

Have you blocked access to admin configuration?Have you configured firewall rules properly?Penetration Testing

CachingHave you validated application caching?Ex: CacheWarp

Cross functional/Non Functional RequirementsTesting infrastructure is nothing but testing the Cross-

functional Requirements

Chef Recipe

Chef - Templatize Configuration

Chef Demo

Chef is a systems and cloud infrastructure automation framework that makes it easy to deploy servers and applications Infrastructure as CodeKnife is a Chef utilityUsing the Knife SSH command to test the infrastructure

RSpec Tests

Demo

CacheWarpDemo

https://rubygems.org/gems/cachewarp

QA Skills Required:

Good Understanding of the InfrastructureAbility to foresee Issues Command-line proficiency, VimAbility to debug and troubleshoot issuesShould be able to write automated tests (rspec)Basic shell scripting knowledge

Myths about Infrastructure Testing

No need to test Infrastructure. If it works locally it will work on any environmentApplication and Infrastructure Testing strategies are similarApplication Testing is more important than Infrastructure TestingThere aren’t too many different things to test in infrastructureThere is no business value in testing Infrastructure

ToolsChef + KnifeGangliaSplunkGONagiosMingleCacheWarpVagrantVimTmuxPutty

Questions