Automating Security Tests for Continuous Integration

Stephen de Vries @stephendv

www.continuumsecurity.net

About Continuum Security

• Founded 2012• Services: Security Testing, BDD-Security jump start• Products: Securing the SDLC

– Open Source• BDD-Security Testing Framework• OWASP ZAP integration with JUnit• Nessus Java client API

– Commercial• IriusRisk Risk Management for Application Security: www.iriusrisk.com

Security Testing

• Performed after build• Uses external testers• Process is opaque to

dev/opts

Unit/Integration/Functional Testing

• Performed during build• Owned by dev/test• Tests visible to the team

Design Build Unit Tests

Integration Tests

AcceptanceTests Deploy

Development Pre-prod Production

Agile

• Short iterative cycles• Extensive automated testing• Low/zero cost to test• Tests can replace documentation

SecurityTesting

Waterfall

Design Build Unit Tests

Integration Tests

AcceptanceTests Deploy

Development Pre-prod Production

Continuous Delivery with DevOps

• Automated delivery into pre-prod

• Automated acceptance tests

Design Build Unit Tests

Integration Tests

AcceptanceTests Deploy

Development Pre-prod Production

Continuous Deployment with DevOps

SecurityTesting

• Etsy: 50+ deploys per day• Amazon: 300+ per hour• Gov.uk: 10+ deploys per day

• Everyone is responsible for

• Move testing closer to the code

• Continuous automated testing

• Tests are visible to the team

quality

quality

security

securitysecurity

^

DesignAuto. Security Tests

BuildIntegration TestsUnit

TestsAcceptance

Tests Deploy

Development Pre-prod Production

Continuous Deployment with SecDevOps: Blocking tests

Manual Security Tests

Design Build Integration TestsUnit Tests

AcceptanceTests Deploy

Development Pre-prod Production

Continuous Deployment with Semi-SecDevOps: Parallel tests

Manual Security Tests

Auto. Security Tests

Who owns the security tests?

A) Security team

• Benefits of automation• Fast feedback• Poor collaboration• Lack of ownership by DevOps

Who owns the security tests?

B) DevOps team with oversight by Security

• Better collaboration• More sense of ownership of security• Good stepping stone to…

Who owns the security tests?

SecDev

OpsC) Sec + Dev + Ops in a cross-functional team

• Security testing is our problem• We have the tools and skills to manage it

Automated Security Tests should:

• return either a pass or fail result• execute quickly (similar to acceptance tests)• test infrastructure and application tiers• test functional security features, e.g. Login, Password Reset• capture manual testing processes and automate them,

i.e. security regression tests• be checked into version control along with the code• be understandable by the whole team

BDD-Security Testing Framework

https://github.com/continuumsecurity/bdd-security

BDD-Security = JBehave +

OWASP ZAP + Nessus + Internal security tools + Pre-written baseline security specifications

Selenium +

Infrastructure Security Testing

Application Security Testing

HTTP/S Proxy

Manual Application Security Testing with OWASP ZAP

Selenium

ZAP

API

HTTP/S Proxy

Manual Application Security Testing with OWASP ZAPAutomated

^

BDD-Security

Functional Security Tests

Integrating with Jenkins• Configuration• Test run

Summary

• Security testing is just another form of software testing• Automate as much as possible for faster feedback• Security Tests can be treated as security requirements

• Self Verifying Requirements!• Tests written in a BDD language foster collaboration between

sec, dev and ops• Automated Security tests should include more than just

scanning

Other related tools

• Mittn (Python + Burp Intruder) https://github.com/F-Secure/mittn

• ZAP-JUnit (Java) https://github.com/continuumsecurity/zap-webdriver

• Guantlet (Ruby) http://gauntlt.org/

• OWASP ZAP Jenkins plugin https://wiki.jenkins-ci.org/display/JENKINS/Zapper+Plugin

Thank you

www.continuumsecurity.net@stephendv