Trustworthy Software Everywhere

Tao Xie

Automated Software Engineering GroupDepartment of Computer ScienceNorth Carolina State University

https://sites.google.com/site/asergrp/

Trustworthy Software Everywhere

Static Verification Problem: API properties are not available▪ E.g., fopen’s return needs to be NULL-CHECK

Solution: mining API properties from client code

Dynamic Verification (a.k.a. Software Testing)

3

Searching + Mining

Supported by NSF CSR and ARO

4

Development Assisted by Searching+Mining

PARSEWeb [ASE 07]

PARSEWeb Source object type &

Destination object type

Method-invocation sequence

MAPO [ECOOP 09] API method Frequent subsequences of API methods SpotWeb [ASE 08] Framework hotspots/coldspots

5

Verification Assisted by Searching+Mining

Detect deviant behavior as bugs in programs

• Neglected-condition bugs [ASE 09]• Exception-handling bugs [ICSE 09]• Error-handling bugs [FASE 09]• API-sequencing bugs [ESEC/FSE 07]

6

Mining Text in Software Engineering• Detect duplicate bug reports [ICSE 08]• Identify security bug reports [MSR 10]• Mine resource specifications from Javadoc

[ASE 09, Best Paper Award, SIGSOFT Distinguished Paper]

javax.resource.cci.ConnectioncreateInteraction():“Creates an interaction associated with this connection.”getMetaData():“Gets the information on the underlying EIS instance represented through an active connection.”close():“Initiates close of the connection handle at the application level.”

Supported by IBM Jazz Award

Software Testing Problems

=?

Outputs

Expected

Outputs

Program

+

Test inputs

Test Oracles

Test Generation Generating high-quality test inputs (e.g.,

achieving high code coverage)

Test Oracles Specifying high-quality test oracles (e.g.,

guarding against various faults)

Test Generation

Human Expensive, incomplete, …

Brute Force Pairwise, predefined data, etc…

Random: Cheap, Fast “It passed a thousand tests” feeling

Dynamic Symbolic Execution: Pex, CUTE,EXE Automated white-box Not random – Constraint Solving

Dynamic Symbolic Execution

Code to generate inputs for:

Constraints to solve

a!=null a!=null &&a.Length>0

a!=null &&a.Length>0 &&a[0]==1234567890

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Observed constraints

a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890

a!=null &&a.Length>0 &&a[0]==1234567890

Data

null

{}

{0}

{123…}a==null

a.Length>0

a[0]==123…T

TF

T

F

F

Execute&MonitorSolve

Choose next path

Done: There is no path left.

Negated condition

Loops Fitnex [DSN 09]

Generic API functions e.g., RegEx matching IsMatch(s1,regex1) Reggae [ASE 09-sp]

Method sequences MSeqGen [ESEC/FSE 09]

Environments e.g., file systems, network, db, … Parameterized Mock Objects [AST 09, ASE 10-sp]

Opportunities Regression testing [ICSE 09-nier] Developer guidance (cooperative developer testing)

Challenges of DSE

Supported by NSF SoD, NSF SHF, NSF CAREER, Microsoft Research Award

Loops Fitnex [DSN 09]

Generic API functions e.g., RegEx matching IsMatch(s1,regex1) Reggae [ASE 09-sp]

Method sequences MSeqGen [ESEC/FSE 09]

Environments e.g., file systems, network, db, … Parameterized Mock Objects [AST 09, ASE 10-sp]

Applications Test network app @Army division, Fort Hood, Texas Test DB app of hand-held medical assistant device@FDA Test .NET base libraries @Microsoft

NCSU Industry Tech Transfer

Pex Visual Studio Power Tool

Download counts (20 months)(Feb. 2008 - Oct. 2009 )

Academic: 17,366 Devlabs: 13,022 Total: 30,388

Pex for Fun (Coding Duels)http://www.pexforfun.com/

Various countries/regions Software internationalization▪ Locating constant strings to translate [ICSE 09,

FSE 10]▪ E.g., translating Megamek (a realtime strategy

game)

Various programming languages PL translation▪ E.g., translating Java to C# [ICSE 10]

Trustworthy Software Everywhere

Various types of software Database applications [ASE 10-sp] Network/file-system applications [AST 09] Game applications [ICSE 09] Cyber-physical systems (power grid,

medical device software, …) Could applications Social network applications …

Trustworthy Software Everywhere

Supported by NSF SHF, NSF CAREER

Various types of quality attributes Functional correctness Security (NIST/Fermi Lab collaboration)▪ Testing/verification of access control policies [WWW

07, ACSAC 08, SIGMETRICS 08, IEEE TC 10]▪ Testing/verification of firewall policies [SRDS 08/09,

LISA 10 Best Student Paper]▪ Identification of security bug reports [MSR 10]▪ Attack generation

Performance▪ IBM RTP collaboration on Rational Performance Tester

…

Trustworthy Software Everywhere

Supported by NSF CyberTrust, NIST, IBM Faculty Awards

Trustworthy Software Everywhere

http://people.engr.ncsu.edu/txie/https://sites.google.com/site/asergrp/