An Integrated Data Model Verifier with Property Templates

Jaideep Nijjar Ivan Bocic Tevfik Bultan

{jaideepnijjar, bo, bultan}@cs.ucsb.edu

University of California, Santa BarbaraDepartment of Computer Science

Verification Lab

FormaliSE 2013

Web Software Everywhere• Commerce, entertainment, social interaction

• We will rely on web apps more in the future

• Web apps + cloud will make desktop apps obsolete

Model-View-Controller Pattern

DB

• Benefits of the MVC pattern:• Separation of

concerns • Modularity• Abstraction

• MVC pattern has become the standard way to structure web applications• Ruby on Rails• Zend for PHP• CakePHP• Struts for Java• Django for Python• …

ViewModel

Controller

MVC Application

• Ruby on Rails

Data Model

• ActiveRecords

Formal Model

• Sets and Relations

Verification

A Data Model Verification Approach

MVC Design Principles

Automatic Extraction

Add data model properties

• Bounded and Unbounded

iDaVer: Integrated Data Model Verifier

formal data model + property + technique

(expressed using Templates)

Active Records

Property Failed +

Counterexample

Property Verified

Unknown

Model Extraction

Choice of technique

Verification

Bounded (Alloy)

Unbounded (SMT Solver)

Unbounded (FOL Thm Prover)

Properties

Outline

• Motivation• Overview of Approach• Rails Data Models

• Basic Relations• Options to Extend Relations• Formalization of Semantics

• Property Templates• Verification Techniques• Case Study• Conclusions and Future Work

A Rails Data Model Exampleclass User < ActiveRecord::Base

has_and_belongs_to_many :roleshas_one :profile, :dependent => :destroyhas_many :photos, :through => :profile

endclass Role < ActiveRecord::Base

has_and_belongs_to_many :usersendclass Profile < ActiveRecord::Base belongs_to :user has_many :photos, :dependent => :destroy has_many :videos, :dependent => :destroy, :conditions => "format='mp4'"endclass Tag < ActiveRecord::Base

belongs_to :taggable, :polymorphic => trueendclass Video < ActiveRecord::Base belongs_to :profile has_many :tags, :as => :taggableendclass Photo < ActiveRecord::Base ...

Role

*

0..1

1

User

Profile

*

1

Video

*

1

Taggable

*

Tag

1

* 1Photo

*

1

format=.‘mp4’

Rails Data Models

• Data model verification: Analyzing the relationships between data objects

• Specified in Rails using association declarations inside the ActiveRecord files• The basic relationships

• One-to-one• One-to-many• Many-to-many

• Extensions to the basic relationships using Options• :through, :conditions, :polymorphic, :dependent

The Three Basic Relationships in Rails• One-to-One

.

• One-to-Many

class User < ActiveRecord::Base has_one :profileend.

class Profile < ActiveRecord::Base belongs_to :userend

class Profile < ActiveRecord::Base has_many :videosend.

class Video < ActiveRecord::Base belongs_to :profileend

User

Profile

1

Profile

Video

*

1

1

The Three Basic Relationships in Rails

• Many-to-Manyclass User < ActiveRecord::Base has_and_belongs_to_many :usersend

class Role < ActiveRecord::Base has_and_belongs_to_many :rolesend

User

Role

*

*

Options to Extend the Basic Relationships

• :through Option• To express transitive relations

• :conditions Option• To relate a subset of objects to another class

• :polymorphic Option• To express polymorphic relationships

• :dependent Option• On delete, this option expresses whether to delete the associated

objects or not

The :through Optionclass User < ActiveRecord::Base

has_one :profilehas_many :photos, :through => :profile

endclass Profile < ActiveRecord::Base belongs_to :user has_many :photosendclass Photo < ActiveRecord::Base belongs_to :profileend

Profile

User Photo

*

*

0..1 1

1

1

The :dependent Option

• :delete directly delete the associated objects without looking at its dependencies

• :destroy first checks whether the associated objects themselves have associations with the :dependent option set

class User < ActiveRecord::Base has_one :profile, :dependent => :destroyend

class Profile < ActiveRecord::Base belongs_to :user has_many :photos, :dependent => :destroyend

PhotoProfileUser *1 10..1

Formalizing Rails Semantics

• S: The sets and relations of the data model (data model schema)• e.g. { Photo, Profile, Role, Tag, Video, User} and the relations

between them

• C: Constraints on the relations• Cardinality constraints, transitive relations, conditional relations,

polymorphic relations

• D: Dependency constraints • Express conditions on two consecutive instances of a relation such

that deletion of an object from the first instance leads to the other instance

Formal data model: M = <S, C, D>

Outline

• Motivation• Overview of Approach• Rails Data Models• Property Templates• Verification Techniques• Case Study• Conclusions and Future Work

Property Templates

• User-friendly and makes it easy to express properties• Manually writing properties is a time-consuming and error-prone

process• Requires familiarity with input modeling language of solver

• Templates are language-neutral• Do not require familiarity with SMT-LIB, Spass and Alloy

languages, and understanding of output specifications• Make it easy to rerun the tool and switch the verification

technique, by not requiring the user to rewrite the property

• Eight property templates available for the most common data model properties

Property Templates

• alwaysRelated[classA, classB]• To check that objects from classA are always related to objects of

classB• someMultipleRelated[classA, classB]

• To check that it is possible for objects of classA to be related to more than one object of classB

• someUnrelated[classA, classB]• To check that it is possible for an object of classA to not be

related to any objects from classB• transitive[classA, classB, classC]

• To check that the relation between classA and classC is the composition of the relations between classA and classB, and classB and classC

Property Templates

• noDangling[classA, classB]• To check that when an object of classA is deleted, there are no

objects from classB that are left with a dangling reference to the deleted object

• deletePropagates[classA, classB]• To check that when an object of classA is deleted, related objects

in classB are also deleted• noDeletePropagation[classA, classB]

• To check that when an object of classA is deleted, related objects in classB are not deleted

• noOrphans[classA, classB]• To check that deleting an object from classA does not cause

related objects in classB to be orphaned • An orphan is an object that is related to no other object

Outline

• Motivation• Overview of Approach• Rails Data Models• Property Templates• Verification Techniques

• Bounded Verification with Alloy• Unbounded Verification with SMT Solver• Unbounded Verification with FOL Theorem Prover

• Case Study• Conclusions and Future Work

iDaVer

SMTSolver

instance or unsat or unknown

formulaSMT-LIB Encoder

Results Interpreter

UNBOUNDED VERIFICATION

AlloyAnalyzer

instance or unsat

formulaAlloy Encoder

Results Interpreter

BOUNDED VERIFICATION

formal data model + property

Active Records

Model Extraction

Choice of techniqueProperties

Theorem Prover

proof found or completion foundor timeout

formulaFOL Encoder

Results InterpreterProperty Failed +

Counterexample

Property Verified

Unknown

Bounded Verification

UNBOUNDED VERIFICATION

AlloyAnalyzer

instance or unsat

formulaAlloy Encoder

Results Interpreter

BOUNDED VERIFICATION

formal data model + property

Active Records

Model Extraction

Choice of techniqueProperties

SMTSolver

instance or unsat or unknown

formulaSMT-LIB Encoder

Results Interpreter

Theorem Prover

proof found or completion foundor timeout

formulaFOL Encoder

Results InterpreterProperty Failed +

Counterexample

Property Verified

Unknown

Alloy Language

• A declarative specification language for object modeling

• Based on first-order logic

• Set-based representation of objects• Defines sets of objects using signatures (sigs)• Defines relations using fields inside the signatures• Add additional constraints about the model as facts

• Well-suited for formally specifying data models• Can add assertions to specify properties about the

specification

Alloy Analyzer

• Automated verification of Alloy specifications is undecidable if the domains are not bounded

• To ensure decidability, Alloy Analyzer restricts the domains to a finite scope• User-specified • A finite bound on the sizes of the domains

• SAT-based bounded verification• Alloy Analyzer translates the Alloy verification query to a Boolean

logic formula satisfiability query • Then invokes an off-the-shelf SAT-solver

Sample Translation to Alloy

class User < ActiveRecord::Base has_one :profileend

class Profile < ActiveRecord::Base belongs_to :userend

sig Profile {}sig User {}one sig State {

profiles: set Profile,users: set User,relation: Profile lone -> one User

}

Bounded Verification of Data Models

• Automatically translate the formal data model extracted from the Active Records and the property to Alloy

• User may specify a bound or use the default

• Use Alloy Analyzer to perform bounded verification

• Possible outputs:• Assertion holds within the

given bound• A counterexample proving

falsified assertions

Formal data model + Property+ Bound

Alloy Encoder

instance or unsat

formula

AlloyAnalyzer

BOUNDED VERIFICATION

Results Interpreter

Property Fails + Counterexample

Property Verifies (within bound)

Unbounded with SMT Solver

UNBOUNDED VERIFICATION

AlloyAnalyzer

instance or unsat

formulaAlloy Encoder

Results Interpreter

BOUNDED VERIFICATION

formal data model + property

Active Records

Model Extraction

Choice of techniqueProperties

Theorem Prover

proof found or completion foundor timeout

formulaFOL Encoder

Results InterpreterProperty Failed +

Counterexample

Property Verified

Unknown

SMTSolver

instance or unsat or unknown

formulaSMT-LIB Encoder

Results Interpreter

Satisfiability Modulo Theories (SMT)

• SMT-solvers automatically check the satisfiability of first-order formulas with respect to a set of theories

• Typical theories include: Linear Arithmetic, Arrays, Bit Vectors, Equality with Uninterpreted Functions• Only implicit universal quantification

• Theory of Equality with Uninterpreted Functions.• Language: Variables, Constants, Uninterpreted function symbols,

Predicate ‘=‘, and Boolean connectives• Uninterpreted functions have no properties except their

signature and functional consistency: a = b => F(a) = F(b)• Example formula: F(x) = F(G(y)) ˅ H(x,y) = 1

SMT Solvers

• There are many SMT solvers out there that support popular theories

• However many of them are not suitable for us • we need support for quantified formulas to handle the property

templates• Microsoft’s Z3 supports quantified expressions in the theory of

uninterpreted functions• Uses heuristics for eliminating quantifiers in formulas• May return ‘unknown’ during satisfiability check

SMT-LIB

• Defines a standard input language for SMT Solvers• Defines theories and logics in which formula can be written• Lisp-like format: Specifications are a series of s-expressions• Declare types using declare-sort command

• (declare-sort User)

• Declare uninterpreted functions using declare-fun command• (declare-fun foo (Domain) Range)

• Quantifier commands: (forall )and (exists )• Add constraints and properties using assert command• Check satisfiability by using (check-sat) command

Sample Translation to SMT-LIB

class User < ActiveRecord::Base has_one :profileend

class Profile < ActiveRecord::Base belongs_to :userend

(declare-sort User)(declare-sort Profile)(declare-fun relation (Profile) User)(assert (forall ((p1 Profile)(p2 Profile))

(=> (not (= p1 p2)) (not (= (relation p1) (relation p2) ))

) ))

Unbounded Verification using SMT Solvers

SMTSolver

instance or unsat or unknown

formula SMT-LIB Encoder

Results Interpreter

UNBOUNDED VERIFICATION

• Automatically translate formal data model and property into the theory of uninterpreted functions with quantification (SMT-LIB)

• Use the SMT solver Z3 to perform satisfiability check

• For failing assertion properties, our tool interprets outputs and forms a counterexample

• Unknowns (and timeouts) possible since the theory is undecidable

Formal data model + Property

Property Failed +

Counterexample

Property Verified

Unknown

Unbounded Verification

UNBOUNDED VERIFICATION

AlloyAnalyzer

instance or unsat

formulaAlloy Encoder

Results Interpreter

BOUNDED VERIFICATION

formal data model + property

Active Records

Model Extraction

Choice of techniqueProperties

SMTSolver

instance or unsat or unknown

formulaSMT-LIB Encoder

Results Interpreter

Property Failed +

Counterexample

Property Verified

Unknown

Theorem Prover

proof found or completion foundor timeout

formulaFOL Encoder

Results Interpreter

FOL Theorem Provers

• Rails data models and properties are expressible in first-order logic (FOL) with equality and quantifiers• Note that this is an undecidable theory

• There are automated theorem provers for first-order logic• They use search strategies to find proofs • However due to undecidability of the FOL they cannot always

give a definite answer• We use the FOL theorem prover, Spass

Modeling Active Records using FOL

• Model object types by declaring a unary predicate• Returns true if an object is a member of that class

• Model relations between data objects using binary predicates• Returns true if the two objects are related

• Axioms are used to express constraints on the data model, e.g.• Specifying cardinality, dependency, and transitivity constraints on

relations• Specifying that predicates denoting classes not related by

inheritance are mutually exclusive • Conjectures are used to model the property to be checked

• To verify the property holds on the data model, send the following formula to the theorem prover:

axioms => conjecture

Sample Translation to Spass

list_of_symbols. sorts[Profile, User]. predicates[(relation, 2)]. end_of_list.list_of_formulae(axioms). formula(forall([Profile(a)], not(User(a)))). formula(forall([User(a)], not(Profile(a)))). formula(forall([a, b], implies( relation(a, b), and(Profile(a), User(b))))). formula(forall([a, b1, b2], implies( and(relation(b1,a), relation(b2,a)), equal(b1,b2)))). formula(forall([a, b1, b2], implies( and(relation(a,b1), relation(a,b2)), equal(b1,b2)))). formula(forall( [Profile(a)], exists([b], relation(a, b)) )).end_of_list.

Unbounded Verification of Data Models using Theorem Provers

Theorem Prover

formula FOLEncoder

Results Interpreter

UNBOUNDED VERIFICATION

• Automatically translate formal data model and property into Spass’s input language (FOL)

• Use the theorem prover Spass to see if formula provable• Interpret results to determine

whether property holds• Does not produce

counterexamples since theorem provers are not designed to do so

• If Spass times out (due to undecidability of theory), iDaVer returns ‘Unknown’

Formal data model + Property

Property FailedProperty Verified

Unknown

proof found or completion found

or timeout

Outline

• Motivation• Overview of Approach• Rails Data Models• Property Templates• Verification Techniques• Case Study• Conclusions and Future Work

Case Study• LovdByLess, a social networking application

• LOC: 3787

• iDaver input:• Path of the directory containing the Active Record files• Name of the file containing the properties to check

(Expressed using property templates!)• Verification technique

• Number Active Record files: 13

Case Study

• Check: alwaysRelated[Photo, Profile]• Solver: Spass

+ Unbounded verification− No sample instance− May report unknown or timeout

Case Study

• Check: someUnrelated[ForumTopic, ForumPost]• Solver: Z3

+ Unbounded verification + Sample instance − May report unknown or timeout

Case Study

• Check: deletePropagates[Profile, Photo]• Solver: Alloy

Case Study

• Check: deletePropagates[Profile, Photo]• Solver: Alloy

+ Counterexample data model instance+ Always returns a result (for small domains)− Bounded− Slower

Summary of Technique Pros and Cons

Alloy SMT Solvers Theorem Provers

Unbounded Verification No Yes Yes

Produces Counterexample Yes Yes No

GeneratesUnknown No Yes Yes

Speed Slow Fast Fast*

*In particular, Spass was slightly faster than Z3 in our experiments and timed out less frequently

Outline

• Motivation• Overview of Approach• Rails Data Models• Property Templates• Verification Techniques• Case Study• Conclusions and Future Work

Conclusions and Future Work

• It is possible to extract formal specifications from MVC-based data models and analyze them

• We were able to find data model errors in real-world applications using some of these techniques (ISSTA’11, ASE’12)

• Integration of multiple automated verification tools makes overall approach more flexible

• Property templates simplify property specification • We have some recent work on

• automated property inference (ISSTA’13)• analyzing actions that update the data store (submitted)

• Main goal: Verifiable data model specification!

Questions?

Related Work

• Verification of Web Applications • [Krishnamurti et al, Springer 2006 ] focuses on correct handling of

the control flow given the unique characteristics of web apps • Works such as [Hallé et al, ASE 2010] and [Han et al, MoDELS

2007] use state machines to formally model navigation behavior• In contrast to these works, we focus on analyzing the data model

• Formal Modeling of Web Applications• WebAlloy [Chang, 2009]: user specifies the data model and

access control policies; implementation automatically synthesized• WebML [Ceri et al, Computer Networks 2000]: a modeling

language developed specifically for modeling web applications; no verification

• In contrast, we perform model extraction (reverse engineering)

Related Work• Verification of Ruby on Rails applications

• Rubicon [Near et al, FSE 2012] verifies the Controller whereas we verify the Data Model

• Requires manual specification of application behavior, whereas we verify manually written properties

• Limited to bounded verification• Data Model Verification using Alloy

• [Cunha and Pacheco, SEFM 2009] maps relational database schemas to Alloy; not automated

• [Wang et al, ASWEC 2006] translates ORA-SS specifications to Alloy, and uses the Analyzer to produces instances of the data model to show consistency

• [Borbar et al, Trends 2005] uses Alloy to discover bugs in browser and business logic interactions

Related Work

• Unbounded Verification of Alloy Specifications using SMT Solvers• [Ghazi et al, FM 2011], approach not implemented• More challenging domain since Alloy language contains

constructs such as transitive closures • Specification and Analysis of Conceptual Data Models

• [Smaragdakis et al ASE 2009, McGill et al ISSTA 2011] use Object Role Modeling to express data model and constraints

• Focus is on checking consistency and producing test cases efficiently

• Using Patterns to Facilitate Formal Property Specification• First proposed for temporal logic properties [Dwyer et al ICSE 1999]• The templates we present are not temporal and are specific to

data model analysis