Witnessing the Elimination of Magic Wands Stefan Blom and Marieke Huisman University of Twente November 8, 2013 Abstract. This paper discusses the use and verifica- tion of magic wands. Magic wands are used to specify incomplete resources in separation logic, i.e., if missing resources are provided, a magic wand allows one to ex- change these for the completed resources. We show how the magic wand operator is suitable to describe loop in- variants for algorithms that traverse a data structure, such as the imperative version of the tree delete prob- lem (Challenge 3 from the VerifyThis@FM2012 Program Verification Competition). Most separation-logic-based verification tools do not provide support for magic wands, possibly because va- lidity of formulas containing the magic wand is, by it- self, undecidable. To avoid this problem, in our approach the program annotator has to provide a witness for the magic wand, thus circumventing undecidability due to the use of magic wands. We show how this witness in- formation is used to encode a specification with magic wands as a specification without magic wands. Concretely this approach is used in the VerCors tool set: annotated Java programs are encoded as Chalice programs. Chalice then further translates the program to BoogiePL, where appropriate proof obligations are generated. Besides our encoding of magic wands, we also discuss the encoding of other aspects of annotated Java programs into Chalice, and in particular, the encoding of abstract predicates with permission parameters. We illustrate our approach on the tree delete algo- rithm, and on the verification of an iterator of a linked list. 1 Introduction Verification of sequential programs with pointers has sig- nificantly profited from the advance of separation logic. Separation logic is an extension of classical Hoare logic that explicitly considers the heap [39], which makes it highly suitable to reason about pointer structures, the permission to access a heap location, and (absence of) aliases. In classical Hoare logic [18], a program is extended with annotations that express properties about the pro- gram’s intermediate state. Separation logic is an exten- sion of Hoare logic that explicitly separates the program state into the heap and the store. Characteristic for sep- aration logic is that annotations can also express prop- erties about resources, where the most fundamental kind of resource is access permission to a part of the heap, i.e., to read or write a location on the heap. Formulas about access permissions can be combined into larger formulas using the separating conjunction operator ?. A key fea- ture of separation logic is that in a formula φ 1 ?φ 2 , the formula is only valid for a heap, if φ 1 and φ 2 are valid for disjoint parts of the heap. Magic Wands A less common feature of separation logic is the magic wand operator, also known as the separat- ing implication, usually written -- ? . A formula φ 1 -- ? φ 2 holds for a heap h if whenever h is extended with a heap h 1 that satisfies φ 1 , then the combined heap satisfies φ 2 . In the literature, this operator is often described as a trading operator: the resources associated to φ 1 are traded for the resources associated to φ 2 . Another way to think about it is to see the magic wand operator as a promise : it denotes a special kind of resource that pro- vides the ability to exchange one set of resources (the required resources) to a different (possibly larger) set of resources (the ensured resources). This paper shows how the magic wand operator can be used to elegantly specify loop invariants for itera- tive algorithms that explore data structures: knowledge about the current location in the data structure can po- tentially be exchanged for knowledge about the complete brought to you by CORE View metadata, citation and similar papers at core.ac.uk provided by Universiteit Twente Repository
Transcript
Witnessing the Elimination of Magic Wands
Stefan Blom and Marieke Huisman
University of Twente
November 8, 2013
Abstract. This paper discusses the use and verifica-tion of magic wands. Magic wands are used to specifyincomplete resources in separation logic, i.e., if missingresources are provided, a magic wand allows one to ex-change these for the completed resources. We show howthe magic wand operator is suitable to describe loop in-variants for algorithms that traverse a data structure,such as the imperative version of the tree delete prob-lem (Challenge 3 from the VerifyThis@FM2012 ProgramVerification Competition).
Most separation-logic-based verification tools do notprovide support for magic wands, possibly because va-lidity of formulas containing the magic wand is, by it-self, undecidable. To avoid this problem, in our approachthe program annotator has to provide a witness for themagic wand, thus circumventing undecidability due tothe use of magic wands. We show how this witness in-formation is used to encode a specification with magicwands as a specification without magic wands. Concretelythis approach is used in the VerCors tool set: annotatedJava programs are encoded as Chalice programs. Chalicethen further translates the program to BoogiePL, whereappropriate proof obligations are generated. Besides ourencoding of magic wands, we also discuss the encoding ofother aspects of annotated Java programs into Chalice,and in particular, the encoding of abstract predicateswith permission parameters.
We illustrate our approach on the tree delete algo-rithm, and on the verification of an iterator of a linkedlist.
1 Introduction
Verification of sequential programs with pointers has sig-nificantly profited from the advance of separation logic.
Separation logic is an extension of classical Hoare logicthat explicitly considers the heap [39], which makes ithighly suitable to reason about pointer structures, thepermission to access a heap location, and (absence of)aliases.
In classical Hoare logic [18], a program is extendedwith annotations that express properties about the pro-gram’s intermediate state. Separation logic is an exten-sion of Hoare logic that explicitly separates the programstate into the heap and the store. Characteristic for sep-aration logic is that annotations can also express prop-erties about resources, where the most fundamental kindof resource is access permission to a part of the heap, i.e.,to read or write a location on the heap. Formulas aboutaccess permissions can be combined into larger formulasusing the separating conjunction operator ?. A key fea-ture of separation logic is that in a formula φ1 ? φ2, theformula is only valid for a heap, if φ1 and φ2 are validfor disjoint parts of the heap.
Magic Wands A less common feature of separation logicis the magic wand operator, also known as the separat-ing implication, usually written −−? . A formula φ1 −−? φ2
holds for a heap h if whenever h is extended with a heaph1 that satisfies φ1, then the combined heap satisfiesφ2. In the literature, this operator is often described asa trading operator: the resources associated to φ1 aretraded for the resources associated to φ2. Another wayto think about it is to see the magic wand operator as apromise: it denotes a special kind of resource that pro-vides the ability to exchange one set of resources (therequired resources) to a different (possibly larger) set ofresources (the ensured resources).
This paper shows how the magic wand operator canbe used to elegantly specify loop invariants for itera-tive algorithms that explore data structures: knowledgeabout the current location in the data structure can po-tentially be exchanged for knowledge about the complete
brought to you by COREView metadata, citation and similar papers at core.ac.uk
2 Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands
data structure explored so far. We use this approach tospecify a loop invariant for Challenge 3 from the Veri-fyThis@FM2012 Program Verification Competition [22].This challenge is to verify an iterative tree delete algo-rithm: the removal of the element with the least key ina binary search tree. In the literature, also several otherexamples that illustrate the usefulness of the magic wandoperator can be found, e.g., to specify an iterator proto-col (see [17,2] (discussed in Section 6.2), to reason aboutsharing in data structures [19], and to specify severalcommon object-oriented design patterns [27].
In addition, we also discuss how tool support to rea-son about magic wands is developed. There are severaltools that allow reasoning about programs annotated us-ing separation logic, such as VeriFast, SmallFoot, and jS-tar [24,5,12]. However, none of these tools support themagic wand operator. Only the notion of lemma in Veri-Fast [25] resembles a magic wand partially, but it cannotbe used to exchange knowledge about permissions in thesame way as magic wands (see Section 7.1 for more in-formation).
We believe that the main reason that most separation-logic-based program verification tools do not supportreasoning about magic wands, is that deciding the valid-ity of a formula containing magic wands is almost alwaysundecidable [8]. To overcome this, this paper providesthe means to associate a proof term with an assertionintroducing a magic wand formula, by letting the userdescribe how to construct proofs. The proof term is thenused to help the verifier to establish the correctness ofthe implementation w.r.t. the specification. Concretely,a proof annotation syntax is defined that allows one tospecify how to build a proof term for a magic wand. Sup-port for this proof annotation syntax has been integratedinto our VerCors tool set, a tool set for the verificationof concurrent Java programs.
In order to actually construct the term, we intro-duce the notion of a witness object to encode the magicwand formula. The witness object stores the resourcesthat are needed to exchange the required resources ofthe magic wand into the ensured resources. Moreover, italso contains a description how to perform the conver-sion. This witness object is generated based on the proofterm provided by the user. It contains several methodsthat model the logic rules for this particular magic wandformula.
Our solution to the verification problem of the magicwand has been inspired by the Curry-Howard isomor-phism [20], which turns a verification problem into a typechecking problem by constructing a proof term. This in-tuition is further emphasised by the way that we writeannotations: formulas are typically manipulated usinglogical rules, while witnesses are manipulated by meth-ods defined on them. Thus, the encoding of magic wandformulas in this paper transforms the program verifi-cation problem into the programmatic manipulation ofspecification-only (or ghost) objects. We believe that this
approach is attractive for software engineers that havea more imperative way of thinking about program be-haviour, while they struggle with logical manipulationof complex formulas.
Predicates A commonly used extension of separationlogic is the use of abstract predicates [34]. They pro-vide abstraction and allow one to reason about largerstructures, and recursive definitions. Typical use of anabstract predicate is to encapsulate the state of an ob-ject and to hide the internal implementation details fromthe specification. Abstract predicates can be parame-terised by parameters, which can be program variablesor permissions. The latter is useful, for example to spec-ify different access permissions to different parts of adata structure. Abstract predicates behave as resourcesin their own right. In the tree delete challenge, we useabstract predicates to capture for example the state ofthe tree.
To be able to reason about abstract predicates witharbitrary parameters (including permissions), there isready-available tool support, but using a different styleof specifications. In particular, VeriFast supports predi-cates with arguments, but it does not support reasoningabout resources and functional properties separately inorder to reuse existing functional specifications. This dis-agrees with the philosophy of VeriFast. Jost and Sum-mers present an approach to reason about predicatesthat can be split into a resource and a functional predi-cate [26]. However, there are many predicates for whichthis approach does not work, for example when the pred-icate specifies different access permissions to differentparts of a data structure. Therefore, in this paper we alsopropose an encoding of abstract predicates, using thesame idea of constructing a witness object. Concretely,a predicate with parameters is encoded as a witness ob-ject with fields to encode the parameters. This allowsreplacing the original predicate by a predicate over thenewly defined object that refers to the object’s fields,where the original predicate referred to its parameters.Thus, again we trade complexity on the level of logicalformulas for complexity on the level of specification-onlycode.
Approach Concretely, in this paper, we consider speci-fications in permission-based separation logic using themagic wand operator. Permission-based separation logicis an extension of standard separation logic, where frac-tional permissions [7] are used to distinguish read andwrite access permissions to a location on the heap [6,16], which makes it suitable to reason about concurrentprograms.
Permission-based separation logic is used as a spec-ification language for the VerCors tool set [1], whichtargets the verification of concurrent Java programs. Itleverages existing verification tools, and in particularit encodes annotated Java programs into Chalice pro-
Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands 3
grams [29]. Chalice is another verification tool for a con-current programming language, but working on a moreidealised programming language (without inheritance),based on the theory of implicit dynamic frames [40].This is an alternative approach to reason explicitly aboutpointers in the heap, which is equivalent to separationlogic [37].
Notice that all examples in this paper are sequential.Therefore, in most examples all points-to predicates aredecorated with write permissions only.
We use our encoding to verify the tree delete chal-lenge. Concretely, we show that the data structure istreated properly, i.e., only locations for which appropri-ate permissions are held are being accessed. Moreover,we show that the list of elements represented by the re-sult is the tail of the list represented by the input. Inaddition, we also illustrate our approach on a linked listwith an iterator (with remove).
Overview The remainder of this paper is organised asfollows. First, we provide an introduction to separationlogic, and how this is supported in Chalice and the Ver-Cors tool set. Section 3 presents the tree delete challengeof the VerifyThis competition and discusses an intuitivesolution for the challenge that uses a magic wand. Sec-tion 4 focuses on the encoding of predicates with param-eters. We continue in Section 5 with the elimination ofthe magic wand. Then in Section 6, we present machine-checked versions of the challenge and of an additionalexample, namely an iterator protocol. Finally, we con-clude and discuss related and future work.
2 Background
This section first introduces permission-based separationlogic. Then it introduce Chalice as a tool that can checka subset of the logic and finally it explains the basicsof the VerCors tool set, and the syntax for permission-based separation logic in the tool set.
2.1 Permission-based Separation Logic
As mentioned above, separation logic [39] was originallydeveloped as an extension of Hoare logic [18] to rea-son about programs with pointers, as it allows to rea-son explicitly about the heap. In classical Hoare logic,assertions are properties over the state, while in separa-tion logic, the state is explicitly divided in the heap anda store, related to the current method call. Parkinsonadapted separation logic for Java [33], and this is thevariation of the logic that we will use.
Separation logic is also suited to reason modularlyabout concurrent programs [30]: two threads that oper-ate on disjoint parts of the heap, do not interfere, andthus can be verified in isolation. However, classical con-current separation logic requires use of mutual exclusion
mechanisms for all shared locations, and it forbids si-multaneous reads to shared locations. To overcome this,Bornat et al. [6] extended separation logic with fractionalpermissions. Permissions, originally introduced by Boy-land [7], denote access rights to a shared location. Afull permission 1 denotes a write permission, whereasany fraction in the interval (0, 1) denotes a read per-mission. Permissions can be split and combined, thusa write permission can be split into multiple read per-missions, and sufficient read permissions can be mergedinto a write permission. The use of permissions makespermission-based separation logic suitable to prove datarace freedom of multithreaded programs using differentsynchronisation mechanisms.
Most variants of separation logic strictly distinguishlogical values in specifications from the contents of loca-tions on the program heap. This provides a more eleganttheory, but allows less freedom when writing specifica-tions. Therefore, we choose to use the more liberal styleexemplified by Total heap Permission Logic [35].
Formally, we define the set of formulas in this re-source logic as follows:
R ::= b | κ(−→e ) | Perm(e.f , S) | R ? R
| R −−? R | b ⇒ R | *α:T
R(α)
S ∈ (0, 1], κ ∈ Pred
where b is a first-order logic formula, κ is a predicate inset Pred, e denotes an expression (with −→e for a vectorof expressions), α is a logical variable with type T , andπ is a fractional permission, i.e., a value in the inter-val (0, 1]. Expressions can be qualified expressions of theform e.f , where f is a field name. Below we will also givethe grammar of the permission-based separation logic,as supported by the VerCors tool set. All examples willbe given following the VerCors grammar. However, inthe text, we will sometimes also use the more classical,mathematical notation.
Validity of this logic is defined over resources R,which consist of a heap and a partial permission tablefor the locations in this heap. The permission predicatePerm(x.f, π) expresses that the field x.f points to a lo-cation on the heap and the current thread holds a per-mission π on this location. If a thread holds permission1, this denotes a write permission; any permission lessthan 1 denotes a read permission. Permissions can besplit and combined (by division and addition). Whenreasoning about concurrent programs, permissions canbe transferred between threads, and they can be storedas resource invariants into locks. A global correctnessproperty of the logic is that the total number of permis-sions to access a certain location never exceeds 1, whichensures that any program that can be verified is free ofdata races.
The separating conjunction operator ? expresses thata formula φ1 ? φ2 holds for a resource R if the resourcecan be split into two disjoint resources R1 and R2, such
4 Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands
that φ1 holds for R1, and φ2 holds for R2. Separatingconjunction can be lifted over a set of formulas, rangedover by the logical variable α : T , using the universalseparating conjunction, written *
α:TR(α).
The magic wand operator −−? is formally defined asfollows. A resource R satisfies a formula φ1 −−? φ2 if forevery disjoint resourceR′ satisfying the property φ1, thecombination ofR andR′ satisfies φ2. In other words: anyinformation or permissions described by the formula φ1
can be exchanged for the formula φ2. It is important tonote that by obtaining φ2, one has to give up φ1, thuslinear reasoning is used here. We mimic this in our en-coding of magic wands. When creating a witness objectfor a a magic wand formula, we store permissions insidethe witness object, which can be retrieved in exchangefor the permissions or the formula required by the magicwand.
Finally, a predicate κ is a, possibly recursive, defini-tion of a separation logic formula:
κ(−→e ) = φ(−→e ) .
Every predicate must be well-defined, so termination ofthe definition must be guaranteed. An abstract predi-cate is a predicate whose definition is not available tothe client of a class. This is useful to encapsulate de-tails about the state that should not be used by theclient. It is also conveniently used for axiomatizing thebehaviour of low-level data structures and operations,such as volatile variables, locks and thread operations.By declaring a predicate abstract, the client is preventedfrom having access to internal details, these are only vis-ible to the instances of the class itself. This feature isused in the iterator example (see Section 6.2) to enforcethe calling convention for iterators: only by following theprotocol can the client get access to the predicates thatare required for the next call. For the verification of amethod, abstract predicates that are within the scope ofthe receiver object can be opened and closed.
An important restriction on formulas is that theymust be self-framed, which means that the set of loca-tions that must be read to evaluate the formula is a sub-set of the set of locations for which the formula specifiesat least read access.
Finally we introduce a derived notation: when wewant to express both permission and value at the sametime, we use the PointsTo predicate that is used in clas-sical separation logic:
PointsTo(x.f, π, v)def= (Perm(x.f, π) ? x.f = v)
Notice that this relation between Perm and PointsTo
is formally established by Parkinson and Summers [35].The way we write the logic corresponds closely to theirdefinition of Total Heap Permission Logic, which in turnis closely related to the theory of implicit dynamic frames[40] as supported by Chalice.
2.2 Chalice
Chalice is a tool for the verification of concurrent pro-grams [29], written in a simple object-oriented languagewith built-in specification features. The VerCors tool setuses Chalice as an intermediate language to encode an-notated Java programs. Before providing more detailsabout the VerCors tool set, we first briefly introduceChalice.
Supported features of the Chalice language are ba-sic classes (no static members or inheritance), fields andthree kinds of ‘methods’:
– standard methods, which can be used in execution;– functions, i.e., to evaluate a property about the state,
which do not use any access permissions and cantherefore be used both during execution and in spec-ifications; and
– predicates, which can contain access permissions andcan only be used in specifications.
The specification language of Chalice supports fieldpermissions in the same way as the logic in the previ-ous section, albeit with a different syntax (acc, instead ofPerm, and && is used as a connective for permissions, in-stead of ?). Standard boolean expressions and functionscan be used in specifications. Additionally, Chalice hassupport for (recursive) predicates, however these predi-cates cannot have explicit parameters, i.e., they are lim-ited to the implicit parameter this. Both functions andpredicate definition should terminate.
The Chalice tool verifies annotated code by generat-ing an annotated Boogie program [3], for which the Boo-gie verifier subsequently will generate first-order logicproof obligations. Standard methods and functions mapinto Boogie naturally and can make full use of all of theautomatic verification features of Boogie. However, asBoogie is not natively aware of permissions, these areencoded into Boogie’s first-order logic annotations, andtherefore reasoning about them is less automatic. Simi-larly, for predicates reasoning is less automatic, and theuser has to tell the verifier explicitly to replace a pred-icate by its definition (unfold) or introduce a predicatein place of its definition (fold).
As a consequence, in this paper we write explicitlywhen to fold and unfold predicates, while we do not haveto do this when reasoning about functions, as these defi-nitions are opened and closed automatically by the Boo-gie verifier. Instead, to ensure that the verification pro-cess succeeds, we only sometimes have to add a (prov-able) property about a function definition.
2.3 The VerCors Tool Set
The target of the VerCors tool set is the verificationof multi-threaded object-oriented programs, in particu-lar of annotated Java code. Rather than developing yet
Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands 5
Java
back ends
ToolVerCors
Chalice Boogie
input language
SAT/SMT
Common Object Language
Fig. 1. VerCors tool set architecture
another verification condition generator, we leverage ex-isting verifiers. In particular, we use the existing verifiersChalice [29] and Boogie [3]. This means that we will en-code verification problems in the input language as eitherChalice or Boogie programs.
Input for the tool is source code that has been anno-tated with method contracts. These contracts are trans-lated via Chalice and Boogie into the formalism used bya back-end prover. The diagnostic messages provided bythe back-end tool are then (partially) reverse engineeredto provide diagnostic output messages to the user. Eachfailure can optionally be accompanied by the full detailsprovided by the underlying verification engine.
2.3.1 Tool Architecture
The VerCors tool set is built along the classical patternof a compiler. That is, the input programs are parsed intoan abstract syntax tree on which several transformationsare applied before they are passed on to one of the back-ends. The arrows in Fig. 1 indicate the possible pathsa problem can take from input to solver. They reflectthat Chalice works by translating its input into Boogieand Boogie in turn works by generating a problem foran SMT solver, such as Z3 [11]. The direct arrows fromthe intermediate format (Common Object Language) toChalice and Boogie indicate that (depending on the pre-cise verification task) the tool will transform programsinto input programs for Chalice, or for Boogie directly.In this paper, we only consider the encoding via Chalice.
2.3.2 Supported Specification Language
An important goal of the VerCors project is to build auser-friendly tool for verifying realistic Java programs.Therefore, we have derived the concrete syntax for ourseparation logic from the syntax of JML, the de factostandard for writing specifications for Java programs[28]. This will allow us to reuse much of the experienceand research on specification writing, and in addition itwill allow users to extend JML specifications with con-currency details, rather than having to write these spec-ifications from scratch.
The tool considers two classes of expressions: resourceexpressions (R, typical element ri) and logical expres-sions (E, typical elements ei). An important subset ofthose are the logical expressions of type boolean (B, typ-ical elements bi). Using those classes, the syntax for theformulas that is supported by the tool is defined as fol-lows:
E ::= any pure expressionB ::= any pure expression of type boolean
| (\forall T v; b1; b2)| (\exists T v; b1; b2)
where T is an arbitrary type, f a field name, frac a frac-tion of a permission, v a variable name, P a (dynamic)predicate and S a static predicate. Notice the correspon-dence with the mathematical formulation of resourcelogic in Section 2.1.
Note that we include both static predicates, which aredefined independently of object instances, and dynamicpredicates, whose definition is tied in with an instance ofan object (passed as an implicit argument).
The notation for implication ==> is borrowed fromJML, as are the notations for universal and existentialquantification (using keywords \forall and \exists, respec-tively). VerCors uses (\forall∗ T v; b; r) to define the uni-versal separating conjunction.
In addition to Java’s standard primitive types, thespecification language has two additional primitive types:resource and frac, to type permission and fraction ex-pressions, respectively. As in Chalice, the domain of frac
is a value between 1 and 100, where 100 means a fullwrite permission and any value less than 100 denotes aread-only permission. This restriction is made becausewe use Chalice as a back-end, not because the encodingrequires it. In principle the techniques described in thispaper work over any separation algebra [10].
Note that the syntax for resource formulas does notallow disjunction or negation, except for the special caseof a boolean expression acting as a guard for a resourceformula. This restriction does not apply to the booleanexpressions, which can be used as part of a resource for-mula.
It may seem that this contradicts our earlier claim ofextending JML, where disjunction is used heavily in or-der to choose between possible contracts. However, thisdoes not introduce a real contradiction, as any methodwith multiple contracts can be respecified with a singlecontract, using an additional ghost parameter.
Like in JML, all comments that start with a @ arepart of the specification. This holds for both single line(//@ ...) and multiple-line (/∗@ ... ∗/) comments.
Predicates are written in a simple functional syntax,preceded by the keyword resource. For example, a pred-
6 Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands
icate p providing write access to the field f is written asfollows:
/∗@ resource p()=Perm(f,100); ∗/To indicate that a method is pure, as in JML, we add
a pure modifier to the method declaration./∗@ pure ∗/ boolean m(){
2 return true;}In JML, a method is said to be pure when all the
allocated objects on the heap in the prestate are notchanged in the poststate. However, our notion of pureis even stricter: a pure expression cannot have any side-effects.
Alternatively, if the method is used in specificationsonly, the tool also allows to write this in a notation thatresembles the predicate syntax:
/∗@ boolean m()=true; ∗/In method contracts, we will often employ ghost pa-
rameters and ghost return values. These are declared bygiven and yields clauses that precede the method decla-ration. For example, an integer ghost parameter x and aboolean ghost return value b are specified as:
given int x;yields boolean b;
Implicitly, a method contract is universally quantifiedby the variables in the given clause, and existentiallyquantified by the variables in the yields clause.
For specification convenience, the tool also provides apolymorphic list or sequence type seq<T>, where T canbe any type (not necessarily a class). This type trans-lates directly to the Chalice type of the same name. Thesyntax for a constant list borrows from the syntax for aconstant array, e.g., the list [1,2,3] is written as
seq<int>{1,2,3}Several standard operations on sequences are available.Given sequences s,t, we have:
– concatenation: s + t;– first element: head(s);– other elements: tail(s); and– length: s.length.
Further, we provide syntactic sugar for the followingcommon specification pattern:
(e != null) ==> e.pred(e1, · · · ,en)
which we abbreviate as:e−>pred(e1, · · · ,en)
Finally, the tool implements inheritance using thetheory of abstract predicates and inheritance by Bier-man and Parkinson [36], but cannot use the full powerof that theory because specifications are restricted tomonotone predicate families, as introduced by Haackand Hurlin [16,23]. We will not discuss the details ofinheritance, because those are not needed for this paper.
In the examples below, whenever necessary we willexplain more details of the specification syntax.
3 The Tree Delete Challenge
As a motivating example for our work, we use the it-erative removal of the element with the least key froma binary search tree, i.e., Challenge 3 from the Veri-fyThis@FM2012 Program Verification Competition [22]:
Given: a pointer t to the root of a non-empty bi-nary search tree (not necessarily balanced). Verifythat the following procedure [in Fig. 3] removesthe node with the minimal key from the tree. Af-ter removal, the data structure should again be abinary search tree.
Input for the tree delete algorithm is a binary searchtree, in our case defined by the following (recursive) classdefinition:
public class Tree {2 public int data;
public Tree left;4 public Tree right;
//...6 }
The goal of the algorithm is to delete the elementwith the smallest key, i.e., the left-most node from thetree, and the challenge is to prove that the resulting treeremains a binary search tree.
To provide a specification for the algorithm, we firstdefine:
– the predicate state, representing permissions to thefield locations making up the tree;
– the pure function contents, capturing the list of inte-gers stored in the tree; and
– the predicate state contents, expressing the permis-sions and the stored values simultaneously.
resource state() =2 Perm(data,100) ∗∗
Perm(left,100) ∗∗ left −>state() ∗∗4 Perm(right,100) ∗∗ right−>state();
resource state contents(seq<int> L) =18 state() ∗∗ contents(this) == L;
The state predicate defines the permissions on thetree. If one holds the state predicate, one has write per-mission on the fields data, left and right, and recursivelyalso on the subtrees pointed to by left and right, providedthey are not null. The pure method contents defines the
Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands 7
VerCors
public Tree del min(Tree t){2 //@ unfold t.state();
if (t.left==null) {4 //@ assert contents(t.left)
//@ == new seq<int>{};6 return t.right;
} else {8 t.left=del min(t.left);
//@ fold t.state();10 return t;
}12 }
Fig. 2. The recursive implementation of tree delete
contents of a tree to be the contents of the nodes’ datafields, read from left to right.
Using these specifications, the method del min, whichimplements the deletion of the element with the minimalkey is specified as follows.
This contract states that the algorithm removes thefirst element of the list that is represented by a tree. Asthe element with the minimal key in a binary search treeis the first element of the list representation of the tree,this contract is sufficient to meet the challenge. The per-missions used are full write permissions, which impliesthat the underlying linked data structure is tree-shaped,and it cannot be a DAG or contain cycles. The con-tents of the result are the contents of the input, minusthe first element, which implies that the result tree isagain a binary search tree. Note that the result is evenslightly stronger than required. According to the chal-lenge it would have been acceptable to require that thecontents are ordered, but due to the way in which thecontract is written, this information is not needed.
Figure 2 contains a recursive implementation of thisalgorithm. It is easy to see, and to verify, that this im-plementation respects the specification of del min. To il-lustrate this, we have decorated this recursive imple-mentation with the annotations that are needed for theVerCors tool to verify that this implementation respectsthe specification above: essentially all that is needed areopening of the state predicate at the beginning of themethod, closing of the state predicate at the end of themethod body, and an explicit assertion that if t.left isnull then the contents of t.left are the empty list.
In contrast to the recursive version, the verificationof the iterative version of this algorithm, as requested by
the actual challenge description, is much more involved.One needs to specify an appropriate loop invariant thatretains all permissions on the entire tree data structureduring the iterations that compute the left-most nodein the tree. The invariant must be written in such away that the deletion of the left-most node afterwards isallowed and that the permissions on the whole tree canbe recovered.
The core of the problem is the treatment of permis-sions, which are given in the form of a tree. In eachiteration the focus on the tree (i.e. the variable current)is shifted by one step. However, once you have reachedthe left-most node, you want to move back the focus tothe root of the tree, i.e., after the loop has finished, themethod should continue with access to the root of thetree. The magic wand is highly suited to handle this:all the permissions on the traversed path are stored “in-side” the magic wand, and by giving up the focus on thecurrent node, focus on the root can be retrieved. Noticethat our solution with the magic wand is general: it can-not only be used to specify the shift of focus to the rootof the tree, but also to other nodes in the tree.
Fig. 3 contains the iterative implementation of thetree delete algorithm, with the key loop invariants neces-sary to verify this method1. Note that rather than usingthe competition version verbatim, we have eliminated asuperfluous variable and renamed all variables in orderto make the code more understandable. Also note thatwhile our ultimate goal is to further develop the tool setin such a way that this code can be verified essentiallyas it is written in Fig. 3, the current version of our toolset needs a lot more annotations. The fully annotatedversion, which can actually be verified with the VerCorstool set, will be given later in Section 6.1.
The variables in the algorithm denote the following:
– top is the pointer to the root of the complete tree;– cur is the currently explored node; and– left is the left subtree of cur.
The loop invariant expresses the following:
– the thread holds permissions for the currently ex-plored node (state predicate);
– the currently explored node is the root of a tree (state
predicate);– the relationship between cur and left is as described
above; and– a promise (by means of a magic wand) that if cur is
modified to represent a tree with the left-most ele-ment removed, and if the thread holds access per-missions to this tree, then this can be exchanged toaccess permissions on a larger tree, which also hasthe left-most element removed compared to the treeat the start of the procedure.
1 For clarity of presentation, we have left out one bookkeepinginvariant and a lot of proof script.
8 Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands
pseudo VerCors
Tree del min (Tree top) {2 if (top.left == NULL) {
Fig. 3. The iterative implementation of tree delete
With this loop invariant, we can correctly capturesufficient “promises” about the rest of the data struc-ture, in order to verify correctness of the algorithm.
To make this proof amenable to automated tool sup-port, we need an encoding of the magic wand formulausing classes, separating conjunction and the points-topredicate. To be able to do this encoding, the user hasto provide some evidence that the magic wand is in-deed correct, i.e., that the required resources can indeedby exchanged for the promised resources. We introducethis encoding in two steps. First, we introduce the ba-sic idea behind encoding formulas as objects using thewitness encoding of predicates with parameters. Second,we show how magic wands can be encoded in a similarstyle. Finally, we show how we can use this approach tocomplete the verification of the tree delete challenge.
4 The Encoding of Predicates
This section shows how predicates with explicit param-eters, different from the implicit this parameter, are en-coded in Chalice. Below, we first discuss how some pred-icates can be split in a data part and a resource part.However, there are many predicates for which this ap-proach does not work. Therefore, we propose an alterna-tive approach, encoding a predicate with parameters asa witness object. We describe this approach first for sim-ple predicates, and then for recursive predicates. Finally,we describe the encoding algorithm.
4.1 Splittable Predicates
To provide the intuition behind our approach, we firstdiscuss the most simple case, where it is easy to separatea predicate into a permission and a data part.
For example, suppose we have a predicate on a linkedlist expressing that one has write access to the list andthat the list contains a given sequence of elements. Acommon way to define this in separation logic is as fol-lows:
Obviously, since Chalice does not support predicates withparameters, this cannot be directly encoded in Chalice.However, it is straightforward to see that this recursivepredicate can be split into a permission part, a valuepart and a non-recursive predicate, each with the samecontrol flow. Notice that non-recursive predicates can al-ways be inlined, so they do not provide any difficultiesfor verification, however they are very useful for clarityof specifications.
resource state()=2 Perm(val,100) ∗∗
Perm(next,100) ∗∗ next−>state();4
requires l−>state();6 pure boolean contains(List l,seq<int> L)=
Note that after splitting, the resource part of the orig-inal list predicate is written as a recursive predicate state
that no longer has a parameter, and thus it can be en-coded directly in Chalice. The data part of the originallist predicate is captured by the recursive pure methodcontains, which still has a parameter. However, this is noproblem, as this can be encoded directly as a Chalicefunction. Finally, there is a non-recursive predicate list,which has parameters, but those are easy to eliminatein the encoding: simply inline the definition throughoutthe specification.
4.2 Predicate Witnesses
It is not always possible to split a predicate into a per-mission part without parameters and a data part. For
Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands 9
VerCors
class List {2 int val;
List next;4 /∗@
resource state(frac p,frac q)6 = Perm(val,p)
∗∗ Perm(next,q)8 ∗∗ rec:next−>state(p,q);
∗/10 }
Fig. 4. Recursive state predicate with permission parameters
VerCors
public class Twice {2 //@ resource state(frac p)=true;
4 /∗@ given frac p;requires state(p);
6 ensures state(p); ∗/void m(){};
8
/∗@ given frac q;10 requires state(q);
ensures state(q); ∗/12 void twice(){
m();14 m();
}16 }
Fig. 5. Class Twice
example, suppose you wish to define a predicate thatcaptures that you have fraction p to access the elementsstored in the list (via the val pointer), and fraction q totraverse the next pointer to the next element in the list,as is done in Figure 4. For this predicate, the split asdescribed above is impossible. To get around this prob-lem, we introduce the notion of witness. This witness isa carefully encoded object, containing a valid predicate,which holds for the witness object if and only if the orig-inal predicate holds.
4.2.1 Witnesses for Non-Recursive Predicates
To describe how this witness object is constructed andreasoned about, we first consider the encoding of a verysimple predicate, whose body is just true. Class Twice inFig. 5 defines such a predicate, called state.
Fig. 6 shows the definition of the witness object inChalice that encodes the state predicate declared in classTwice. The witness object is an instance of class Twice state.This class definition is generated by the VerCors tool set.The class has two fields: ref refers to the object where theoriginal predicate is defined, and p holds the value of the
Fig. 6. Chalice encoding of the witness class Twice state
parameter p. Further, it defines a predicate valid thatencodes the original state predicate, but using the fieldsof the witness object, instead of the original predicate’sparameters. In addition, the class contains a functioncheck that relates the predicate parameters in the origi-nal specification to the fields of the witness object, andrequires them to be the same. If o is the witness ob-ject for a predicate state, then an assertion this.state(p)
becomes essentially o.valid() ? o.check(this,p) in the en-coding (see for example Lines 4, 5 and 10, 11 in Fig. 8).To complete the description of the class Twice state, italso defines getter methods for all fields in the class.
Using the witness object of the predicate, we wish toshow that class Twice in Figure 5 is correct. For a human,is easy to see that the body of twice satisfies its contract:the pre- and postconditions of m and twice are all thesame, and the calls to m thus can be put in sequence.
To ensure that the tool can establish the correctnessof twice, we need to decorate it with some additionalproof annotations, as shown in Fig. 7. First of all, ev-ery usage of predicate state has been prefixed with a
10 Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands
VerCors
public class Twice {2 //@ resource state(frac p)=true;
label, which refers to a witness (an instance of the classTwice state after encoding). Thus, for example in1 is usedto refer to the witness object for predicate state that ispassed as argument to the call of m, while out1 is usedto refer to the witness object for the predicate returnedby this call. Also the method invocations have been an-notated with a label (syntax: label name), and with ablock to instantiate the ghost parameters of the method(syntax: with block). The label can be used to refer tothe invocation, and to its return values later. Thus forexample, call1 labels the first call to m. The with blockinstantiates the variables declared in the given block forthe method (in this case frac p), and the witness objectassociated with the precondition of this call.
Finally, Fig. 8 shows the encoding of class Twice inChalice. Notice how the state predicate is encoded as acombination of valid and check. The information from thewith block is used to generate parameters for the Chalicemethod calls.
4.2.2 Witnesses for Recursive Predicates
When a predicate is recursively defined, such as the statepredicate on linked lists in Fig. 4, the witness encodingdoes not result in a single object, but in a tree of witnessobjects. For every recursive invocation of the predicatein the predicate definition, the witness contains a fieldthat refers to the witness that provides the evidence forthe recursive call. Thus, the witness for a predicate be-ing valid on an object is actually a tree of witness ob-jects, whose structure matches the calling structure ofthe evaluation of the original predicate. For example, inFig. 9, the linked list of witnesses at the top reflects thatthe definition of rec:state(p,q) applied to the list of three
elements at the bottom makes two recursive calls. Thedefinition of the witness object is given in Fig. 10, wherethe field rec refers to the witness object for the recursivecall of the predicate. Note further how the conditionalpart of the valid predicate in lines 17-19, matches theconditional invocation rec:next−>state(p,q) in the origi-nal predicate definition.
4.3 Recipe for the Encoding
As presented above, the encoding into Chalice generatesa witness class for every predicate definition, replacespredicate invocations in logical statements by validitychecks on witnesses, and adds getter methods for use bywitness classes as well as variables that are needed tostore witness objects.
The complete recipe for the encoding is as follows:
1. Every predicate definitionresource pred(type1 arg1,...,typeN argN) = body ;
declared inside class Class, gives rise to the declara-tion of a sibling of Class, called Class pred, containing:– a field Class ref, to refer to the object for which
the validity of the original predicate is encoded;
Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands 11
Fig. 10. Fragment of the List state witness encoding
– fields type1 arg1, . . . , typeN argN, to store the pa-rameters of the original predicate;
– a predicate valid, whose definition is the separat-ing conjunction of write access to the fields of thepredicate class and the translation of the body ofpred;
– a function check(Class ref,type1 arg1, ..., typeN argN)
that can validate if the reference and parametersmatch; and
– getter methods for all fields.2. In the method specifications and other annotations,
every predicate invocation name:field.pred(args) is re-placed by the following separating conjunction:
This encoding depends on name being defined, there-fore some additional declarations are necessary, de-pending on where the invocation occurred:– in a requires clause: add a parameter
Class pred name to the given clause;– in an ensures clause: add a return value
Class pred name to the yields clause; or– in the body of a predicate definition: add a field
Class pred name to the definition of Class pred andalso add Perm(name,100) to the valid predicate ofthe class Class pred.
3. The original class is modified as follows:– the predicate definition is removed;– every occurrence of
unfold name:obj.pred(expr1,...,exprN)
is replaced by unfold name.valid; and– every occurrence of
fold name:obj.pred(expr1,...,exprN);
is replaced by the block{
name = new Class pred();name.ref = obj;name.arg1 = expr1;...name.argN = exprN;fold name.valid;}
5 The Encoding of Magic Wands
Now that we have seen how predicates with parametersare encoded in Chalice using witness objects, we discusshow magic wands are encoded. The syntax that was usedabove, labeling calls and uses of predicates, and provid-ing explicit instantiations for the ghost arguments, willalso be used in this section. Additionally, we introducenew syntax to create a magic wand and to apply it. Thecreation of a magic wand requires the user to provide aproof script. The proof script language will be discussedbelow.
5.1 General Idea
The general idea behind our encoding is as follows: eachmagic wand formula is encoded as an instance of classWand. Assuming that we have a type that can representformulas, the formulas describing the required and en-sured resources of the magic wand are fields of this class.In addition, the class contains a description of how therequired resources can be exchanged for the ensured re-sources (encoded in the method apply2) and of the extraresources needed to do so. These extra resources are in-haled by the constructor, and not returned. Instead theyare stored inside the magic wand object (in the extra
field), and folded into the valid predicate. The executingthread cannot directly access these extra resources anymore, the only way to retrieve them is by using the apply
method. The specification of the apply method requiresthe required formula of the magic wand and ensures theensured formula, while its body is the exchange descrip-tion. In addition, the apply method also requires thevalid predicate, for two reasons: first, this prevents theapply method from being called more than once; and
second, this allows the apply method to retrieve the ex-tra resources needed for the conversion. Correctness ofthe apply method w.r.t. its specification ensures that theresources stored in the wand, together with the required
2 We use apply as the method name because we are going touse apply as a keyword.
12 Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands
pseudo VerCors
class Wand {2 Formula f1;
Formula f2;4 Formula extra;
Proof p;6
resource valid()=extra();8
requires extra();10 ensures this.f1 == f1 && this.f2 == f2;
ensures this.p == p && this.extra==extra;12 ensures valid();
Wand(Formula f1, Formula f2,14 Formula extra, Proof p)
{16 this.f1 = f1;
this.f2 = f2;18 this.extra = extra;
this.p = p;20 fold valid();
}22
requires valid() ∗∗ f1();24 ensures f2();
void apply() {26 unfold valid();
p();28 }}
Fig. 11. Idealised wand encoding
resources are sufficient to establish the ensured resourcesof the magic wand. Figure 11 shows the definition of thiswand class.
Notice that this is not a valid encoding, as it usestypes such as Formula and Proof, which are not supported.Therefore, below we discuss how the ideas of this ide-alised encoding can be realised by a correct Chalice class.
5.2 Encoding of Magic Wands in Chalice
This section discusses how the idea described above isused to generate a specific class for each type of wandformula that is used. Moreover, the proof script cannotbe passed as a parameter, instead we encode it by anidentifier, and generate a body of apply that selects theappropriate proof script, depending on the value of theidentifier.
Consider for example the WandDemo class in Fig. 12.This class implements a field x, and setter and gettermethods for that field. The contracts are written to en-force a non-standard set/get protocol3, which was in-spired by the iterator protocol of Hurlin and Haack [17].
3 We chose this protocol for illustrative purposes; not becauseof its practical value. However, more meaningful protocols can betreated in a similar way, as shown in Section 6.2.
VerCors
class WandDemo {2 int x;
4 /∗@resource readonly()=Perm(x,25);
6 resource writeonly()=Perm(x,100);@∗/
8
//@ requires readonly();10 /∗@ pure ∗/ int get()=this.x;
Every WandDemo object can be in two states: read modeand write mode. When an instance is created, it is cre-ated in write mode. In write mode, the only method thatcan be called is set. After setting, the object is in readmode. In read mode, the only method that can be calledis get, and the object stays in read mode. However, it isalways possible to reset the object to a different value.
Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands 13
To do so, write mode must be reestablished. Hence, theset method ensures not just the predicate readonly(), butin addition also the magic wand readonly()−∗writeonly(),which can be applied to enable writing. Figure 12 alsodefines a method demo to illustrate how the magic wandis used.
The code is mostly self-explanatory, but three ele-ments are worth noting:
– Every magic wand is given a label, which is used tobe able to identify which magic wand is addressed.In this example, labels recover and wand are used.
– The syntax for creating a magic wand iscreate {
proof scriptqed wand formula;}
To write the proof script, the usual proof hints (fold/unfold/etc.) are allowed, and additionally two newstatements are introduced: use R; and qed R;. Theuse statement asserts that formula R holds at thetime the magic wand is created and stores the re-sources represented by this formula inside the magicwand. For example, the proof in lines 27-32 stores75% of the permission on the field x by means ofthe statement use Perm(this.x,75). The qed statementends the proof script for the magic wand.
– In the demo method, the witness keyword in line 37is used to indicate that the label wand refers to awitness for the corresponding magic wand formula.
Our encoding declares a class for the witness object,called Wand readonly for writeonly, as shown in Fig. 13,to represent the magic wand. Moreover, it rewrites theWandDemo class to replace magic wand formulas by ma-nipulations of the witness object (see Fig. 14).
Instances of Wand readonly for writeonly are used as wit-nesses to the validity of all magic wands that matche1.readonly() −∗ e2.writeonly() for arbitrary expressions e1
and e2. Therefore, it is necessary to indicate which proofis to be used by the apply method, by setting the lemma
field to the correct value. The results of evaluating theexpressions at the time when the wand was created isstored in the fields in 1 and out 1, which have matchinggetter methods. Variables that are used in the proof arealso stored in the object. In this case the value of this forlemma 1 is the only value needed and it is stored in thefield this 1.
The contract of the apply method essentially requiresthe wand to be valid, and the the readonly predicate onin 1 to hold. It ensures the writeonly predicate on out 1.The actual code uses getters and has to deal with non-nullness and temporal issues as well.
The body of the apply method consists of an unfoldstatement, followed by a case distinction over the proofsthat have to be verified. For each proof, the proof hintsare copied into the corresponding branch in the apply
method. In order to get error messages to point to the
intermediate VerCors
class Wand readonly for writeonly2 {
int lemma;4
WandDemo in 1;6 /∗@
requires this.valid();8 ensures true;
@∗/10 WandDemo get in 1()=in 1;
12 WandDemo out 1;/∗@
14 requires this.valid();ensures true;
16 @∗/WandDemo get out 1()=out 1;
18
/∗@20 requires this.valid()∗∗get in 1()!=null
∗∗get in 1().readonly()∗∗get out 1()!=null;22 ensures \old(get out 1()).writeonly();
@∗/24 void apply(){
//@ unfold this.valid();26 if (lemma==1){
//@ unfold this 1.readonly();28 //@ fold this 1.writeonly();
Fig. 13. Encoding of the wand formula in class WandDemo
14 Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands
correct place, an assert statement is added, which in caseof a bad proof will trigger error messages for the specificproof, rather than for the apply method in general.
The valid predicate specifies (write) access to all fieldsin the object and for every proof it has a conditional re-quirement that all of the required permissions and prop-erties are valid. For our example this means that for ex-ample if lemma is 1 then we have 75% permission onthis 1.x and this 1 == in 1 == out 1. Finally, the predi-cate valid also states that lemma must contain a validproof number.
In Java, the logical way of creating new witness ob-jects is to have an overloaded constructor for every proofscript. That is not possible in Chalice, so instead wegenerate a factory method with a unique name for ev-ery proof script. These methods are placed in the classthat calls them. Thus, in our case, the factory methodWand readonly for writeonly lemma 1, which requires the per-missions and properties used in the proof and ensures amagic wand witness, is put into the WandDemo class.
We do not include a full listing of the generated Chal-ice code; however it can be generated using the onlineversion of the VerCors tool set4.
5.3 Correctness of the Encoding
This sections aims to provide an intuition why our en-coding is correct. We will do so based on the view that astatic verifier is a tool that, given a program with specifi-cations establishes the existance of a correctness proof ofthose specifications. For example, for a sequential pro-gram it would establish the existence of a Hoare logicproof. Therefore, we will show that the program beforethe witness transformation can be proven correct if andonly if the program after the witness transformation canbe proven correct.
In order to avoid unnecessary clutter, we will ignorescoping and visibility rules rules for variables, as it iswell-known how to fix these issues. Moreover, we willfocus on the places where magic wands are introducedand eliminated. In proofs, magic wands can also occurin many other places, as they are carried along in proofsand specifications. However, as long as a magic wandis not introduced or eliminated, it is no different fromany other formula and thus irrelevant for showing thecorrectness of the encoding.
The existence of proofs in a proof system is denotedwith the symbol `. That is,
F1, · · · , Fn ` Gdenotes that there is a proof that G logically follows fromF1 ? · · · ? Fn. The classical introduction and eliminationrules for magic wands are the following [17]:
Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands 15
F1 ` G1 −−? G2 F2 ` G1(E −−? )
F1, F2 ` G2
These rules are used in low level proofs to show thatone claim logically follows from the previous one. Wewill show that a proof that uses these two rules can betransformed into a proof that does not use them. Hence,we consider the following two fragments that use therules:
{H ? F}// because F ` G1 −−? G2
{H ? (G1 −−? G2)}
and{H ? G1 ? (G1 −−? G2)}// by application of E −−?{H ? G2}
We show that we can eliminate the magic wand for-mulas from these fragments by transforming them intomagic wand free equivalents. As a first step, we lift theproof script and the application of the magic wand for-mula from the fragments into a separate class Wand:
Wand { // version 12 resource valid()=G1 −−? G2;
requires F ;4 ensures valid();
Wand(){6 {F}
// because F ` G1 −−? G2
8 {G1 −−? G2}fold valid();
10 {valid()}}
12 requires G1 ? valid();ensures G2;
14 void apply(){{G1 ? valid()}
16 unfold valid();{G1 ? (G1 −−? G2)}
18 // by application of E −−?{G2}
20 }}
Using this class the fragments can be rewritten to{H ? F}Wand witness=new Wand();{H ? witness.valid()}
Instead of using a magic wand in the fragments directly,we now have encapsulated the magic wand in the valid
predicate of the ghost class Wand. Note how the resourcesneeded to prove the magic wand (F2) are explicitly re-quired in the contract of the constructor.
The reverse of this first step is inlining, which isknown to be correct, so this first step preserves correct-ness of the annotated program.
All occurrences of the magic wand are now located in-side the ghost class Wand, and in particular in the defini-tion of the predicate valid. As a second step, we eliminatethe magic wand from the definition of valid, by movingthe proof of the magic wand from the constructor to theapply method:
Wand { // version 22 resource valid()=F ;
requires F ;4 ensures valid();
Wand(){6 {F}
fold valid();8 {valid()}
}10 requires G1 ? valid();
ensures G2;12 void apply(){
{G1 ? valid()}14 unfold valid();
{G1 ? F}16 // because F ` G1 −−? G2
{G1 ? (G1 −−? G2)}18 // by application of E −−?
{G2}20 }}Clearly, since the specifications of the constructor
and method apply did not change compared to the en-coding in version 1 of class Wand, any program provencorrect using the first Wand encoding, will also be provencorrect using the Wand version 2 encoding and vice versa.Notice further that the correctness proofs of the con-structor and the method apply in version 2 of class Wand
are correct if and only if they are correct in version 1 ofclass Wand.
Finally, as a last step to get to the encoding that wehave actually presented above, we can use a cut elimina-tion theorem from the underlying proof theory (similarto [32]), and simplify the annotated body of the apply
method as follows:{G1 ? valid()}unfold valid();{G1 ? F}// because F, G1 ` G2
{G2}This removes the last occurrence of the magic wand fromthe annotated code.
We have sketched how a program before the witnessencoding can be proven correct if and only if the programafter the transformation can be proven correct. Thus thewitness transformation for magic wands is both soundand complete.
For this correctness sketch, we have conveniently for-gotten about visibility of fields and methods. However,
16 Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands
requires valid();20 requires P(get in 1(),· · · );
ensures Q(old(get out 1()),· · · );22 void apply(){
unfold valid();24 if (lemma==1) { script ′1 }
...26 if (lemma==L) { script ′L }
}28 }
Fig. 15. Template for a witness class
the actual implementation does take care of those. Sim-ilarly the implementation takes care of multiple proofsby having multiple factory methods instead of a singleconstructor and by using a case distinction in the bodiesof the valid predicate and apply method, to distinguishbetween the different proof resources and proofs.
5.4 Recipe for the Encoding
To conclude, we sketch the complete translation algo-rithm from a program that has been annotated usingmagic wands into a program that is annotated usingwitnesses. This requires to translate all occurrences ofmagic wand formulas and all occurrences of magic wandproof scripts.
The implementation supports magic wand formulaswhere both the required formula and the ensured for-mula are a conjunction of predicate invocations. For thispresentation, we limit ourselves to a single required pred-icate and a single ensured predicate. We can do this with-out loss of generality because any formula can be turnedinto a single predicate formula, where the predicate bodyis the conjunction of the individual formulas and wherethe parameters are the free variables in the formulas.
Thus, we assume magic wands of the form:name : P(−→e ) −∗ Q(
−→f )
For every magic wand formula that uses the samecombination of predicates, we use the same witness class,whose template is given in Fig. 15. First we generate alist of field declarations for the witness class, declaringall the parameters used in the magic wand formula:
– for i = 1 · · · |−→e |: typeof (ei) in i;– for i = 1 · · · |
−→f |: typeof (fi) out i;
where typeof is a meta operator that extracts the typeof an expression.
Further, the witness class defines getters for all of itsfields. This allows us to replace the magic wand formula
name : P(−→e ) −∗ Q(−→f )
with a formula that states that name is a valid witnessof this formula:
name != null ∗∗ name.valid() ∗∗|−→e |
*i=1
name.get in i()==ei ∗∗|−→f |
*j=1
name.get out j()==fj
Note that the quantifiers in this formula have tobe expanded at code generation time because they usemathematical meta-notation that is not part of our syn-tax.
The annotated program will contain several proofscripts to create a witness, all matching the followingtemplate.
create {script ;
qed name : P(−→e ) −∗ Q(−→f );
}Let the total number of proof scripts be L, and let
the scripts be numbered 1, · · · , L.For each script, we compute the list of free variables
used in the script and/or the wand formula−−→free, and
their types−→T , respectively. The list of free variables by
definition contains this. In order to avoid name clashes,we prime all variable names, resulting in the list
−−→free ′.
Each proof script is then replaced with a call to a factorymethod, using an appropriate proof script identifier id togenerate a unique name, as defined in Fig. 16:
name = Wand ... create id(−−→pars,−−→free);
The given proof script script id is used to construct theformula extra ′id and the proof script script ′id, as follows:
– For every occurrence of use φ in script id, a conjunctφ′ is added to extra ′id.
– Every other proof hint is renamed by priming all vari-ables and then added to the proof script script ′id.
When turning magic wands into witnesses, witnessvariables must be declared for each usage of the magicwand formula.
– when used in requires, the variable is declared in thegiven of this method;
Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands 17
VerCors
requires extra ′id2 ensures \result!=null ∗∗ \result.valid();
ensures (\forall∗ f ∈ −−→pars,−−→free ′;
4 \result.get f()==f);
Wand ... create id(−−−−−−→Tpars pars,
−−−−→T free ′){
6 Wand result=new Wand();result.lemma=id;
8 for(f ∈ −−→pars,−−→free ′){
result.f = f;10 }
fold valid();12 return result;}
Fig. 16. Factory method for a witness
– when used in ensures, the variable is declared in theyields clause of this method; and
– when used in witness or loop invariant, the variable isdeclared as a local variable.
Finally, every application of a magic wand
apply name : P(−→e ) −∗ Q(−→f );
is replaced by a call to the apply method of the witness:name.apply();
This completes the description of the encoding as itis implemented.
6 Magic Wand Examples
This section presents two more-involved examples thatuse the magic wand in their annotations. First, we con-sider the tree delete challenge. Second, we consider aniterator protocol for iterators on a list of integers. Forboth examples, we show how to provide full annotations,so that it can be verified by the VerCors tool set.
6.1 Verification of the Tree Delete Challenge
Using the encoding, we can verify the tree delete chal-lenge. Taking the annotated algorithm in Fig. 3 as astarting point, we have to provide proof scripts wheneverwe create a magic wand formula in order to make it ver-ifiable. Since the magic wand formula is used in the loopinvariant (and a new instance of it is needed with everyiteration of the loop), we actually require witnesses forthe creation of magic wand objects in two places in theannotated program: we need a witness to create a magicwand formula to show that the loop invariant holds uponloop initialisation, i.e., before the loop is actually exe-cuted, and in addition we need to provide a witness tocreate a magic wand formula inside the loop body, toshow that every iteration of the loop preserves the loop
ready readyForNext
hasNext()
remove() next()
readyForRemove
new
magic wandmethod call
Fig. 18. State machine for the iterator protocol
invariant. Fig. 17 shows the resulting fully annotatedversion of the tree delete algorithm.
The creation of the the magic wand formula in thedifferent annotations uses different witness proofs (seeLines 25, and 38-48). However, the two different instancesare used both in the loop invariant, which shows that itis essential to have a single apply method that can beused in different proof scripts.
The online version of the VerCors tool set can beused to inspect the full Chalice encoding of this example.Using the Chalice encoding, the VerCors tool can verifythe iterative tree delete algorithm without any problems.The entire example verifies in 6 minutes on an Intel i7-2600 (3.40GHz).
6.2 The Iterator Protocol
As a second example, we present a variant of the iter-ator protocol from Haack and Hurlin [17]. To simplifyour presentation, we have chosen to work with a list ofintegers rather than a list of objects.
The iterator protocol uses the following three states:ready, readyForNext and readyForRemove. The entire proto-col is displayed in Fig. 18. When an iterator is created,permissions on the current list are folded in the ready
state. In this state, one may apply a magic wand to re-cover the permissions on the current list, or one may callhasNext to test for the existence of a next element. If suchan element exists, the next state is readyForNext, other-wise the next state is ready. If the state is readyForNext,the next method can be used to retrieve the currentelement and the next state will be readyForRemove. Inthe readyForRemove state, one can use either the remove
method or the magic wand provided by next to go backto the ready state.
The specifications of the integer list and the list itera-tor can be found in Fig. 19 and 20, respectively. We haveannotated implementations of these interfaces, which havebeen verified with the VerCors tool set. The fully anno-tated versions are available from the same website as thetree delete example.
We have also verified a small example that illustratesthe usage of the list and the iterator, see Fig. 21. In thisexample, we create a list containing [−1, 0, 1] and thenuse an iterator to remove the negative elements. Theentire example verifies in 19 seconds on an Intel i7-2600(3.40GHz).
In this paper, we have introduced two witness trans-formations from Java with separation logic annotationsinto a form that can be checked with Chalice. The firsttransformation replaces predicates with parameters bywitness objects with parameter-free predicates (exceptfor the implicit this parameter). The second transforma-tion replaces magic wands by witness objects. To over-come undecidability when reasoning about magic wands,the user has to provide a proof script that can be used tocheck that resources are indeed correctly exchanged bythe magic wand. Both transformations are not Chalice-specific, i.e., in principle they can be used as an en-coding for any object-oriented language with separationlogic annotations. This is reflected in the implementa-tion, which will remove the witness extensions and re-place them with objects and simpler predicates. In thisprocess unknown expressions and annotation are simplycopied, which means that the current implementationwill work for other similar back-ends without modifica-tion. However, it might require additional or differentproof hints.
In this paper, we have shown that the transforma-tions that we define are sound and complete, i.e., if the
20 Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands
original specification is correct, if and only if the trans-formed specification is correct.
It should be emphasised that all examples in thispaper have been machine-checked, however we feel it istoo early to make any claim about the class of programsthat can be validated with our approach: the fact that aproof exists does not always mean that it can be foundautomatically by a prover.
To illustrate our approach, we have presented twolarger examples: the tree delete example, which demon-strates how magic wands can be used in loop invariants,and the iterator example, which shows how a magic wandis used to enforce that method calls happen in the pre-scribed order. Both examples with full annotations areautomatically verified by the VerCors tool set. In thispaper, we have not presented an example that uses bothwitness transformations at the same time, but such anexample is available online.
7.1 Related Work
The Tree Delete Challenge Our use of the magic wandto verify the iterative implementation of the tree deletealgorithm is just one of the ways of solving the tree deletechallenge. Another way of solving the problem is usingTurk’s rule for loops [41], which effectively offers the pos-sibility of writing loop annotations as if the loop were arecursive function. In other words, the loop body cancontain a block of proof hints, whose application is de-layed, forming a stack of delayed proof steps. When theloop exits, all delayed steps are applied. As these stepscan implicitly set aside resources too, each delayed stepis equivalent to a magic wand. The advantage of usingmagic wand syntax rather than the Turk rule is thatmagic wands can be used in any location in the code,whereas the Turk rule is only applicable to loops. Yetanother mechanism for specifying the iterative versionof the tree delete problem is by using block contracts.The Krakatoa tool [14] implements this generalizationof the Turk rule, which allows attaching pre- and post-conditions to arbitrary statement blocks, instead of justto loop bodies.
The tree delete challenge can also be addressed by us-ing the Zipper data structure from functional program-ming [21]. This is an alternative way to treat the shift offocus: with little it allows to write the invariant in such away that in each iteration the focus is shifted by one step.Once the left-most node is removed, the Zipper structureallows to move the focus back to the root. To use a Zip-per structure, one would have to write the basic datastructure, which requires quite a few lines of specifica-tion, but which can easily be automated. For those caseswhere the pre-defined functions work, the annotationwork load would then be limited to a few instructionsthat move the focus. However, if a non-standard moveis required, it would have to be spelled out completely
at a considerable specification effort, because only pre-defined moves would be automatically generated.
Tool Support The VeriFast tool implements lemmas [25],which are equivalent to a subset of magic wands. That is,lemmas can transform sets of resources, just like magicwand can. But magic wands can exchange resources aswell, for example, a magic wand can express the capa-bility to exchange a read-only permission for accessinga location on the heap to a full write permission for thesame location. If one were to express this using a lemma,the lemma would have to be paired with a predicate thatcaptures the extra permissions in order to match thefunctionality of the magic wand.
Jost and Summers have written a tool that translatesVeriFast specification to Chalice [26]. They analyse Ver-iFast predicates to see if they can be split into a param-eterless permission predicate and a boolean function. Ifsuch a split is impossible they resort to using ghost vari-ables in the object to represent the predicate parame-ters. The advantage of such an approach is that it leadsto a simpler encoding that requires less annotations tobe verified. The disadvantage is that the encoding doesnot work if the same predicate must be held twice onthe same object, whereas our encoding supports this.
Automated proving for special cases of the magicwand is being worked on. For example, in the settingof the logic of boolean bunched implications (a close rel-ative of separation logic) Park et al. have given an algo-rithm for deciding the magic wand [32]. Further, Atkeyhas shown that for a restricted magic wand syntax it ispossible to directly derive verification conditions [2].
Finally, the transformation that turns magic wandsinto objects resembles defunctionalization of closures ina functional programming language (see for example [38]).
The specification style of the generated object, defin-ing a valid predicate is inspired by the standard method-ology for Boogie of Barnett et al. [4].
7.2 Future Work
Annotation Generation Clearly, the major drawback ofour approach is the large number of (long) annotationsthat the user has to provide at the moment. To addressthis issue, we will first of all investigate heuristics tocome up with good default specifications. We will alsoinvestigate if methods for automatically deriving speci-fications can be adapted to our situation. For example,it might be possible to use the constraint solving algo-rithm developed by Ferrara and Muller [13] to infer alarge number of the witness management annotations.
Additionally, as mentioned above, the approach takenby Jost and Summers in their translation of VeriFast toChalice can deal with a number of cases with less anno-tation load. We will study if we can integrate their work,so that the simple cases can take advantage of their en-
Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands 21
coding, while the encoding of this paper can be used formore complicated cases.
The current implementation of the witness transfor-mation for magic wands requires that all the resourcesthat are stored inside the magic wand are explicitly men-tioned using the use keyword. The problem of finding outwhich resources have to be stored is similar to the prob-lem of finding out which resources to pass to a methodduring a call and which resources to keep. Therefore, wewill study the techniques for solving this, such as the useof frame inference and bi-abduction [9], and see if theycan be reused and/or adapted to our approach.
Extensions We believe the witness encoding approachcan be extended to basic permissions on fields too. Suchan extension replaces the permissions-as-logic-formulasview completely by a permissions-as-a-data-type view.This would make any check on permissions executable,enabling the re-use of existing run-time checkers for run-time checking of permissions. It might even be possibleto reuse existing static verifiers that do not support per-missions, however there is a risk that this would evenfurther increase the annotation workload, which wouldmake this approach practically unfeasible.
We are currently investigating how permission-basedseparation logic can be used to reason about Scala pro-grams. This requires the possibility to specify the be-haviour of closures. We are investigating if we can extendour approach also to encode closure specifications.
We also plan to further investigate if temporal as-pects have an impact on the encoding. In the currentencoding, everything in the footprint of the magic wandis evaluated in the current state, while everything else isevaluated against the state in which the magic wand iseventually applied. We have to investigate further if thiscauses any problems.
Acknowledgement
The work presented in this paper is supported by ERCgrant 258405 for the VerCors project.
References
1. A. Amighi, S. Blom, M. Huisman, and M. Zaharieva-Stojanovski. The VerCors project: Setting up basecamp.In Programming Languages meets Program Verification(PLPV 2012), pages 71–82. ACM Press, 2012.
2. R. Atkey. Amortised resource analysis with separationlogic. In A. D. Gordon, editor, ESOP, volume 6012of Lecture Notes in Computer Science, pages 85–103.Springer, 2010.
3. M. Barnett, B.-Y. E. Chang, R. DeLine, B. Jacobs, andK. R. M. Leino. Boogie: A modular reusable verifier forobject-oriented programs. In Formal Methods for Com-ponents and Objects, volume 4111 of LNCS, pages 364 –387. Springer-Verlag, 2005.
4. M. Barnett, R. DeLine, M. Fahndrich, K. R. M. Leino,and W. Schulte. Verification of object-oriented programswith invariants. Journal of Object Technology, 3(6):27–56, 2004.
5. J. Berdine, C. Calcagno, and P.W. O’Hearn. Smallfoot:Modular automatic assertion checking with separationlogic. In F.S. de Boer, M.M. Bonsangue, S. Graf, andW.P. de Roever, editors, FMCO, volume 4111 of LectureNotes in Computer Science, pages 115–137. Springer,2005.
6. R. Bornat, C. Calcagno, P.W. O’Hearn, and M.J. Parkin-son. Permission accounting in separation logic. In Pals-berg and Abadi [31], pages 259–270.
7. J. Boyland. Checking interference with fractional per-missions. In R. Cousot, editor, Static Analysis Sym-posium, volume 2694 of LNCS, pages 55–72. Springer-Verlag, 2003.
8. R. Brochenin, S. Demri, and E. Lozes. On the almightywand. Inf. Comput., 211:106–137, 2012.
9. C. Calcagno, D. Distefano, P.W. O’Hearn, and H. Yang.Compositional shape analysis by means of bi-abduction.J. ACM, 58(6):26, 2011.
10. C. Calcagno, P. W. O’Hearn, and H. Yang. Local actionand abstract separation logic. In LICS, pages 366–378.IEEE Computer Society, 2007.
11. L. Mendonca de Moura and N. Bjørner. Z3: An effi-cient SMT solver. In C.R. Ramakrishnan and J. Rehof,editors, TACAS, volume 4963 of LNCS, pages 337–340.Springer-Verlag, 2008.
12. D. DiStefano and M. Parkinson. jStar: Towards practi-cal verification for Java. In ACM Conference on Object-Oriented Programming Systems, Languages, and Appli-cations, pages 213–226. ACM Press, 2008.
13. P. Ferrara and P. Muller. Automatic inference of accesspermissions. In Proceedings of the 13th InternationalConference on Verification, Model Checking, and Ab-stract Interpretation (VMCAI 2012), LNCS, pages 202–218. Springer-Verlag, 2012.
14. J.-C. Filliatre and C. Marche. The Why/Krakatoa/Ca-duceus platform for deductive program verification. InW. Damm and H. Hermanns, editors, CAV, volume 4590of Lecture Notes in Computer Science, pages 173–177.Springer, 2007.
15. Roberto Giacobazzi and Radhia Cousot, editors. The40th Annual ACM SIGPLAN-SIGACT Symposium onPrinciples of Programming Languages, POPL ’13, Rome,Italy - January 23 - 25, 2013. ACM, 2013.
16. C. Haack and C. Hurlin. Separation logic contracts fora Java-like language with fork/join. In J. Meseguerand G. Rosu, editors, Algebraic Methodology and Soft-ware Technology, volume 5140 of LNCS, pages 199–215.Springer-Verlag, 2008.
17. C. Haack and C. Hurlin. Resource Usage Protocols for It-erators. Journal of Object Technology, 8(4):55–83, 2009.
18. C.A.R. Hoare. An axiomatic basis for computer pro-gramming. Communications of the ACM, 12(10):576–580, 1969.
19. A. Hobor and J. Villard. The ramifications of sharingin data structures. In Giacobazzi and Cousot [15], pages523–536.
20. W. Howard. The formulae-as-types notion of construc-tion, pages 479 – 490. ACM, 1980. Original papermanuscript from 1969.
22 Stefan Blom and Marieke Huisman: Witnessing the Elimination of Magic Wands
21. G. P. Huet. The zipper. J. Funct. Program., 7(5):549–554, 1997.
22. M. Huisman, V. Klebanov, and R. Monahan. Veri-fyThis verification competition 2012 – organizer’s re-port. Technical Report 2013-01, Department of In-formatics, Karlsruhe Institute of Technology, 2013.Available at http://digbib.ubka.uni-karlsruhe.de/
volltexte/1000034373.23. C. Hurlin. Specification and Verification of Multithreaded
24. B. Jacobs and F. Piessens. The VeriFast program ver-ifier. Technical Report CW520, Katholieke UniversiteitLeuven, 2008.
25. B. Jacobs, J. Smans, and F. Piessens. VeriFast: Imper-ative programs as proofs. In VSTTE workshop on Tools& Experiments, August 2010.
26. D. Jost and A. J. Summers. An automatic encodingfrom VeriFast predicates into Implicit Dynamic Frames.In Verified Software: Theories, Tools and Experiments(VSTTE), 2013.
27. N. R. Krishnaswami, J. Aldrich, L. Birkedal, K. Svend-sen, and A. Buisse. Design patterns in separation logic.In Andrew Kennedy and Amal Ahmed, editors, TLDI,pages 105–116. ACM, 2009.
28. G.T. Leavens, E. Poll, C. Clifton, Y. Cheon, C. Ruby,D. R. Cok, P. Muller, J. Kiniry, and P. Chalin. JMLReference Manual, February 2007. Dept. of ComputerScience, Iowa State University. Available from http://
www.jmlspecs.org.29. K.R.M. Leino, P. Muller, and J. Smans. Verification of
concurrent programs with Chalice. In Lecture notes ofFOSAD, volume 5705 of LNCS, pages 195–222. Springer-Verlag, 2009.
30. P. W. O’Hearn. Resources, concurrency and local rea-soning. Theoretical Computer Science, 375(1–3):271–307,2007.
31. J. Palsberg and M. Abadi, editors. Proceedings of the32nd ACM SIGPLAN-SIGACT Symposium on Prin-ciples of Programming Languages, POPL 2005, LongBeach, California, USA, January 12-14, 2005. ACM,2005.
32. J. Park, J. Seo, and S. Park. A theorem prover forboolean BI. In Giacobazzi and Cousot [15], pages 219–232.
33. M. Parkinson. Local reasoning for Java. Technical Re-port UCAM-CL-TR-654, University of Cambridge, 2005.
34. M. J. Parkinson and G. M. Bierman. Separation logicand abstraction. In Palsberg and Abadi [31], pages 247–258.
35. M. J. Parkinson and A. J. Summers. The relationship be-tween separation logic and implicit dynamic frames. Log-ical Methods in Computer Science, 8(3:01):1–54, 2012.
36. M.J. Parkinson and G.M. Bierman. Separation logic, ab-straction and inheritance. In G.C. Necula and P. Wadler,editors, POPL, pages 75–86. ACM, 2008.
37. M.J. Parkinson and A.J. Summers. The relationship be-tween separation logic and implicit dynamic frames. InG. Barthe, editor, ESOP, volume 6602 of LNCS, pages439–458. Springer-Verlag, 2011.
38. J. C. Reynolds. Definitional interpreters for higher-order programming languages. Higher-Order and Sym-bolic Computation, 11(4):363–397, 1998.
39. J.C. Reynolds. Separation logic: A logic for shared muta-ble data structures. In Logic in Computer Science, pages55–74. IEEE Computer Society, 2002.
40. J. Smans, B. Jacobs, and F. Piessens. Implicit dynamicframes. ACM Trans. Program. Lang. Syst., 34(1):2, 2012.
41. T. Turk. Local reasoning about while-loops. In R. Joshi,T. Margaria, P. Muller, D. Naumann, and H. Yang, edi-tors, VSTTE 2010. Workshop Proceedings, pages 29–39.ETH Zurich, 2010.
