Type Checking for Low-Level C Code

Scalable Contract Checking for Systems Software using SMT solversShaz QadeerRiSE, Microsoft Research

Joint work with Jeremy Condit and Shuvendu Lahirihttp://research.microsoft.com/en-us/projects/havoc/1Context: Scalable module verificationTarget: OS components (kernel, drivers, file-systems)~100KLOC of lines of codes with >1000 of proceduresModuleA set of public/entry procedures A set of private/internal proceduresSpecsInterface specificationSpecs for public methodsSpecs for external modulesProperty assertionInitialize(..);

while(*) {choice= nondet();If (choice == 1){ [assume pre_1] call Public_1();} else if (choice == 2){[assume pre_2]call Public_2(); } }Cleanup();HarnessDesirable goalsFind bugs Violations of property assertionsLow false alarmsUse contracts Modular checking for scalabilityReadable contracts are formal documentationReduce testing cost by providing high assurance in the verifierFormal documentation of assumptionsSimple meta-theory for proofsExisting methods on these examplesLarge difference between theory and practice

ImpreciseModeling of lists/arraysUnsoundModeling of lists/arraysAliasing, pointer arithmeticRestricted harnessComplex proof calculusCombination of analysesInitialize(..);

while(*) {choice= nondet();If (choice == 1){ [assume pre_1] call Public_1(); } else if (choice == 2){[assume pre_2]call Public_2(); } }Cleanup();HarnessFull functional correctness is not a goalNeither is minimizing the trusted computing baseProof method: Floyd-Hoare TripleFloyd-Hoare triple {P} S {Q}

P, Q : predicates/propertyS : a program

From a state satisfying P, if S executes, No assertion in S fails, and Terminating executions end in a state satisfying Q

Select(f1,b) = 5 f2 = Store(f1,a,5) Select(f2,a) + Select(f2,b) = 10is valid{ b.f = 5 } a.f = 5 { a.f + b.f = 10 }is valid theory of equality: =theory of arithmetic: 5, 10, +theory of arrays: Select, StoreiffProgram verification Formula [Nelson & Oppen 79]Cite nelsonSatisfiability-Modulo-Theory (SMT)Boolean satisfiability solving + theory reasoningGround theoriesEquality, arithmetic, Select/StoreNP-complete logicsPowerful methods to combine decision procedures for theories[Nelson & Oppen 79]Phenomenal progress in the past few yearsYices, Z3, Mathsat, . Simple type-state propertyAllocation type-state of DEV_OBJDevice Objects (DEV_OBJ) allocated and freed

Property to check for a moduleIoDeleteDevice() only called on elements in MyDevObj~MyDevObjMyDevObjIoCreateDevice()IoDeleteDevice()Simple property simple invariantstypedef struct _DEV_OBJ{DEV_EXT *DevExt; } DEV_OBJ;

typedef struct _DEV_EXT{DEV_OBJ *Self; } DEV_EXT;

requires (do MyDevObj)NT_STATUS PnP(DEV_OBJ do, IRP *pirp){PDEV_EXT data = do->DevExt; .switch(pirp->MajorFn){case IRP_MN_REMOVE_DEVICE: IoDeleteDevice(data->Self); }}DevExtSelfdoDEV_OBJDEV_EXTx MyDevObj. x->DevExt->Self = xSimple property simple invariantsNT_STATUS Unload(){ . iter = hd->First; while(iter != null) {RemoveEntryList(iter);iter = iter->Next;IoDeleteDevice(iter->Self); }.}

FirstNextSelfDevExthdx Btwn(Next, hd->First,NULL). x DevExt x DevExt. x->Self->DevExt = xDEV_OBJDEV_EXTx DevExt. x->Self MyDevObj DEV_OBJNextSelfDevExtDEV_EXTIntuitively define type-states etc.11Limitations of SMT solversNo support for precise reasoning with reachability predicate Incompleteness in Floyd-Hoare proofs for straight line codeBrittle support for quantifiersComplexity: NP-complete (ground) undecidableLeads to unpredictable behavior of verifiersProof times, proof success rateNo support, until recently. How do motivate the concise reasoning part?Limitations of SMT solversAnswer the query {P} S {Q} for loop-free and call-free programsTo handle loops and procedures, contracts are neededLoop invariantsPre/post-conditionsInfeasible to manually supply internal contracts for large modulesContributionsEfficient decision procedure for verifying list-based programs

Verifying and exploiting C type annotations

Annotation inference for large modulesnextfgnextfgnextfgyxBtwn(next,x,y)Reachability predicate: BtwnfExpress properties of collectionsx Btwn(next, next(hd), hd). state(x) = LOCKED //cyclic

Arithmetic reasoning on data (e.g. sortedness)x Btwn(next, hd, null) \ {null}. y Btwn(next, x, null) \ {null}. d(x) d(y)

Expressive logicEfficient decision procedureDecides the validity of {P} S {Q} Worst-case exponential time but works well in practiceDecision problem is NP-complete Cannot expect any better with propositional logicRetains the complexity of current SMT logicsImplemented in the Z3 SMT solverLeverages powerful ground-theory reasoning (arithmetic, arrays, uninterpreted functions)ContributionsEfficient decision procedure for verifying list-based programs

Verifying and exploiting C type annotations

Annotation inference for large modulesC languageC typesScalars (int, long, char, short)Pointers (int*, struct T*, ..)Nested structs and unions Array (struct T a[10];) Function pointersVoid *

Difficult to establish type safety in presence of pointer arithmetic, castsType Safety (spatial) memory safetyImportant default property to check

Lack of types hurts property checkingDifficult to disambiguate heap pointersDifficult to write concise type invariantsIRPIRPFlinkBlinkListEntryFlinkBlinkListEntryExample: Type Checkingpq = CONTAINING_RECORD(p, IRP, ListEntry) = (IRP*)((char*)p - &((IRP*)0->ListEntry))Type Checker: Does variable q have type IRP*?qProperty Checker: Is r->Data1 unchanged?...q->Data2 = 42;IRPIRPFlinkBlinkListEntryExample: Property CheckingqData2Data1FlinkBlinkListEntryData2Data1rExample: Property CheckingFlinkBlinkListEntryData2Data1FlinkBlinkListEntryData2Data2 / Data1For all we know,Data1 and Data2could be aliased!qrOur ApproachImplement a type checker in HAVOCProvide formal semantics for C and its types

Use types to improve the property checkerProvide Java-style field disambiguation

Fully automated using Z3 SMT solverFormalizing Type SafetyA C program is type safe if the run-time value of every variable and heap location corresponds to its compile-time type. Mem : addr -> value Type : addr -> type HasType : value x type -> bool

for all a in addr, HasType(Mem(a), Type(a))Examplerequires( HasType(ENCL(p), record*) && ENCL(p) != NULL)void init_record(list *p) { record *r = CONTAINING_RECORD(p, record, node); r->data2 = 42;}requires( forall(q, Btwn(next, p, NULL), q != NULL ==> HasType(ENCL(q), record*) && ENCL(q) != NULL))void init_all_records(list *p) { while (p != NULL) { init_record(p); p = p->next; }}#define ENCL(x) CONTAINING_RECORD(x, record, node) Decision ProcedureTranslation results in verification conditions that refer to Mem, Type, and HasType

Can be encoded into an NP-complete logicNo worse than SAT solvingProvide decision procedure using an SMT solver

26ExperimentsImplementation supports full C languageSupports polymorphismSupports user-defined, dependent typesFancier type invariants => slower checkingPay only for what you use!Annotated and checked four Windows driversSample drivers provided with Windows DDKAbout 2.3 KLOC total, with 225 annotationsChecking time: ~1 minute each27ContributionsEfficient decision procedure for verifying list-based programs

Verifying and exploiting C type annotations

Annotation inference for large modulesSimple property simple invariantsNT_STATUS Unload(){ . iter = hd->First; while(iter != null) {RemoveEntryList(iter);iter = iter->Next;IoDeleteDevice(iter->Self); }.}

FirstNextSelfDevExthdx Btwn(Next,hd->First,NULL). x DevExt x DevExt. x->Self->DevExt = xDEV_OBJDEV_EXTx DevExt. x->Self MyDevObj DEV_OBJNextSelfDevExtDEV_EXTIntuitively define type-states etc.29Need to simplify the problemModuleA set of public/entry procedures A set of private/internal proceduresSpecsInterface specificationProperty assertion

Require the user to provide a module invariant

Initialize(..);[loop_inv moduleInv]while(*) {choice= nondet();If (choice == 1){ [assume pre_1] call Public_1();} else if (choice == 2){[assume pre_2]call Public_2(); } }Cleanup();HarnessModule invariantsModule invariantsInvariant about all objects of a given typeInvariants on global variablesPreserved by the public functions Low overheadOn steady state and therefore succinctOnly needed to be written at module levelIntra-module inferenceGiven module M, interface specs, property and module invariantsInfer annotations on internal procedures and loopsUse annotations to verify property and module invariantChallengesModule invariants are temporarily brokenInference has to be scalableModule invariant brokenrequires (TypeInvDO)ensures (TypeInvDO)

void publicFoo () { PDEV_OBJ do = NewDEV_OBJ(); privateBar(do);} requires (TypeInvDOExcept(do))requires (TypeInvDO)ensures (TypeInvDO)

void privateBar (PDEV_OBJ do) { do->DevExt->Self = do;}DevExtSelfxDEV_OBJDEV_EXT#define TypeInvDO \ x MyDevObj. x->DevExt->Self = x \Inference takes care of methodoligies.33Houdini algorithm (Flanagan-Leino 01)

Problem statement Given a set of procedures P1, , PnA set of C of candidate annotations for each procedureReturns a subset of the candidate annotations such that each procedure satisfy its annotationsAlso known as monomial predicate abstraction

AlgorithmPerforms a greatest-fixed point starting from all annotationsRemove annotations that are violatedRequires a quadratic (n * |C|) number of theorem prover callsUses a modular checkerCandidate assertionsCandidate assertionsType-states in module invariantsOver parameters, globals, locals and their fieldsModule invariant exceptions (next slide)Conditional annotationsDisjunction of above annotationsModule invariant exceptionsTypeInvDOExcept({do,de->Self},requires))TypeInvDOExcept({do,de->Self},ensures))void privateBar (PDEV_OBJ do, PDEV_EXT de) { do->DevExt->Self = do;}#define TypeInvDO \ x MyDevObj. x->DevExt->Self = x \#define TypeInvDOExcept({a,b},ANNOT) \ANNOT(x MyDevObj. x = a x = b x->DevExt->Self = x)\ANNOT(a->DevExt->Self = a) \ANNOT(b->DevExt->Sefl = b) \Exceptions come from parameters, return, globals, fieldsInference takes care of methodoligies.36ObservationsAble to synthesize most intermediate invariantsclose to the module invariant (simple)readableInvariants contain quantifiers, Boolean structureChecking all Boolean combinations expensive (from NP-Complete PSPACE-complete [CADE09])Retains scalability of the Houdini inference

ExperimentsBenchmarks4 device drivers (~7KLOC each), contains lists, arrays#Internal methods: ~30#loops: ~20Propertiesdouble-free, lock-usageUser provides module invariantTool infers intermediate invariants and modifies clauses

ResultsVerified the properties with 0 false alarmsModule invariant overheadNumber of module invariants ~5-10Reused across multiple driversMost internal annotations inferredApprox 1500 inferred annotation per driverLess than 5 manual annotation per driverMostly conditional annotations (e.g. predicated on return value)Inference time < 5X of the checking timeContributionsEfficient decision procedure for verifying list-based programs

Verifying and exploiting C type annotations

Annotation inference for large modulesQuestions?