Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Apply Program Model CheckingApply Program Model Checking
Qian LingQian Ling
Oct. 28, 2004Oct. 28, 2004
Bell Labs Research ChinaBell Labs Research China
2
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
AgendaAgenda
• MotivationMotivation• Program Model CheckingProgram Model Checking• Program Model Checking ToolsProgram Model Checking Tools
– The Spin ToolThe Spin Tool– The Verisoft ToolThe Verisoft Tool– The SLAM ToolThe SLAM Tool
• Use Spin tool to check TTS ServerUse Spin tool to check TTS Server• Future WorkFuture Work
3
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
MotivationMotivation
• Testing of concurrent programsTesting of concurrent programs– VMS TTS Server ProgramVMS TTS Server Program– OS Adapter ProjectOS Adapter Project– POSIX thread impl. on VxWorks 5.4POSIX thread impl. on VxWorks 5.4
• Progress in Program Model CheckingProgress in Program Model Checking– New tech and tools developedNew tech and tools developed
4
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
AgendaAgenda
• MotivationMotivation• Program Model CheckingProgram Model Checking• Program Model Checking ToolsProgram Model Checking Tools
– The Spin ToolThe Spin Tool– The Verisoft ToolThe Verisoft Tool– The SLAM ToolThe SLAM Tool
• Use Spin tool to check TTS ServerUse Spin tool to check TTS Server• Future WorkFuture Work
5
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
SW Quality Control TechSW Quality Control Tech
• Software TestingSoftware Testing– Unit/integration/system testUnit/integration/system test– Functional test, Conformance testFunctional test, Conformance test– Stress testStress test– API test, etcAPI test, etc
• Code ReviewCode Review– Peer ReviewPeer Review– Design ReviewDesign Review
• Program Model CheckingProgram Model Checking– Formal MethodFormal Method– Software VerificationSoftware Verification
• Software Quality AssuranceSoftware Quality Assurance
6
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Software TestingSoftware Testing
Unit A
{.......}
Unit B
{........}
Unit C
{...........}Unit TestIntegration
Test
System Test“Sequential, Stand-alone”
7
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Code ReviewCode Review
do{ if (select(FD_SETSIZE, &read_fds, NULL, NULL, &TimeOut) != -1) { if (FD_ISSET(sock, &read_fds)) { amount = read(sock, buf, TTS_RECEIVE_BUFFER_LENGTH);
..............
else { /*Timeout waiting for socket*/ iRet = TTS_ERROR_TIMEOUT; break; }
} while ((amount >= 0) && (!bAllDone));
>
“Static”
8
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Facing Concurrent & Facing Concurrent & Distributed SoftwareDistributed Software
It is not too surprising that thebehavior of even non-buggy distributed applications can easilydefy our human reasoning skills.
Gerard J. Holzmann
9
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Facing Concurrent & Facing Concurrent & Distributed Software, Distributed Software, contcont
• DeadlockDeadlock• Livelock, starvationLivelock, starvation• UnderspecificationUnderspecification
– unexpected reception of messagesunexpected reception of messages
• OverspecificationOverspecification– Dead codeDead code
• Violations of constraintsViolations of constraints– Buffer overrunsBuffer overruns– Array bounds violationsArray bounds violations
• Assumptions about speedAssumptions about speed– Logical correctness vs.real-time performanceLogical correctness vs.real-time performance
Really Hard IssueReally Hard Issue
10
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
What is Model Checking?What is Model Checking?
• [Clarke & Emerson 1981][Clarke & Emerson 1981] “ “Model checking is an automated technique that, given a Model checking is an automated technique that, given a
finite-state model of a system and a logical property, finite-state model of a system and a logical property, systematically checks whether this property holds for (a systematically checks whether this property holds for (a given initial state in) that model.”given initial state in) that model.”
• Model checking tools automatically Model checking tools automatically verify whether verify whether M |= φ M |= φ HoldsHolds
where M is a (finite-state) model of a system and where M is a (finite-state) model of a system and property φ is stated in some formal notationproperty φ is stated in some formal notation
• Problem: Problem: state space explosionstate space explosion• SPINSPIN is one of the most powerful model is one of the most powerful model
checkerchecker
11
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Classic v.s. Modern MCClassic v.s. Modern MC
WaterfallSW Lifecycle
Process
12
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Classic Model CheckingClassic Model Checking
13
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Modern Model CheckingModern Model Checking
• Abstraction: To fight with explosionAbstraction: To fight with explosion• Focus on Modern Model CheckingFocus on Modern Model Checking
14
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
AgendaAgenda
• MotivationMotivation• Program Model CheckingProgram Model Checking• Program Model Checking ToolsProgram Model Checking Tools
– The Spin ToolThe Spin Tool– The Verisoft ToolThe Verisoft Tool– The SLAM ToolThe SLAM Tool
• Use Spin tool to check TTS ServerUse Spin tool to check TTS Server• Future WorkFuture Work
15
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Tools for program model Tools for program model CheckingChecking
• SPINSPIN– Gerard J. Holzmann (Bell-Labs), 1991~ 1991~– Proving the correctness of process interaction, Proving the correctness of process interaction,
asynchronized controlasynchronized control
• VerisoftVerisoft– Patrice Godefroid (Bell Labs), 1996~Patrice Godefroid (Bell Labs), 1996~– Runtime analyses for multi-process programsRuntime analyses for multi-process programs
• SLAMSLAM– Thomas Ball (MS Research), 1999~Thomas Ball (MS Research), 1999~– Static analyses for “API Usage Rules” in programsStatic analyses for “API Usage Rules” in programs
16
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Tools for program model Tools for program model CheckingChecking, cont, cont
• EraserEraser– Stefan Savage (University of Washington) & DEC, 1997~Stefan Savage (University of Washington) & DEC, 1997~– Dynamic detecting data-race in lock-based multithread Dynamic detecting data-race in lock-based multithread
programsprograms
• FLAVERSFLAVERS– Matthew B. Dwyer, L. A. Clarke (University of Kansas Matthew B. Dwyer, L. A. Clarke (University of Kansas
State/Massachusetts Amherst) &DoD, DAAH, NSF, State/Massachusetts Amherst) &DoD, DAAH, NSF, IBM… , 1999~IBM… , 1999~
– Check if all executions adhere to specification, data flow Check if all executions adhere to specification, data flow analysis, stackanalysis, stack
• SMVSMV– CMU, 1992~CMU, 1992~– Symbolic model checkerSymbolic model checker
• ……
17
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Tool ComparisonTool Comparison
• Modeling LanguagesModeling Languages• Static Analyze v.s. DynamicStatic Analyze v.s. Dynamic• Explicit State space?Explicit State space?• Scope of verificationScope of verification
– Safety onlySafety only– Acceptation cycle, etcAcceptation cycle, etc
• Stages in developmentStages in development– Design phaseDesign phase– Code ReadyCode Ready– Incremental developmentIncremental development
18
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
AgendaAgenda
• MotivationMotivation• Program Model CheckingProgram Model Checking• Program Model Checking ToolsProgram Model Checking Tools
– The Spin Tool The Spin Tool – The Verisoft ToolThe Verisoft Tool– The SLAM ToolThe SLAM Tool
• Use Spin tool to check TTS ServerUse Spin tool to check TTS Server• Future WorkFuture Work
19
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
The SPIN Model CheckerThe SPIN Model Checker
• SPIN = SPIN = SSimple imple PPromela romela InInterpreterterpreter– is a tool for analysing the logical is a tool for analysing the logical
consistency of concurrent systems, consistency of concurrent systems, specifically of data communication specifically of data communication protocols.protocols.
– Concurrent systems are described in the Concurrent systems are described in the modelling language called Promela.modelling language called Promela.
– state-of-the-art model checkerstate-of-the-art model checker
20
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
PromelaPromela
• Promela Promela = = ProProtocol/tocol/ProProcess cess MeMeta ta LaLanguagenguage– allows for the dynamic creation of concurrent allows for the dynamic creation of concurrent
processes.processes.– communication via message channels can be communication via message channels can be
defined to bedefined to be• synchronous (i.e. rendezvous), orsynchronous (i.e. rendezvous), or• asynchronous (i.e. buffered).asynchronous (i.e. buffered).
– resembles the programming language Cresembles the programming language C– specification language to model finite-state specification language to model finite-state
systemssystems
21
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Modern Model Checking with Modern Model Checking with SPINSPIN
Program Source Code(in C/C++,Java, LISP...)
PREMELA Model
Program Errors
Mapping,Abstraction
Verification
Specification
LTL(Linear Temporal Logic)
Formulate
22
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Foundation of SPINFoundation of SPIN
• Based on Based on automata theoretic modelautomata theoretic model– System ModelSystem Model
• Translate process tmpl to automatonTranslate process tmpl to automaton• State space=product of all automataState space=product of all automataGlobal behavior modeled by an automatonGlobal behavior modeled by an automaton
– SpecificationSpecification• Correctness requirement in LTL formulaCorrectness requirement in LTL formula• Convert LTL to Büchi automatonConvert LTL to Büchi automaton
– VerificationVerification• global automatonglobal automaton× × Büchi automatonBüchi automaton
– (Accepted language = empty) (Accepted language = empty) not satisfiednot satisfied– (Accepted language ‡ empty) (Accepted language ‡ empty) satisfied behaviorssatisfied behaviors
• SPIN formalize the erroneous behaviorsSPIN formalize the erroneous behaviors– Prove such behavior are impossibleProve such behavior are impossible– Provide detail matched behaviorsProvide detail matched behaviors
Vardi & Wolper 83Vardi & Wolper 83
23
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Algorithms of SPINAlgorithms of SPIN
• Nested Depth-First Search Nested Depth-First Search (Cycles detection)(Cycles detection)– 11stst search: from initial state, find accepting states search: from initial state, find accepting states– 22ndnd search (nested): from all accepting states search (nested): from all accepting states
• LTL to Büchi automatonLTL to Büchi automaton– Normalize LTL formulasNormalize LTL formulasrecursive conversionrecursive conversion– Make use of Promela condition statement Make use of Promela condition statement
• Partial Order ReductionPartial Order Reduction– Represent classes of execution sequence with one single Represent classes of execution sequence with one single
path, staticallypath, statically• Memory ManagementMemory Management
– State Compression State Compression (60%~80% reduction)(60%~80% reduction)• Separate local state (small) and global stateSeparate local state (small) and global state• Global state use index to refer to local stateGlobal state use index to refer to local state
– Bit-State Hashing Bit-State Hashing (Improve problem coverage (Improve problem coverage 1)1)• 2 bits are used to store a reachable state2 bits are used to store a reachable state• 2 bits: F1(S1) = M[m1], F2(S2)=M[m2]2 bits: F1(S1) = M[m1], F2(S2)=M[m2]• Collision problemCollision problem
Problem coverage = M / (N*S)
M=2^m: memory space in bytes
N=2^n: true # of Reachable states
S=2^s: bytes for each state
24
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Debug with SPINDebug with SPIN
• Correctness Requirement:Correctness Requirement:– Inline assertionsInline assertions
• assertassert(…)(…)– End-state label / deadlockEnd-state label / deadlock
• endendxxxxxx::– Progress-state label / livelockProgress-state label / livelock
• progressprogressxxxxxx::– Acceptance-state labelAcceptance-state label
• acceptacceptxxxxxx::– LTL propertiesLTL properties
• Make use of LTL specificationMake use of LTL specification– Never-claimsNever-claims
• never {…}never {…}
Commonly Used
Commonly Used
25
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Extensions to SPIN (1)Extensions to SPIN (1)
• dSpin = dSpin = ddynamic ynamic SPINSPIN– R. Iosif & R. Sisto, Kanas State University, 1999~R. Iosif & R. Sisto, Kanas State University, 1999~– Solve Limitations in SPIN, Solve Limitations in SPIN,
• Object creation & deletionObject creation & deletion• Pointer and reference variablesPointer and reference variables• Polymorphic function callsPolymorphic function calls
– SolutionsSolutions• Memory managementMemory management
– Dynamic memory allocationDynamic memory allocation– Reference mechanismReference mechanism
• Functional managementFunctional management– Function declaration, call and referenceFunction declaration, call and reference– Local scopingLocal scoping
26
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Extensions to SPIN (2)Extensions to SPIN (2)
• FeaVer/Modex = FeaVer/Modex = ModModel el ExExtractor for Ctractor for C– Gerard J. Holzmann, Bell Labs, 1998~Gerard J. Holzmann, Bell Labs, 1998~– Extract Promela model from ANSI CExtract Promela model from ANSI C– 3 phases:3 phases:
• Parsing: Parsing: C C Full Parse Tree Full Parse Tree• Interpretation: Interpretation: Tabled AbstractionTabled Abstraction• OptimizationOptimization
– Applications:Applications:• Lucent PathStar™ Access ServerLucent PathStar™ Access Server
27
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Extensions to SPIN (3)Extensions to SPIN (3)
• JPF1/JPF2 = JPF1/JPF2 = JJava ava PPath ath FFinderinder– Automated Software Engineering Group, NASAAutomated Software Engineering Group, NASA– JPF1, since 1998JPF1, since 1998
• A translator from Java 1.0 to PromelaA translator from Java 1.0 to Promela• Suitable for multi-threaded Java programsSuitable for multi-threaded Java programs• ApplicationsApplications
– Find deadlocks in Game ServerFind deadlocks in Game Server– Analyze Remote AgentAnalyze Remote Agent
– JPF2, since 2000JPF2, since 2000• Work on Java Byte Code, 100% Java supportWork on Java Byte Code, 100% Java support• Garbage CollectionGarbage Collection• Static analysis dependencies for partial-order reductionStatic analysis dependencies for partial-order reduction• Contain runtime race-cond analysis by “Eraser” AlgorithmContain runtime race-cond analysis by “Eraser” Algorithm• Use Bandera to calculate slicesUse Bandera to calculate slices
28
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Extensions to SPIN (4)Extensions to SPIN (4)
• Bandera ToolsetBandera Toolset– Santos Laboratory, since 2000Santos Laboratory, since 2000– source code source code FSM model checker, SPIN/SMV FSM model checker, SPIN/SMV
• Analyze, Analyze, • Abstraction Abstraction • TransformationTransformation
– More than just a SPIN extension!More than just a SPIN extension!
29
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
AgendaAgenda
• MotivationMotivation• Program Model CheckingProgram Model Checking• Program Model Checking ToolsProgram Model Checking Tools
– The Spin Tool The Spin Tool – The Verisoft ToolThe Verisoft Tool– The SLAM ToolThe SLAM Tool
• Use Spin tool to check TTS ServerUse Spin tool to check TTS Server• Future WorkFuture Work
30
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
VeriSoftVeriSoft
• A tool for systematically testing A tool for systematically testing concurrent reactive softwareconcurrent reactive software
• Stateless search, for:Stateless search, for:– DeadlocksDeadlocks– LivelocksLivelocks– Assertion violationsAssertion violations– DivergenceDivergence
• Replay Error ScenariosReplay Error Scenarios
• Applications:Applications:– 4ESS Heart-Beat Monitor 4ESS Heart-Beat Monitor
debugging and unit debugging and unit testingtesting
– WaveStar 40G R4 WaveStar 40G R4 integration and system integration and system testingtesting
– 7R/E PacketStar Gateway 7R/E PacketStar Gateway Solutions Feature Server Solutions Feature Server unit testingunit testing
– CDMA Call Processing CDMA Call Processing Library testingLibrary testing
VeriSoft
BA C
System Processes
deadlock
VeriSoft
BA C
System Processes
VeriSoft
BA C
System Processes
deadlockdeadlock
31
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
AgendaAgenda
• MotivationMotivation• Program Model CheckingProgram Model Checking• Program Model Checking ToolsProgram Model Checking Tools
– The Spin Tool The Spin Tool – The Verisoft ToolThe Verisoft Tool– The SLAM ToolThe SLAM Tool
• Use Spin tool to check TTS ServerUse Spin tool to check TTS Server• Future WorkFuture Work
32
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
SLAMSLAM
• Check Temporal safety property in sequential Check Temporal safety property in sequential C programsC programs
• Based on Boolean ProgramBased on Boolean Program• Solutions:Solutions:
– Preparation:Preparation:• Model conterexample in SLICModel conterexample in SLIC• C C BP (C2BP) BP (C2BP)
– Verification:Verification:• BEBOP: perform reachability analysis of BPBEBOP: perform reachability analysis of BP• NEWTON: refine BP by discover additional predictsNEWTON: refine BP by discover additional predicts
– Error in BP Error in BP Error in P Error in P
• ApplicationsApplications– API in Windows XP device driversAPI in Windows XP device drivers
33
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
AgendaAgenda
• MotivationMotivation• Program Model CheckingProgram Model Checking• Program Model Checking ToolsProgram Model Checking Tools
– The Spin Tool The Spin Tool – The Verisoft ToolThe Verisoft Tool– The SLAM ToolThe SLAM Tool
• Use Spin tool to check TTS ServerUse Spin tool to check TTS Server• Future WorkFuture Work
34
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
VMS TTS ServerVMS TTS Server
• Multi-threadedMulti-threaded– 1 server thread1 server thread– 1~10 tts working threads1~10 tts working threads– 1~10 IBM viavoice synthesis threads1~10 IBM viavoice synthesis threads
• Asynchronzed Message QAsynchronzed Message Q– server server working threads working threads
• Synchronized Synchronized – synthesis threads synthesis threads working threads working threads
IBM V_V Engine
working thread 1
working thread 10
Socket ServerThread
socketreq
...........
35
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
VMS TTS ServerVMS TTS Server, cont, cont
• A Sequence DiagramA Sequence Diagram
36
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
PurposePurpose
• Check safety propertiesCheck safety properties– potential deadlockspotential deadlocks– Unexpected end statesUnexpected end states– Unreachable codeUnreachable code
• Check if it works for 10 V V instancesCheck if it works for 10 V V instances– Currently we have only 1 instanceCurrently we have only 1 instance
• Check about the Msg LossCheck about the Msg Loss
37
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Result in SimulationResult in Simulation
• No Error found in program (simulation)No Error found in program (simulation)– 1 bug found in promela model1 bug found in promela model
• Deadlock in server threadDeadlock in server thread• Forget to add lock when access globals in abstr.Forget to add lock when access globals in abstr.
– There are un-reached codeThere are un-reached code– Abstraction is not a easy workAbstraction is not a easy work
• Programs works for 10 threadsPrograms works for 10 threads• There are message lossThere are message loss
– Under high stressUnder high stress• Every thread are full loadedEvery thread are full loaded
– Increase # of buffers Increase # of buffers fewer loss fewer loss– Increase # of instance Increase # of instance fewer loss fewer loss– Decrease synthesis delayDecrease synthesis delay
• As for performanceAs for performance– Simulation is not trustableSimulation is not trustable
38
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Result in PANResult in PAN
39
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Restrictions, LessonsRestrictions, Lessons
• No Timer in PromelaNo Timer in Promela– Use loop to simulate a timerUse loop to simulate a timer– Different behavior in simulation & PANDifferent behavior in simulation & PAN
• Hard to locate an errorHard to locate an error– Too many messagesToo many messages– Generally, subtle errors occur after quite a Generally, subtle errors occur after quite a
long timelong time
• Platform dependant APIs are hard to be Platform dependant APIs are hard to be abstractedabstracted– What is the size of the Win32 thread What is the size of the Win32 thread
message queue?message queue?
40
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
AgendaAgenda
• MotivationMotivation• Program Model CheckingProgram Model Checking• Program Model Checking ToolsProgram Model Checking Tools
– The Spin Tool The Spin Tool – The Verisoft ToolThe Verisoft Tool– The SLAM ToolThe SLAM Tool
• Use Spin tool to check TTS ServerUse Spin tool to check TTS Server• Future WorkFuture Work
41
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Future WorkFuture Work
• More applicationsMore applications– OS Adapter ProjectOS Adapter Project– POSIX threadPOSIX thread– ……
• Tool enhancementsTool enhancements– Locate bug easier ?Locate bug easier ?– Timer ?Timer ?
• Automata Automata LTL ? LTL ?
42
Lucent TechnologiesLucent TechnologiesBell Labs InnovationsBell Labs Innovations
Reference SiteReference Site
• SPINSPIN– http://spinroot.com/spin/whatispin.htmlhttp://spinroot.com/spin/whatispin.html
• VerisoftVerisoft– http://cm.bell-labs.com/who/god/verisoft/http://cm.bell-labs.com/who/god/verisoft/
• SLAMSLAM– http://research.microsoft.com/slam/http://research.microsoft.com/slam/