Idempotent Transactional Workflow
(POPL 2013)
G. RamalingamKapil Vaswani
Microsoft Research India
Application
The Problem
Partitioned Data
scale-out
Can we simplify
writing suchapplications?
Transfer (amt, acct1, acct2) {Debit amt from acct1;Credit amt to acct2;
}
Transfer (amt, acct1, acct2) atomic {
Debit amt from acct1; Credit amt to acct2;
}
ACID Transaction+ Strong consistency− Distributed
transaction
Transfer (amt, acct1, acct2) atomic {Debit …}; atomic {Credit …};
Workflow− Weaker consistency
− No isolation+ No distributed transaction
What about process failure? Claim: Workflows are common in applications over partitioned data
Storage Layer
Application Logic
Stopping (non-byzantine) failure
The Problem
(failures handled by storage layer)
Goal• Fault-tolerance in application• A transactional workflow
engine• decentralized!
Modern Cloud Platforms
request response
Making Workflows Fault-Tolerant
Request or response
may be lost!
Taking a step back …
Resending messages
is a critical elementof fault-tolerance
Must be Idempotent!
(tolerate duplicatemessages)
Transfer (amt, acct1, acct2) {Debit amt from acct1;Credit amt to acct2;
}
Goal:Idempotent Fault-Tolerance
• (Idempotent Workflow)• A program is said to be idempotent & fault-
tolerant iff– its behavior is unaffected by process failures– its behavior is unaffected by duplicate input
requests• Behavioral equivalence:– duplicate output responses allowed– progress (liveness) conditions
• slightly weakened
request response
Making WorkflowsIdempotent & Fault-Tolerant
request response
Making Computations Idempotent
Make every effectful step idempotent:1. Associate unique id with every step2. Modify step to log execution of step3. Modify step to check if it has already
executedAll must be done atomically !
AutomatedIdempotent Fault-Tolerance
• As a library– In C# & F#– Technically, a monad
• As a compiler
• As a programming-language construct
Formal ResultsTheorem. A well-typed monadic program isidempotent and fault-tolerant.
Theorem. compile[e] is an idempotent and fault-tolerant realization of e.
Any (well-typed) program e can be automatically translated (compiled) into a program compile[e]
Idempotence: A Language Construct
• “idworkflow uid e’’transfer (uid, amt, acct1, acct2) { idworkflow uid {
atomic T1 Debit amt from acct1 atomic T2 Credit amt to acct2}}
}
Extensions• Compensating actions– Undo earlier actions when later actions
encounter logical failure
• Automatic retry– Detect process failures & restart
• Checkpointing– Restart at most recent checkpoint
Questions?
Fault-Tolerance & Idempotence: Simpler Together
Storage Layer
Application Logic
client
service
partitioneddata
Problem Setting