Lazy Asynchronous I/O For Lazy Asynchronous I/O For Event-Driven ServersEvent-Driven Servers

Khaled Elmeleegy, Anupam Chanda and Alan L. Khaled Elmeleegy, Anupam Chanda and Alan L. CoxCox

Department of Computer ScienceDepartment of Computer ScienceRice University, Houston, Texas.Rice University, Houston, Texas.

Willy ZwaenepoelWilly ZwaenepoelSchool of Computer and Communication School of Computer and Communication

SciencesSciencesEPFL, Lausanne, Switzerland.EPFL, Lausanne, Switzerland.

22

Event-Driven ArchitectureEvent-Driven Architecture

Event-driven architecture is widely Event-driven architecture is widely used for serversused for servers PerformancePerformance ScalabilityScalability

33

ProblemProblem

Developing event-driven servers is Developing event-driven servers is hard for many reasonshard for many reasons We focus on problems with non-blocking We focus on problems with non-blocking

I/O:I/O: Incomplete coverageIncomplete coverage Burden of state maintenanceBurden of state maintenance

44

Lazy Asynchronous I/O Lazy Asynchronous I/O (LAIO)(LAIO)

Addresses problems with non-blocking Addresses problems with non-blocking I/OI/O UniversalityUniversality

Covers all I/O operationsCovers all I/O operations SimplicitySimplicity

Requires less codeRequires less code

High performance for event-driven High performance for event-driven serversservers Meets or exceeds alternativesMeets or exceeds alternatives

55

OutlineOutline

BackgroundBackground Lazy Asynchronous I/O (LAIO)Lazy Asynchronous I/O (LAIO) EvaluationEvaluation ConclusionsConclusions

66

OutlineOutline

BackgroundBackground Event-driven serversEvent-driven servers

Lazy Asynchronous I/O (LAIO)Lazy Asynchronous I/O (LAIO) EvaluationEvaluation ConclusionsConclusions

77

Event-Driven ServersEvent-Driven Servers

Event loop Event loop processes processes incoming eventsincoming events

For each incoming For each incoming event, it dispatches event, it dispatches its handlerits handler

Single thread of Single thread of executionexecution

EventLoop

Handler #1

Handler #2

Handler #k

88

Event HandlerEvent Handler

I/O operation(Network/Disk)

Complete handlingevent

To event loop

99

Event HandlerEvent Handler

I/O operation(Network/Disk)

Complete handlingevent

Block

If the I/O operation If the I/O operation blocksblocks The server stallsThe server stalls

To event loop

1010

OutlineOutline

BackgroundBackground Lazy Asynchronous I/O (LAIO)Lazy Asynchronous I/O (LAIO)

APIAPI Example of usageExample of usage ImplementationImplementation

EvaluationEvaluation ConclusionsConclusions

1111

LAIO APILAIO API

ReturReturn n

TypeType

Function Function NameName

ParametersParameters

intint laio_syscalllaio_syscall int number,…int number,…

void*void* laio_gethandlaio_gethandlele

voidvoid

intint laio_polllaio_poll laio_completion[] laio_completion[] completions, completions,

int ncompletions,int ncompletions,

timespec* tstimespec* ts

1212

laio_syscall()laio_syscall() Lazily converts any system call into Lazily converts any system call into

an asynchronous callan asynchronous call If the system call doesn’t blockIf the system call doesn’t block

laio_syscall() returns immediatelylaio_syscall() returns immediately With return value of system callWith return value of system call

Else if it blocks:Else if it blocks: laio_syscall() returns immediatelylaio_syscall() returns immediately With return value -1With return value -1 errno set to EINPROGRESSerrno set to EINPROGRESS Background LAIO operationBackground LAIO operation

1313

laio_gethandle()laio_gethandle()

Returns a handle representing the Returns a handle representing the last issued LAIO operationlast issued LAIO operation If operation didn’t block, NULL is If operation didn’t block, NULL is

returnedreturned

1414

laio_poll()laio_poll() Returns a count of completed Returns a count of completed

background LAIO operationsbackground LAIO operations Fills an array with completion entriesFills an array with completion entries

One for each operationOne for each operation Each completion entry hasEach completion entry has

HandleHandle Return valueReturn value Error valueError value

1515

Event Handler With LAIOEvent Handler With LAIO

laio_syscall()

Operationcompleted

Complete handlingevent

Yes

No

To event loop

If operation If operation completescompletes Handler continues Handler continues

executionexecution

1616

Event Handler With LAIOEvent Handler With LAIO

laio_syscall()

Operationcompleted

Complete handlingevent

Yes

No

To event loop

If operation doesn’t If operation doesn’t completecomplete laio_syscall() laio_syscall()

returns immediatelyreturns immediately Handler records Handler records

LAIO handleLAIO handle Returns to event Returns to event

looploop Completion Completion

notification arrives notification arrives laterlater

Handle= laio_gethandle()

1717

LAIO ImplementationLAIO Implementation

LAIO uses scheduler activationsLAIO uses scheduler activations Scheduler activationsScheduler activations

The kernel delivers an upcall when an The kernel delivers an upcall when an operationoperation

Blocks - laio_syscall()Blocks - laio_syscall() Unblocks - laio_poll()Unblocks - laio_poll()

1818

laio_syscall() – Non-blocking laio_syscall() – Non-blocking casecase

Issue operation

• Save context• Enable upcalls

System call

blocks?

• Disable upcalls• Return retval

No

laio_syscall()

Application

Library

1919

laio_syscall() – Blocking laio_syscall() – Blocking casecase

Issue operation

• Save context• Enable upcalls

System call

blocks?

laio_syscall()

Application

Library

Yes

2020

laio_syscall() – Blocking laio_syscall() – Blocking casecase

Issue operation

• Save context• Enable upcalls

System call

blocks?

laio_syscall()

Application

Library

Upcall on a new threadKernel

Yes

Background laio_syscall

2121

laio_syscall() – Blocking laio_syscall() – Blocking casecase

Issue operation

• Save context• Enable upcalls

System call

blocks?

laio_syscall()

Application

Library

Upcall on a new threadKernel

Yes

upcall handler

Steals old stack using stored context

Library

Background laio operation

2222

laio_syscall() – Blocking laio_syscall() – Blocking casecase

Issue operation

• Save context• Enable upcalls

System call

blocks?

laio_syscall()

Application

Library

Upcall on a new threadKernel

Yes

upcall handler

Steals old stack using stored context

Library

• Disable upcalls• errno = EINPROGRESS• Return -1

Background laio operation

2323

Unblocking caseUnblocking case

List of List of completions is completions is retrieved by the retrieved by the application application using laio_poll()using laio_poll()

• Background laio operation completes, thread dies• Upcall on the current thread

Kernel

upcall handler()

• Construct completion structure:• laio operation handle.• System call return value.• Error code.

• Add completion to list of completions.

Library

2424

OutlineOutline

BackgroundBackground Lazy Asynchronous I/O (LAIO)Lazy Asynchronous I/O (LAIO) EvaluationEvaluation

MethodologyMethodology ExperimentsExperiments

LAIO vs. conventional non-blocking I/OLAIO vs. conventional non-blocking I/O LAIO vs. AMPEDLAIO vs. AMPED

Programming complexityProgramming complexity ConclusionsConclusions

2525

MethodologyMethodology

Flash web serverFlash web server Intel Xeon 2.4 GHz with 2 GB memoryIntel Xeon 2.4 GHz with 2 GB memory Gigabit Ethernet between machinesGigabit Ethernet between machines FreeBSD 5.2-CURRENTFreeBSD 5.2-CURRENT Two web workloadsTwo web workloads

Rice 1.1 GB footprintRice 1.1 GB footprint Berkeley 6.4 GB footprintBerkeley 6.4 GB footprint

2626

Experiments: LAIO vs. Experiments: LAIO vs. conventional non-blocking I/Oconventional non-blocking I/O

Flash-NB-BFlash-NB-B SingleSingle Disk I/ODisk I/O

Flash-NB-AIOFlash-NB-AIO SingleSingle Disk I/O other Disk I/O other than read and than read and writewrite

Flash-LAIO-LAIOFlash-LAIO-LAIO SingleSingle NoneNone

Server-Network-Server-Network-DiskDisk

ThreadedThreaded Blocking Blocking operationsoperations

Compare performance

2727

Performance: Large Performance: Large WorkloadWorkload

Berkeley Workload (warm cache)

0

50

100

150

200

250

300

0 500 1000 1500Number of Clients

Thro

ughp

ut (M

b/s)

Flash-NB-B

Flash-NB-AIO

Flash-LAIO-LAIO

2828

Rice Workload (warm cache)

0

200

400

600

800

1000

0 500 1000 1500Number of Clients

Thro

ughp

ut (M

b/s)

Flash-NB-B

Flash-NB-AIO

Flash-LAIO-LAIO

Performance: Small Performance: Small WorkloadWorkload

2929

Why Be Lazy?Why Be Lazy?

Most potentially blocking operations Most potentially blocking operations don’t actually blockdon’t actually block Experiments: 73% - 86% of such Experiments: 73% - 86% of such

operations don’t blockoperations don’t block Lower overheadLower overhead

Micro-benchmark: AIO is 3.2 times Micro-benchmark: AIO is 3.2 times slower than LAIO for non-blocking slower than LAIO for non-blocking operationsoperations

3030

Experiments: LAIO vs. Experiments: LAIO vs. AMPEDAMPED

Flash-LAIO-LAIOFlash-LAIO-LAIO SingleSingle NoneNone

Flash-NB-AMPEDFlash-NB-AMPED Process-Process-based based helpershelpers

NoneNone

Server-Network-Server-Network-DiskDisk

ThreadedThreaded Blocking Blocking operationsoperations

Compare performance

3131

Asymmetric Multi-process Asymmetric Multi-process Event-Driven (AMPED) Event-Driven (AMPED)

ArchitectureArchitecture Event-driven coreEvent-driven core Potentially blocking Potentially blocking

I/O handed off to a I/O handed off to a helperhelper

Helper can block as Helper can block as it runs on a it runs on a separate thread of separate thread of executionexecution

Helper #1

Helper #2

Helper #n

EventLoop

Handler #1

Handler #2

Handler #k

3232

Flash-NB-AMPEDFlash-NB-AMPED

Stock version of flashStock version of flash Two relevant helpersTwo relevant helpers

File read: reads a file from diskFile read: reads a file from disk Name conversion: checks for a file’s Name conversion: checks for a file’s

existence and permissionsexistence and permissions

3333

Berkeley Workload (warm cache)

0

50

100

150

200

250

300

0 500 1000 1500Number of Clients

Thro

ughp

ut (M

b/s)

Flash-NB-AMPED

Flash-LAIO-LAIO

Performance of LAIO vs. Performance of LAIO vs. AMPEDAMPED

3434

Rice Workload (warm cache)

0

200

400

600

800

1000

0 500 1000 1500Number of Clients

Thro

ughp

ut (M

b/s)

Flash-NB-AMPED

Flash-LAIO-LAIO

Performance of LAIO vs. Performance of LAIO vs. AMPEDAMPED

3535

Programming ComplexityProgramming Complexity

Where performance is comparable: Where performance is comparable: Flash-LAIO-LAIO vs. Flash-NB-AMPEDFlash-LAIO-LAIO vs. Flash-NB-AMPED Flash-LAIO-LAIO is simplerFlash-LAIO-LAIO is simpler

No helpersNo helpers No state maintenanceNo state maintenance

3636

State Maintenance With Non-State Maintenance With Non-blocking I/Oblocking I/O

I/O operation

Operationcompleted

Complete handlingevent

Yes

No

To event loop

If operation does If operation does not completenot complete Handler returns to Handler returns to

event loopevent loop

handler

3737

State Maintenance With Non-State Maintenance With Non-blocking I/Oblocking I/O

I/O operation

Operationcompleted

Complete handlingevent

Yes

No

To event loop

If operation does If operation does not completenot complete Handler returns to Handler returns to

event loopevent loop Operation is Operation is

continued latercontinued later

handler

3838

State Maintenance With Non-State Maintenance With Non-blocking I/Oblocking I/O

I/O operation

Operationcompleted

Complete handlingevent

Yes

NoTo event loop

Store state

If operation does If operation does not complete:not complete: Handler returns to Handler returns to

event loopevent loop Operation is Operation is

continued latercontinued later This may require This may require

storing statestoring state

handler

3939

Lines of Code ComparisonLines of Code Comparison

Total code sizeTotal code size 88608860 80208020

File readFile read 550550 1515

Name Name ConversionConversion

610610 375375

Partial-write Partial-write state state

maintenancemaintenance

7070 00

ComponentComponent Flash-NB-Flash-NB-AMPEDAMPED

lines of codelines of code

Flash-LAIO-Flash-LAIO-LAIOLAIO

lines of codelines of code

9.5% reduction in lines of code

4040

ConclusionsConclusions LAIO is universal LAIO is universal

Supports all system callsSupports all system calls LAIO is simplerLAIO is simpler

Used uniformlyUsed uniformly No state maintenanceNo state maintenance No helpersNo helpers Less lines of codeLess lines of code

LAIO meets or exceeds the LAIO meets or exceeds the performance of other alternativesperformance of other alternatives

4141

LAIO source can be found at:LAIO source can be found at:

http://www.cs.rice.edu/~kdiaa/laiohttp://www.cs.rice.edu/~kdiaa/laio