MSN 2004
Network Memory Servers:An idea whose time has come
Glenford Mapp
David Silcott
Dhawal Thakker
MSN 2004
Motivation
• Networks are now much faster than disks
• Should be quicker to get data from the memory of another computer compared to using local disk
• Not a new idea - so what’s different?
MSN 2004
What’s different?• Networks are faster and cheaper
– Gigabit NICs are £35.00– We could also see 10G NICs in the near future
• Memory is also cheaper– 1GB = £100.00 – Likely to remain stable
• Availability of good “free” Oses– Linux and Free BSD
MSN 2004
Our approach is also different
• Previous approaches– Dominated by the Distributed Shared Memory
crowd (Apollo System)– DSM never became mainstream
• lots of fundamental changes to OS platform required
• Exotic Hardware (e.g Scalable Coherent Interconnect or SCI)
• Network Memory became a casualty of this failure
MSN 2004
Previous Approach cont’d• Remote paging was also one of the key
areas (SAMSON project, NYU)
• Idle machines approach– Use memory of other machines in the network
when no one is logged on but get off when the person returns
– Very complex -• how do you give guarantees to everyone
MSN 2004
Our Approach
• Applied Engineering Approach– what are the real numbers in this area
• Use the power of the Network– use standard networking approach– No DSM, no virtual memory plug-ins
• Client-Server approach– Dedicated servers with loads of memory
MSN 2004
Design of the Network Memory Server (NMS)
• NMS has an independent interface– Can interface with any OS
• not like Network Block Device (NBD) in Linux
• NMS is stateless– Does not keep track of previous interactions
• Actions of the NMS are regarded as atomic– Either complete success or total failure
MSN 2004
Design of NMS cont’d
• NMS deals with blocks of data– Has no idea how the blocks are being used
• Not like NFS
• Each block is uniquely identified by a block_id allocated by the NMS
• Each client is uniquely identified by a client_id
MSN 2004
Block_ids
• 64-bit entities– 32 minor index– 16 major index– 16 bit security tag
• generated when the blocks are created
• checked before any read/write operation on a block
MSN 2004
NMS calls
• GetblockMemory(client_id, size, nblocks, options)– Creates a number of blocks of a certain size
with consecutive block_ids• returns the starting Block_id
• options - backup
• Release(client_id, block_id, nblocks)– Releases a number of consecutive block_ids
MSN 2004
NMS calls cont’d• WriteBlockMemory(client_id, block_id,
offset, length, *buf)– writes data in buffer to a block on the server
• ReadBlockMemory(client_id, block_id, offset, length, *buf)– reads data from a block on the server into a
buffer
MSN 2004
NMS calls cont’d
• GetClientid(password)– creates a new client
• GetMasterBlock(password, client_id)– returns a number of blocks of sector/block_id
mappings
• StoreMasterBlock(block_id, client_id, password, nblocks) – stores a number of sector/block_id mappings
MSN 2004
NMS Client
• How does a client use the NMS?– What interface is presented to the OS
• Interface is one that is used to support hard disks. In Linux, we use the block device interface
• So the OS thinks of the NMS service as a fast hard disk
MSN 2004
NMS Client cont’d
• So the OS tells the NMS client to read and write sectors.
• NMS client will take sectors and map them onto blocks which it gets from the NMS
• When block device is unmounted, we must store the sector/block_id mappings on the NMS
MSN 2004
NMS Cont’d
• The StoreMasterBlock call stores these mappings on the NMS
• When the device is remounted, it must first get the sector/block_id mappings from the NMS and rebuild the sector table.
• The GetMasterBlock call retrieves the mappings from the NMS
MSN 2004
NMS Client Cache
• Client also has a cache of blocks that are used to store recently used sectors– this is a secondary cache as the main caching is
really done by the Unix Buffer Cache
• Design decision to keep our cache as a simple round-robin cache -– replace the next item pointed to in the cache
MSN 2004
NMS Client Operations• Since we are not a normal disk, we do not
need to rearrange read and write operations
• So we attempt to read and write blocks as the requests come in.
• Also developed a write-out thread operation. So a special thread, called the Write-out thread writes modified blocks to the NMS
MSN 2004
NMS Client Implementation
Operating System
Block Device Interface
Sector / Block_idHash Table
Cache
Programs
Unix Buffer Cache
Write-Out Queue
(Two levels)
NMS Block Device
MSN 2004
Getting a sectorIs sector in Hash table
YesIs it in the cache
Is it a readYes
Return Rubbish
Get Block_idFrom NMS. Put Entry inHash Table
Is it a read
Get Data from NMS Server; putin cache entry
Is the cache full
Replace Entry
Has replaced entrybeen modified
Put it on WriteOut Queue
Get New Cache Entry
Read from/ Writeto Cache Entry
OKWrite Data toCache Entry
Yes
No
No
Yes
Yes
No
No
Yes
No
No
MSN 2004
Structures on NMS Server
Client_id Hash Table
Block_idHash Table(Two-level)
Allocated Memory
Memory for Clients
Memory for InternalUse by the NMS
MSN 2004
Testing and Evaluation
• What do we really want to know
• What does it take to operate faster than a hard disk?– Can you use standard hardware (Middlesex)– Do you need special hardware (Cambridge)
• Level 5 Networks
• What are the key parameters in this space
MSN 2004
What do you measure• What happens if we change the block size
of the data transfer
• What happens if we change the number of units transferred in one transfer– Added multi-write operation
• Is local caching any good
• What is the network traffic like
MSN 2004
Using Iozone• Iozone is quite popular
– Measures the memory hierarchy
• Disk particulars– 60 GB, 2MB buffer, 7200 RPM, Seek Time 9.0 ms,
Average latency 4.16ms
• Network -– using Intel E1000 NICs and Netgear Gigabit
Switch (GS 104); using UDP port 6111
• NMS client and server implemented as Linux kernel modules
MSN 2004
Read Performance
0
200000
400000
600000
800000
1000000
1200000
1400000
0 50000 100000 150000 200000 250000 300000 350000
kB file
kB/s
ec
mw4, 2MB_cache, 1kB_msgdisksw, 2MB_cache, 4kB_msg
MSN 2004
Record Rewrite Performance
0
200000
400000
600000
800000
1000000
1200000
0 50000 100000 150000 200000 250000 300000
kB file
kB/s
ec
MW4 2mb cache, 1k
disk system
MSN 2004
Write Performance for Different Transfer sizes
0
50000
100000
150000
200000
250000
300000
0 50000 100000 150000 200000 250000 300000 350000
kB file
kB/s
ec
sw, 2MB_cache, 4kB_msgdisksw, 2MB_cache, 1kB_msgsw, 2MB_cache, 2kB_msg
MSN 2004
Write Performance for Multiples of 1K blocks
0
50000
100000
150000
200000
250000
300000
0 50000 100000 150000 200000 250000 300000
kB file
kB/s
ec
mw4, 2MB_cache, 1kB_msgdiskmw12, 2MB_cache, 1kB_msgmw8, 2MB_cache, 1kB_msgmw16, 2MB_cache, 1kB_msg
MSN 2004
Write Performance for extreme configurations
0
50000
100000
150000
200000
250000
300000
0 50000 100000 150000 200000 250000 300000 350000
kB file
kB/s
ec
disk
mw17k, 2MB_cache, 4kB_msg
mw32k, 8MB_cache, 4kB_msg
sw, 2MB_cache, 1kB_msg
MSN 2004
Maximum data transfer rate
82
83
84
85
86
87
88
50 100 150 200 250
Filesize(MB)
Ra
te(M
b/s
ec
)
Received
Sent
MSN 2004
Buffer cache Hits
0
20
40
60
80
100
120
100 150 200 250
Filesize(MB)
% b
loc
k c
ac
he
hit
s
BCH MAX
BCH MIN
MSN 2004
Conclusions and Future
• We can beat the disk
• Will compare these results with those using Level 5 hardware (Rip Sohan, LCE)
• Open source release planned
• Developing a Network Storage Server
• Building prototypes – running Linux and Windows using NMS
