MMobiiSStore - MobMobile

iinternet File SStorage Platform

Chetna Kaur

Proposed Work

Develop a distributed internet storage platform that will allow internet enabled devices to manage storage over the internet.

What is an Internet File System

User’s view Storage provided on the Internet. Can access the data stored in the File System on the go. No need to buy expensive memory. Seamless integration into the users file system eg: Gmail Drive’

Technically, Not a Real File System Simulates a file system by having a central repository of folder

and file links. Files are stored on multiple storage servers. API’s are provided for users to log in, create and manage files and

folders.

Motivation - Why another Internet File System?

# of mobile internet users increasing exponentially.

Mobile clients have limited memory but growing memory requirements. – Potential User Base

Current internet file systems cannot cater to these clients due to client resource constraints (Memory, Bandwidth).

Why? What are the issues involved? How can I build a file system platform that can provide storage services to all types of clients?

Other sources of motivation

Show Gmail drive demo here

Solution

The Block Level Meta File System- Files are stored and accessed at a

block level. - Allows users with limited memory and

bandwidth requirements to access large files.

- File is downloaded in chunks as needed.

Challenges

Where Will Data Be Stored?Requiring lot’s of space is the easy part of the

problem! Availability Scalability Privacy and security Efficient data updates

Storage Access and Management Problems Customizing system to users’/applications’ needs Designing and managing block level meta data Adaptivity to existing or new applications

How do you share storage on top of blocks? Communication between components

Important Design Decisions

System Interface Scalable and High Availability Architecture Data Structures

What Meta-Data to capture? Fast Indexing User Based Views Ease of file sharing

Efficient Algorithms for creation of new files and allocation of new data blocks

File caching strategy Technology

System Design

Distributed Storage Server

WWW

Access Server

Meta File System Server

System Interface and Functionality Simple API for user login (password

based authentication). APIs to create, and delete files and

directories. APIs to read and write to files at a block

level granularity. API to share files with friends

Access Server - Functionality Users connect to the file system and access

their files via the Access Server. Client requests for File Block.

The Access Server gets the file information from the MFS Server.

Checks if file available in Cache. If not, it connects to the storage server fetches the

first block from the SS. Delegates the task of fetching rest of the blocks to

cache manager. Send the requested data block to the client.

Key Features

Asynchronous Communication Multithreaded – New thread for every

connection (uses a thread pool). Block Level File Caching and Pre-

Fetching. Cache Manager runs in separate thread,

pre-fetches next few blocks. Cache Replacement Policy – Least

Recently Used.

Storage Server

Essentially a flat file store. Each file can be distributed across

several actual files on one or more storage servers.

How then can we allocate new files and blocks on storage servers? Handled by the Meta File System Server.

Meta File System

Stores Meta-data related to users and files Uses hash maps for quick indexing of user

name spaces and files. Efficient allocation of new blocks and files

to storage servers. Tries to first allocate contiguous blocks so that

they can be pre-fetched together. If not possible, tries to allocate all blocks on same

storage server.

Design of the Meta File System

Main Contributions

Propose a Scalable, Highly-Available Architecture for a Distributed Internet File System.

Design and implement an efficient Meta-File structure for a block level file system implementation.

Implemented a simple file caching and pre-fetching mechanism, to improve efficiency of block level file access.

Design and implement APIs for accessing files block by block for client with resource constraints.

Implement a proof of concept system, and built a basic web based file storage application on this platform to demonstrate its capabilities.

Open Research Areas

Persistence of Meta File System structure Improved File Caching Mechanism in the

Access Server Test the performance of the Block

Allocation Algorithm and improve it. Towards a Google Docs type of

implementation (Multiple users can write different blocks of data simultaneously)

Handling Multimedia Files

Technologies Used

+ Front End Web ApplicationPHP

+ ServerApache, C++, Adaptive Communication Environment, STL/BOOST

+ CommunicationJSON over HTTP

Questions?