Grid File System WG – GGF 17

Arun Jagatheesan

San Diego Supercomputer Center

GGF 17May 11, 2006Tokyo, Japan

2Global Grid Forum-17, Tokyo Grid File System Group

IP & ©

The GGF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the GGF Secretariat.

The GGF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights which may cover technology that may be required to practice this recommendation. Please address the information to the GGF Executive Director.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the GGF or other organizations, except as needed for the purpose of developing Grid Recommendations in which case the procedures for copyrights defined in the GGF Document process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the GGF or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE GLOBAL GRID FORUM DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE."

Intellectual Property Statement

Copyright (C) Global Grid Forum (2004). All Rights Reserved.

3Global Grid Forum-17, Tokyo Grid File System Group

Talk Outline

• Our original charter• Status

• What we reported in GGF-16• What we have now at GGF-17• What we will have for GGF-18

• GFS Architecture• What, why, how…

• Summary

4Global Grid Forum-17, Tokyo Grid File System Group

Grid File System WG

• Standard mechanism to describe and organize file-based data in grids

• Provide a higher level, human readable logical namespace of data and resources in a data grid

• An architecture that would allow vendors and users to distinguish compatible products that are “GGF GFS complaint”

5Global Grid Forum-17, Tokyo Grid File System Group

Example: GFS Namespace/grid

ggf jp

aist gtrc

file1 file3file2 pub

file1 file2

ftp://meti.go.jp/gsiftp://aist.go.jp/ http://u-tokyo.ac.jp/

/a/b/file2/d/file1

/a/b/file1/a/b/file3/b/file1 /c/file3

/pub/

GFS Service

File TransportService

Hierarchical namespacewith filesystem metadata Unique path name Access control Write lock Other metadata

6Global Grid Forum-17, Tokyo Grid File System Group

GFS-WG Status reported in GGF-16• Informational Document : GFS-Architecture

• Approved now as a GFD-61• http://www.ggf.org/documents/GFD.61.pdf• That was only version – 1 of the Architecture workbook

• RNS Specification• Under discussion, public review (?)• More discussion on this session too

• New Secretary• Christopher Jordon

• External Alliances• SNIA, IETF NFSv4 (??)

7Global Grid Forum-17, Tokyo Grid File System Group

GFS-WG Status as of GGF-17 (NOW)• Informational Document : GFS-Architecture

• Approved now as a GFD-61 (http://www.ggf.org/documents/GFD.61.pdf)

• Working on next version with end-to-end architecture

• Focus will be on “GFS complaint software” for vendors and users

• RNS Specification• Re-factored version of RNS document - public discussion

• GFSG and community to decide whether incremental progress can be made on this initial draft or wait for community consensus.

• External Alliances

• SNIA – Currently a group here in GGF. Look forward to working with our SNIA friends.

• IETF NFSv4 : We had initial discussions. But, no follow up was made by our GFS as we still need to know more (and less resources to spread out).

8Global Grid Forum-17, Tokyo Grid File System Group

What will we have by GGF –18 ?

• Proposed Deliverables• RNS spec finalized• OGSA - GFS use case• GGF Architecture Workbook version 2• GGF Architecture Standard document – Skeleton

• Topic not limited to…

• Data Grid

• Composition of services

• System and User-defined Meta-data in GFS

• GFS Profile

• Why and what “GGF GFS complaint” product or service

• Work with our Storage friends in GGF and SNIA

9Global Grid Forum-17, Tokyo Grid File System Group

Talk Outline

• Our original charter• Status

• What we reported in GGF-16• What we have now at GGF-17• What we will have for GGF-18

• GFS Architecture• What, why, how…

• Summary

10Global Grid Forum-17, Tokyo Grid File System Group

GFS Grid Resources• Context (Information)

• Information about digital entities (location, size, owners, ..)• Relationship between digital entities (replicas, collection, .)• Behavior of the digital entities (services)

• Content (Data)• Structured and unstructured• Virtual or derived

• Commodity (Producers and consumers)• Physical storage resources and file systems• Logical resources

11Global Grid Forum-17, Tokyo Grid File System Group

GFS Resource Providers

GFS Resource Providers (GRP) providing content

and/or storageGRP

/txt3.txt

GRP

12Global Grid Forum-17, Tokyo Grid File System Group

GFS Administrative Domain

GRP

• GFS Administrative domain with one or more GFS Resource Providers

•Could include their data centers • SNIA note: This is at very higher level than block, file/record level. Yet deals with objects

/txt3.txt

GRP

Research Lab

13Global Grid Forum-17, Tokyo Grid File System Group

GFS Administrative domains

/…/text1.txt /…//text2.txt

GRP GRP GRP GRPGRP GRP GRP

/txt3.txt

GRP

Storage-R-Us Resource Providers

data + storage (50)

Finance Departmentdata + storage (40)

Research Labdata + storage (10)

14Global Grid Forum-17, Tokyo Grid File System Group

Grid File System

/…/text1.txt /…//text2.txt

GRP GRP GRP GRPGRP GRP GRP

/txt3.txt

GRP

Storage-R-Us Resource Providers

data + storage (50)

Finance Departmentdata + storage (40)

Research Labdata + storage (10)

/home/arun.sdsc/exp1/home/arun.sdsc/exp1/text1.txt/home/arun.sdsc/exp1/text2.txt/home/arun.sdsc/exp1/text3.txtdata + storage (100)

Logical Namespace (Need not be same as physical namespace )

15Global Grid Forum-17, Tokyo Grid File System Group

GFS Client Server Software Stack

GFS Server (code)

GFS WS-Client GFS Client ProtocolTraditional FS clients

Users see the same namespace. But, the

functionalities and data types supported may differ

/home/arun.sdsc/exp1/home/arun.sdsc/exp1/text1.txt/home/arun.sdsc/exp1/text2.txt/home/arun.sdsc/exp1/text3.txtdata + storage (100)

/…/text1.txt /txt3.txt

16Global Grid Forum-17, Tokyo Grid File System Group

GFS Architecture Standard Interfaces (?)

GFS Server (code)

GFS WS-Client GFS Client ProtocolTraditional FS clients

/…/text1.txt /txt3.txt

Is the user interface to access GFS a standard one?

Is the server interface by each vendor a standard?

(This will allow server-to-server peer communication

17Global Grid Forum-17, Tokyo Grid File System Group

Questions for GFS Architecture

• Logical resource namespace• Logical resources have been popular

• Work with existing standards • Without loosing advanced functionalities• Vendor interoperability

• Single interface• Composed from others (services)• A single interface of all functionalities (out of the box)

• Block level protocols• Bulk operations, data manipulation, …

18Global Grid Forum-17, Tokyo Grid File System Group

World Wide Datagrid

19Global Grid Forum-17, Tokyo Grid File System Group

What could be done?

• Standards with functionalities or operations used by users in production

• Collaborative logical namespace management

• Inter/Intra/Multi Organizational namespace

• Logical storage resources (no mount points)

• Metadata, annotation support for discovery

• Bulk operations

• User groups and User access control

• Automated data integrity checks

20Global Grid Forum-17, Tokyo Grid File System Group

Suggestions, comments

• Self Assessment• Need to restart bi-weekly telecon (at least monthly)• F2F discussions (or polycom)• More people to work or just provide input than be passive observers

(especially from vendors, we got users)

• Suggestions • Focus on what works and what is used in production• Products and standards emerge based on what works and user

needs• Flexibility of the WG’s to refocus the charter along with other

production groups or users [what is out there and required]

21Global Grid Forum-17, Tokyo Grid File System Group

Summary

• GFS-WG results delivered, has to evolve more• Data Grids : Multiple projects and production

users world-wide• Multiple common features and patterns observed in

academic usage• Vendor or Industry Interest for similar commercial use

cases (work with them also)

• GFS-WG more deliverables• Conference calls, Architecture, re-factored documents

22Global Grid Forum-17, Tokyo Grid File System Group

Acknowledgement

RNS, Active Contributors:• Arun Jagatheesan, SDSC• Christopher Jordon, SDSC• Manuel Pereira, IBM Almaden Research Center• Osamu Tatebe, University of TsukubaArchitecture Contributors:• Ted Anderson (IBM Almaden)• Cameron Bahar (Storage Machines)• Leo Luan (IBM Almaden)• Reagan Moore (SDSC)• Ken Wood (Hitachi Data Systems)• Jane Xu (IBM Storage Software)• Alan Yoder (Network Appliance)