OrangeFS Installation Instructions (2.9)

OrangeFS Installation Instructions (2.9)

Copyright © 2014 by Omnibond Systems, LLC.

All rights reserved. All third-party trademarks and trade names are the property of their respective owners.

i

Table of Contents

Installation Guide ..................................................................................................................... 1

Who Is This Information For? ................................................................................................. 1

About OrangeFS ................................................................................................................... 1

Installation Concepts ................................................................................................................ 2

Plan ....................................................................................................................................... 5

Preview System Requirements ................................................................................................ 5

Preview Security ..................................................................................................................10

Preview OrangeFS Configuration File ......................................................................................15

Build and Configure .................................................................................................................18

Build OrangeFS ....................................................................................................................18

Set Up Security (Build System) ..............................................................................................21

Configuring LDAP for Identity Mapping....................................................................................24

Create OrangeFS Configuration File ........................................................................................29

Results (Build and Configure) ................................................................................................32

Add Servers ...........................................................................................................................36

Copy OrangeFS Installation Directory (Server Systems) ............................................................36

Set up Security (Server Systems) ..........................................................................................37

Run (Server Systems) ..........................................................................................................38

Results (Add Servers) ...........................................................................................................39

Add Clients .............................................................................................................................40

pvfs2tab File .......................................................................................................................43

Kernel Module .....................................................................................................................44

Direct Interface ...................................................................................................................49

FUSE ..................................................................................................................................52

ROMIO MPI Interface ............................................................................................................56

Windows Client Interface ......................................................................................................59

Web Pack Clients .................................................................................................................80

Hadoop Client ......................................................................................................................93

Other Installation Topics ........................................................................................................ 123

Basic Installation Example / Quick Start................................................................................ 123

Berkeley DB Version Support ............................................................................................... 126

Sample Installation Directory and File Listing ........................................................................ 127

Storage Directory Location .................................................................................................. 129

Automating System Startup ................................................................................................ 130

Working With Firewalls ....................................................................................................... 130

ii

iii

OrangeFS Installation Instructions (2.9) Installation Guide

1

Installation Guide

This part of the documentation provides instructions for installing OrangeFS. Topics are organized into these categories:

Installation Concepts (page 2)

An overview of the four basic installation steps, including the concepts and icons associated with them throughout this Guide.

Plan (page 5) Preview of system, configuration and security considerations you must make prior to installation.

Build and Configure (page 18)

Procedures for setting up and configuring the build system, ready for deployment to other systems in your OrangeFS solution.

Add Servers (page 36) Procedures for copying OrangeFS from the build system to each server, setting it up for security, then running it as part of your OrangeFS solution.

Add Clients (page 40) Individual procedures for the variety of client systems, environments and

interfaces supported by OrangeFS.

Other Installation

Topics (page 123)

Miscellaneous topics about installation considerations and scenarios.

Who Is This Information For?

This Installation Guide assumes you are experienced in installing and administering Linux operating system and application software. You should also be familiar with the basic concepts of parallel file

systems and OrangeFS in particular.

About OrangeFS

OrangeFS is a next-generation parallel file system for compute and storage clusters of the future. OrangeFS data resides on multiple servers instead of just one. IO performance is gained by storing a file in objects across multiple servers and accessing these objects in parallel.

OrangeFS Installation Instructions (2.9) Installation Concepts

2

Installation Concepts

Installing OrangeFS involves four basic steps, each with its own group of topics.

The following table introduces these steps, including the concepts and icons associated with them

throughout the Installation Guide.

Topic Description

Step 1: Plan

Preview system requirements

This topic explores system considerations, including:

The three basic system types

Recommended distributions of Linux and related software packages

Hardware requirements

General rules for choosing a network protocol

Preview

security

This topic introduces the three modes of security available for

OrangeFS:

Default

Key-based

Certificate-based

The mode you select will affect the amount of preparation

required before continuing with the installation process. Your

choice will also affect configuration file settings.

Preview OrangeFS configuration file

This topic discusses the OrangeFS configuration file, which is copied to all servers as a single reference point for operation and performance. In this file you specify settings and preferences for all servers in your installation.

During installation, the file is automatically generated in basic

default mode. After installation, you can revisit the configuration file to make more changes and additions from a range of options and their associated values.


3

Topic Description

Step 2: Build and Configure

Build OrangeFS

This involves downloading the source software from

orangefs.org onto a system preconfigured with several

standard Linux packages. On this system you will extract

and build OrangeFS in a portable directory named /opt/orangefs. From there, you can complete its

configuration and deploy it to other Linux systems that

have fewer package requirements.

Important Because clients have different requirements for

the OrangeFS build system, please read through all installation instructions for the client(s) you plan to use BEFORE you build OrangeFS.

Set up security

After you build the OrangeFS installation directory, your selected security mode determines the additional setup and configuration tasks.

Much of this work can be done once on the Build system, then copied to your servers and even your Linux-based clients.

Create

OrangeFS configuration file

The OrangeFS installation directory on the Build system requires

an OrangeFS configuration file. You will enter some basic

information for this file in a program called pvfs2-genconfig.

Once the configuration file has been created, you might need to

make additional modifications (regarding security, for example) for the initial deployment.

Note After standard installation, you can consult the Administration Guide for details on options and values available for fine-tuning the configuration file.

Step 3: Add Servers

Copy OrangeFS Installation directory

Begin your deployment by copying the OrangeFS installation directory from the Build system to each server designated in your OrangeFS solution. You will have already identified these servers in the OrangeFS configuration file.

Set up security

If you select key-based security mode, you must copy a private key generated on the Build system to each of your Server

systems.


4

Topic Description

Run

Running each server involves two administrative tasks:

Initializing the working directories that represent the server's storage space

Starting the server process

A command statement that includes the OrangeFS server

daemon (pvfs2-server) accomplishes both tasks.

Step 4: Add Clients

select client interface(s)

The Client system, with a variety of Client Interfaces supported by OrangeFS, provides many options for accessing OrangeFS.

Client systems are not limited to the Linux operating environment. Depending on the Client Interface you select, the OS on your Client system can be Linux, Windows, MacOS X or even Apache (web-based).

Follow

individual installation instructions

The requirements for client systems and interfaces can be

addressed separately, since their instructions assume the file system servers are already installed and running.

Set up security

As you add each client, you must consider additional setup and configuration tasks, depending on the mode of security you select.

OrangeFS Installation Instructions (2.9) Plan

5

Plan

Once you begin to download and install OrangeFS, you are presented with options based on your environment, resources and objectives for using the file system. Advance planning for these

considerations ensures a smoother installation.

If you take advantage of OrangeFS security features, more considerations and even some prerequisites might need to be in place.

During these preparations, you must understand the significance of the OrangeFS configuration file, a single point of reference for global settings and options in your OrangeFS installation.

These subjects are covered in the following topics:

Preview System Requirements (page 5)

Preview Security (page 10)

Preview OrangeFS Configuration File (page 15)

Preview System Requirements

This topic explores system requirements, including the three basic system types, recommended distributions of Linux and related software packages, hardware requirements and general rules for choosing a network protocol.

Any OrangeFS installation includes three basic system types:

System Type

Description

Build This is the Linux system where you download the source software from orangefs.org.

OrangeFS supports most common Linux distributions. Some examples include Red Hat Enterprise Linux, OpenSUSE, Fedora and Ubuntu.

Additional Linux software packages needed to build OrangeFS are detailed under

“Additional Linux Software for the Build System (page 7)” below.

On the Build system you will compile the OrangeFS source package into executable form. The executable software is copied to Server and Client systems.

Server Servers are scalable; you can install OrangeFS on one server or multiple servers. For best practices, each server system should run the same Linux OS distribution used by the Build system. However, none of the additional packages you add to the Build system is needed for the servers.


6

System Type

Description

Client Unlike the other two system types, Clients can use a variety of operating systems. Given the many client interfaces supported by OrangeFS, the OS running on your client system could be Linux, Windows or MacOS X. OrangeFS also has Web-based client interfaces, which are independent from the Client System OS.

Important Because clients have different requirements for the OrangeFS build system, please read through all installation instructions for the client(s) you plan to use BEFORE you build OrangeFS.

Note You can install OrangeFS on a single system, as long as that system meets the requirements of the build system.

System requirements information for the three system types is organized as follows:

Linux Operating System (page 6)

Additional_Linux_Software_for the_Build_System (page 7)

Berkeley_DB_Version_Support_for the_Server_System (page 9)

Hardware (page 9)

Protocols (page 9)

Considerations for Client_Systems and Interfaces (page 9)

Linux Operating System

For best practices, the same distribution of Linux should be installed on the Build system and any Server systems. This rule also applies for any Linux-based Client systems you add after the servers

have been installed.

In general, any Linux distribution with a kernel version 2.6 or later should support OrangeFS if used commonly across all servers.

Important OrangeFS is not currently compatible with SELinux, integrated into many Linux distributions, so disable it on all your Linux installations. If it is not disabled, you will get a

"permission denied" error when you try to run OrangeFS.

To disable SELinux, use the following command: echo 0 > /selinux/enforce

To prevent SELinux from loading at boot time, edit /etc/selinux/config and set the

SELINUX value to “disabled”, for example,

SELINUX=disabled

The command for disabling SELinux might vary, depending on your Linux version.

Release testing for OrangeFS typically includes recent versions of the following popular Linux distributions:

Fedora

OpenSUSE

RHEL/CentOS

RHEL/Scientific Linux

Ubuntu


7

Notes For a standard installation, all Linux clients across your OrangeFS installation must use the

same kernel version as the Build system. The simplest way to ensure this is to install a common distribution of Linux on all of your OrangeFS systems.

If you need to use multiple kernel versions among your OrangeFS clients, contact Technical Support for the additional steps required.

Additional Linux Software for the Build System

Package Listing by Distribution

The OrangeFS build system requires additional Linux packages that are not part of the base distributions. The following table describes these packages, including the specific names they use across popular distributions of Linux.

Package Description

Package Name for RHEL, CentOS, Fedora, SL

Package Name for OpenSUSE

Package Name for Ubuntu

GNU Compiler Collection gcc gcc gcc

Fast Lexical Analyzer Generator

flex flex flex

Bison Parser Generator bison bison bison

OpenSSL Development Libraries

openssl-devel libopenssl-devel libssl-dev

Berkeley DB Development Libraries

db4-devel libdb-4_8-devel libdb-dev

Kernel Module Builder kernel-devel kernel-source linux-source

Perl perl perl perl

GNU Make make make make

Kernel Headers kernel-headers kernel-devel, kernel-syms

kernel-headers

Zip zip zip zip

OpenSSL openssl openssl openssl

GNU Automake automake automake automake

GNU Autoconf autoconf autoconf autoconf

GNU Patch management patch patch patch

GNU Complier Collection - C++

gcc-c++ gcc-c++ g++

OpenLDAP* openldap libldap-2_4-2 libldap-2.4-2

OpenLDAP Development Libraries*

openldap-devel openldap2-devel libldap2-dev


8

Notes *OpenLDAP is required only if certificate-based security is used. See Security for more

information on security modes.

All packages for the Build system should match their associated libraries installed on Server

and Linux Client systems. For example, the version of openssl-devel on a Build system

should match the version of openssl on any targeted Server system. The easiest way to

ensure this is to use latest versions on all systems. Some packages automatically install additional software as dependencies. For example,

when you install gcc, it will automatically include glibc-devel and glibc-headers,

which must match the version of glibc on your servers and clients.

Package Installation

Following are example instructions for installing the required packages on a build system.

Notes This information is repeated under the Build and Configure (page 18) step later in this

Installation Guide.

These system commands require root privileges.

RHEL, CentOS, Fedora, SL

To automatically install the additional Linux packages on a system running RHEL, CentOS, Fedora or SL, enter the following command:

yum -y install gcc flex bison openssl-devel db4-devel kernel-devel perl make

kernel-headers zip openssl automake autoconf patch gcc-c++

Note If you do not plan to use certificate-based security, omit the last option (openldap-

devel) from the command.

OpenSUSE

To automatically install the additional Linux packages on a system running OpenSUSE, enter the

following series of commands:

zypper install -y gcc flex bison libopenssl-devel libdb-4_8-devel kernel-

source perl make kernel-devel kernel-syms zip openssl automake autoconf

patch gcc-c++

cd /usr/src/Linux-version

cp /boot/config-<version>-type .config

make oldconfig

make prepare

where...

version = version number in Linux source file name

Notes If you will not use certificate-based security, omit the option (openldap2-devel) from the

first command. The package name you include for Berkely DB Development Libraries in the above command is version-specific. It must match the package name version of Berkeley DB

installed on the system (libdb4_8 in this example).


9

Ubuntu

To automatically install the additional Linux packages on a system running Ubuntu, enter the following series of commands:

apt-get install -y gcc flex bison libssl-dev libdb-dev linux-source perl make

autoconf linux-headers-ùname -r` zip openssl automake autoconf patch g++

where...

version = version number in Linux source file name

Note If you will not use certificate-based security, omit the option (libldap2-dev) from the

first command.

Berkeley DB Version Support for the Server System

All servers in your OrangeFS installation must use Berkeley DB version 4.8.30 or later.

In the distributions of Linux supported by OrangeFS, Berkeley DB is included as a standard package. However, not all of those distributions use version 4.8.30 or later.

For detailed information and instructions, see Berkeley DB Version Support (page 126).

Hardware

In general, the hardware or virtual machine requirements for your base installation of Linux will also be sufficient for OrangeFS. For example, if you install Fedora 18 on a server or virtual machine that

meets the hardware requirements for Fedora 18, it will also meet the hardware requirements for OrangeFS.

Protocols

OrangeFS supports the three most common data communication protocols in parallel computing.

Select the one matching the network hardware layer for your servers:

If you have this network hardware... Then you will use this protocol...

Ethernet TCP/IP

Myrinet GM (older), MX (newer)

Infiniband IB

Other protocols

OrangeFS also supports portals and ZOID in specialized implementations for targeted platforms.

For more information, contact Technical Support.

Considerations for Client Systems and Interfaces


10

Among the three system types (Build, Server, Client) the Client system is the most flexible, with a

variety of Client Interfaces supported by OrangeFS.

Client systems are not limited to the Linux operating environment. Depending on the Client Interface you select, the operating system on your Client system can be Linux, Windows, MacOS X or even

Apache (web-based), as shown here:

Client Interface Client System (Operating Environment)

Kernel Module Linux

Direct Interface Linux

FUSE Linux and MacOS X

ROMIO (MPI-IO) Linux

Windows Client Windows

Apache WebDAV / S3 Multiplatform Web Access (OS-Independent)

Apache Hadoop Client Linux

The requirements for these client systems and interfaces can be addressed separately, as their instructions assume the file system servers are already installed and running.

Important Because clients have different requirements for the OrangeFS build system, please read

through all installation instructions for the client(s) you plan to use BEFORE you build OrangeFS.

Preview Security

This topic introduces the three modes of security for OrangeFS: default, key-based and

certificate-based. The one you select will affect the amount of preparation required before continuing with the installation process. Your choice will also affect settings made in the configuration file.

This topic is organized into three sections:

Introduction (page 10)

LDAP_Considerations (page 10)

Security Related_Tasks_During_Installation (page 14)

Introduction

During installation, you can select from three modes of system-wide security: default, key-based and certificate-based. All modes use standard file ownership and permissions to control access to

OrangeFS files and directories. POSIX access-control lists (ACLs) are also supported.

The security mode you select depends on your objectives and existing resources, as shown in the following descriptions.


11

Default Security Mode

This was the sole security mode used in releases of OrangeFS prior to version 2.9.

While it enforces standard file ownership and permissions, this mode is designed for trusted

computing environments and should be used in secure environments. This will prevent unintended use of the API at the Client level.

If you require enhanced security, you should use one of the other two modes.

Pros

Optimal performance

Fast installation

Best for evaluation and testing

Cons

Does not provide optimal security


12

Key-Based Security Mode

In key-based security mode, OrangeFS uses public and private key pairs to authenticate client systems.

Each server and Linux client has a key pair (a public and private key that are cryptographically related).

A file used by the servers, known as the keystore, contains public keys for all servers and clients. Each server and client has its own private key which is kept secret.

The keys and the keystore can be created together then copied the appropriate locations. This is accomplished during the OrangeFS installation in a temporary folder on the build system. If additional keys are needed after installation, a new keystore must be generated and copied to the appropriate

locations.

When a client sends a request to the server, it submits a credential object which is signed by its private key. The server verifies the signature using the known public key of the client.

Pros

Does not require LDAP

Less complex; fewer points of failure than certificate-based

Easier to install than certificate-based

Cons

Keystore is regenerated and

copied to all servers each time the list of keys changes

All servers are restarted each time

keystore changes


13

Certificate-Based Security Mode with LDAP

In certificate-based security mode, all servers share a common CA (certificate authority) certificate with which all other certificates are associated.

Each Linux user has a unique certificate with an associated private key file. These are created with OpenSSL, given an encrypted signature by the CA certificate and stored in the user's home directory.

The subject of the user certificate is mapped to a Linux UID/GID by the server, using an LDAP (Lightweight Directory Access Protocol) directory.

Each server knows where to reference the LDAP directory through an entry in the OrangeFS configuration file.

Pros

Ideal for environments where LDAP already exists

Easier to add users

Changes are dynamic (servers do not need

restarting)

Cons

Requires LDAP installation if it does not exist

More user tasks required to get

and store certificates

More complex; more points of

failure than key-based

LDAP Considerations

Prerequisite for Certificate-Based Security

If you plan to use certificate-based security, you must first establish a user directory that supports the Lightweight Directory Access Protocol (LDAP).


14

Any user who accesses OrangeFS with this security method must be included in the LDAP directory.

OrangeFS servers will locate this directory wherever you implement it on your network, through a simple configuration file entry.

If you need to create an LDAP directory, the Administration Guide includes guidelines and

considerations to optimize it for OrangeFS under Certificate-Based Security.

ID Mapping and LDAP Administration

Review the information in the Administration Guide about ID mapping and other administrative tasks related to your LDAP directory, such as adding and searching for users.

Security Related Tasks During Installation

The following table summarizes the security related tasks, for which instructions are provided throughout the Installation Guide.

Security Mode

Main Installation Step

Security Related Tasks

Default (None) (No additional tasks)

Key-Based

Build and Configure

1. Include an additional security related option in ./configure

command.

2. Create all keys and the keystore file in a temporary directory on the Build system, either manually or with provided scripts.

Tip This means you must know your server and client names ahead of time.

3. Copy keystore to etc directory in OrangeFS installation directory on

Build system.

4. Include all key-based security settings when you run pvfs2-

genconfig, the program that automatically generates an OrangeFS

configuration file.

Add Servers

From the Build system, copy a key to each server's default location, either manually or with provided script.

Add Clients From the Build system, copy keys to each Linux client's default location, either manually or with provided script.

Important When all keys have been distributed, if you decide to keep the temporary directory on the Build system where you created your key files, secure it appropriately using best

practices.


15

Security Mode

Main Installation Step

Security Related Tasks

Certificate-Based

Build and Configure

1. Include an additional security related option in ./configure

command.

2. Obtain an existing CA certificate (with key file) or create one.

3. Use chmod to limit CA certificate access.

4. Include all LDAP and certificate-based security settings when you run

pvfs2-genconfig, the program that automatically generates an

OrangeFS configuration file.

Add Clients 1. Create a certificate (with key file) for each user, either with the

application (pvfs2-get-user-cert) or the manual request

method.

2. Use chmod to limit certificate access.

Preview OrangeFS Configuration File

The OrangeFS configuration file is copied to all servers as a single reference point for operation and performance. You specify options and preferences for

your standard installation in this file. During installation, you are prompted for the most basic settings, so the file can be automatically generated.

After installation, you can revisit the configuration file to make more changes and additions from a rich selection of options (and their related values).

During standard installation, you use a program called pvfs2-genconfig to automatically generate

the OrangeFS configuration file. The program presents a series of prompts for basic required settings for an OrangeFS installation.

You can also run pvfs2-genconfig with command-line options. Run pvfs2-genconfig --help

for a list of the available options.

After installation, you can still make changes to the configuration file. A detailed reference topic is provided in the Administration Guide to help you in this process.

Important After installation, any time you change the configuration file, you must recopy it to all

servers in your OrangeFS installation and restart each server.

This topic has two sections:

Settings_Made_Automatically_during_Installation (page 15)

Settings_Made Manually after_Installation (page 16)

Settings Made Automatically during Installation

During installation, you will run the pvfs2-genconfig program, which collects your settings entries

to create the OrangeFS configuration file.


16

System Settings

The pvfs2-genconfig program first prompts you to enter some basic system settings, including the

following:

The protocol used by your network

The number of the port through which all servers will communicate

The hostname of each OrangeFS server

The directory and file locations where each OrangeFS server will store its log information, storage

data and storage metadata

Note Standard installation places file system storage directories inside the OrangeFS

installation directory under opt for portability. These directories can be located

elsewhere for system optimization and larger space allocations. The directory locations

can even be different for each server, if you edit the options in the <ServerOptions>

context of the OrangeFS configuration file. For more information see the Administration

Guide.

Security Settings

If you select either the key-based or certificate-based mode of security for your installation, pvfs2-

genconfig will prompt you for additional settings relating to those modes.

Key-Based Security

For key-based security, you must make these additional entries:

Server key file name and location

Keystore file name and location

Certificate-Based Security

For certificate-based security, you must make these additional entries:

CA certificate name and location

CA private key file name and location

Root DN for any existing user certificates

Number of days a user certificate is in effect (before expiration)

Location of the LDAP directory host

LDAP bind user DN format

LDAP bind password or password file path

LDAP search mode, search root object, search class, search attribute, search scope, UID attribute

and GID attribute

Settings Made Manually after Installation

After running pvfs2-genconfig, the OrangeFS configuration file is added to your installation

directory.

The configuration file is a simple text file that can be modified manually. It is organized into a number of option categories called contexts. Each context is bracketed by tags and includes a list of one or

more option-value pairs, as shown in this example:

<ContextName>

Option1Name Option1Value

Option2Name Option2Value

</ContextName>

While pvfs2-genconfig will query you about the most important options, many additional options

are assigned default values during installation. You can consider changes for most of these defaults


17

later after installation, when you can reference the Administration Guide for performance-tuning and

optimization.

Important After installation, any time you change the configuration file, you must recopy it to all

servers in your OrangeFS installation and restart each server.

OrangeFS Installation Instructions (2.9) Build and Configure

18

Build and Configure

The Build and Configure step involves downloading the source software from orangefs.org onto a

system preconfigured with several standard Linux packages.

On this system you will build OrangeFS into a portable directory named /opt/orangefs. From

there, you will complete the configuration and any setup required for your selected security mode.

When you are finished, you will have a complete installation isolated in a directory structure that can be copied to each of your Linux servers.

Build and Configure includes the following topics:

Build OrangeFS (page 18)

Setup Security (page 21)

Create OrangeFS Configuration File (page 29)

Review Results (page 32)

Important Because clients have different requirements for the OrangeFS build system, please read through all installation instructions for the client(s) you plan to use BEFORE you build OrangeFS.

Build OrangeFS

Building OrangeFS involves downloading the source software from orangefs.org

onto a system preconfigured with several standard Linux packages. On this system you will extract and build OrangeFS into a portable directory named

/opt/orangefs.

This topic provides the procedure for building OrangeFS.

System Requirements

In addition to a supported distribution of Linux, the OrangeFS Build system requires eight more Linux software packages. The names for these packages vary from one Linux distribution to another. For example, following are the package names you would require on a system running RHEL:

gcc

flex

bison

openssl-devel

db4-devel

kernel-devel

perl

make

The method for installing these packages varies among Linux distributions. For example, to automatically install the required packages on a system running RHEL, you could enter the following command:


openldap-devel

http://www.orangefs.org/


19

Notes If you do not plan to use certificate-based security, omit the last option (openldap-

devel) from the command.

For more details about supported Linux distributions and other required Linux packages,

see Preview System Requirements (page 5).

Procedure

Important Because clients have different requirements for the OrangeFS build system, please read

through all installation instructions for the client(s) you plan to use BEFORE you build OrangeFS.

To build OrangeFS, follow these steps:

1. Go to www.orangefs.org and download the compressed tar file into the /tmp/src directory (or

similar directory for temporary storage). The tar file is named as follows:

orangefs-version.tar.gz

where...

version = version number of the OrangeFS distribution release

Example: orangefs-2.9.tar.gz

2. Change Directory (cd) to /tmp/src, and extract the compressed tar file, then change to the

newly created orangefs directory:

tar -xzf orangefs-version.tar.gz

cd orangefs-version

Following is a sample listing of initial directories and files in the orangefs download directory:

/tmp/src/orangefs-version $ ls

aclocal.m4 COPYING Makefile.in SecuritySetup

AUTHORS CREDITS module.mk.in src

autom4te.cache doc patches test

cert-utils examples prepare windows

ChangeLog include pvfs2-config.h.in

configure INSTALL README

configure.in maint README.name_change

3. Build a Makefile for OrangeFS that includes the installation location and the path of the system

kernel, using the following command line format:

./configure --prefix=/opt/orangefs --with-kernel=kernel_path protocol_options

security_mode_option

where...

kernel_path = path to kernel source

Examples: /usr/src/kernels/2.6.18-194.17.1.el5-x86_64/

/lib/modules/ùname -r`/build

Note In the second example, ùname -r` will return the kernel version.



20

protocol_options = one of the following:

If your network protocol is... Include these options:

TCP None, enabled by default

IB, using Mellanox IB libraries --with-ib=/user --without-bmi-tcp

IB, using OFED --with-openib=/user --without-bmi-tcp

MX --with-mx/user=/user --without-bmi-tcp

GM --with-gm=/user --without-bmi-tcp

Note If you must run OrangeFS on more than one network protocol, please contact Technical Support.

security_mode_option = one of the following:

To use this security mode... Include this option:

Default None, enabled by default

Key-based --enable-security-key

Certificate-based --enable-security-cert

Example:

./configure --prefix=/opt/orangefs --with-kernel=/lib/modules/ùname -

r`/build --enable-security-cert

4. Continue with the standard Linux commands to build and run an executable program:

make

make install

5. Compile and install the kernel module that your OrangeFS Linux clients will need later.

make kmod

make kmod_prefix=/opt/orangefs kmod_install

Important OrangeFS is currently not compatible with SELinux, integrated into many Linux

distributions, so be sure to disable it on all your Linux installations. If it is not disabled, you

will get a "permission denied" error when you try to run OrangeFS.

To disable SELinux, use the following command:

echo 0 > /seLinux/enforce

To prevent SELinux from loading at boot time, edit /etc/seLinux/config and set the


SELINUX=disabled

The command for disabling SELinux can vary, depending on your Linux version.


21

Set Up Security (Build System)

After you build the OrangeFS installation directory, you must continue setup and configuration if you select either the key-based or certificate-based mode of security.

Much of this work can be done once on the build system, then copied to your servers and clients. In future versions of OrangeFS, security will be simplified.

In the Procedure (page 19) section of the previous topic, you specified a security mode when running

./configure (see step 3 (page 19) in Build OrangeFS).

Depending on the mode you chose (default, key-based or certificate-based), refer to the appropriate sections in this topic for additional security setup for the Build system:

Default_Mode (page 21)

Key-Based_Mode (page 21)

Certificate-Based_Mode (page 23)

Default Mode

If security is not a priority, you might have selected the default mode for optimal performance and faster installation. This mode does not require any additional setup, so you can go to the next topic,

Create OrangeFS Configuration File (page 29).

Key-Based Mode

If you selected the key-based mode, you must create your security keys and a keystore file in a temporary directory on the Build system. You must then copy the keystore file to the OrangeFS installation directory.

Notes To complete this procedure, you must know the host names of your OrangeFS servers and

clients.

This procedure assumes the use of an automated script provided with your OrangeFS files. To learn how to create your security keys and keystore manually, see the Administration Guide.

Procedure

The following steps set up the Build system for key-based security. They assume the OrangeFS source

is in /tmp/src/orangefs-version.

1. Create a temporary directory on the Build system, located outside the /tmp/src/orangefs-

version source directory:

cd /opt

mkdir ofs_keys

Note Later, after you have distributed key pairs to your OrangeFS servers and clients, you should either delete or limit access to this directory. If you keep this directory for future changes, secure it appropriately using best practices.

2. Change Directory (cd) to the new directory and copy two script files from /tmp/src/orangefs-

version/examples/keys:

cd ofs_keys

cp /tmp/src/orangefs-version/examples/keys/*.sh .


22

Note You will use only one of these scripts now. You will use the second one when you add

OrangeFS servers later.

3. With the script named pvfs2-gen-keys.sh, use the following command line format to generate

private keys for servers and clients, as well as the keystore:

./pvfs2-gen-keys.sh [-a] [-s servers] [-c clients]

where...

servers = server hostname(s), each separated by a space

Example: orangefs01 orangefs02 orangefs03

clients = client hostname(s), each separated by a space

Example: orangefs01 client01 client02

Note As this example suggests, an OrangeFS server can also be a client.

Example of full command:

./pvfs2-gen-keys.sh -s orangefs01 orangefs02 -c orangefs01 orangefs02

The executed script will generate:

The keystore, named orangefs-keystore by default, is a text file that contains the public keys

for each server and client.

Key File Type

File Name Format Example

Server orangefs-serverkey-

hostname.pem

orangefs-serverkey-

orangefs01.pem

Client pvfs2-clientkey-hostname.pem pvfs2-clientkey-client01.pem

Note The -a option shown in the command line format does not apply during initial

installation. Include this option only if you want the public keys to be appended to an

existing keystore (named keystore by default).

4. Copy the keystore to the etc directory in your OrangeFS installation directory:

cp keystore /opt/orangefs/etc

Note This is the default location for the keystore on all OrangeFS servers. If you specify a different location in the above copy command, you must reflect that change later when you create the OrangeFS configuration file.

Generating Keys for Many Systems

The command line format used in step 3 above can be modified for large numbers of servers and clients, using shell expansion. For example, the following command generates server keys for


23

orangefs-server01 to orangefs-server04 and client keys for orangefs-client01 to

orangefs-client40:

./pvfs2-gen-keys.sh -s orangefs-server0{1..4} -c orangefs-client0{1..9} orangefs-

client{10..40}

See your shell documentation for more information.

Certificate-Based Mode

If you selected the certificate-based mode of security, you must add a CA certificate to the OrangeFS directory on the Build system.

If you already have one you want to use, simply copy the certificate file, along with its private key file,

to /opt/orangefs/etc. Each of the files should be in PEM format (see OpenSSL documentation).

If you need to create a CA certificate, the OrangeFS installation files include some tools to simplify the process. You must have a working knowledge of OpenSSL to tailor your certificate settings beyond the

basic procedure that follows.

Procedure

OpenSSL references a configuration file when it creates certificates, including CA certificates.

Note This file is specifically tied to OpenSSL; it is different from the OrangeFS configuration file.

The default location for this file on the Build system is /etc/ssl/openssl.cnf, but the following

procedure uses an alternative configuration file named orangefs.cnf. That file is located in

/opt/orangefs/examples/certs, and it includes basic "quick start" settings that you can modify

as needed.

Note For complete information on the OpenSSL configuration file format, see the config(5ssl)

Linux man page.

To create a CA certificate (using the example configuration file):

1. Change Directory (cd) to the directory where the example configuration file is located:

cd /tmp/src/orangefs-version/examples/certs

2. If necessary, customize the settings in the configuration file (orangefs.cnf) to reflect the

security settings and policies of your organization.

3. Enter the following command:

openssl req -config orangefs.cnf -new -x509 -outform PEM -out orangefs-ca-

cert.pem -keyout orangefs-ca-cert-key.pem -nodes -days 1825

Notes You can use different file names. You can also select a different expiration; the above example expires in 5 years (1825 days)

The documentation for this command is in the req(1) Linux man page.

You are prompted for configuration values after entering this command.

4. Enter the elements of the CA certificate subject.


24

The configuration file will prompt you for country, state, locality, organization, organizational unit

and common name. You and your security administrator might want to discuss the values any existing certificates use and follow a similar format.

When you submit the entries, the CA certificate and private key you specified (orangefs-ca-

cert.pem and orangefs-ca-cert-key.pm in the example above) will be generated in the

current directory.

5. Move the CA certificate and private key files to the etc subdirectory in your OrangeFS installation

directory:

mv *.pem /opt/orangefs/etc

Using the Script File

The examples/certs directory in your OrangeFS source directory also includes a script (pvfs2-

cert-ca.sh) to streamline the above procedure. Its command line format includes a single optional

parameter for any characters you want to add to the certificate file names.

For example, to achieve the same results as in the above procedure, you would enter:

./pvfs2-cert-ca.sh orangefs

Restricting Access

Be sure to use chmod to restrict access to the CA key.

Configuring LDAP for Identity Mapping

When an OrangeFS server receives a certificate from a client, it performs identity mapping with the

certificate. The certificate contains a subject distinguished name (DN) to identify it, while the server needs a numerical user ID (UID) and primary group ID (GID). In order to do the mapping, an LDAP directory is used. The subject DN is transformed in a configurable way to locate a user object in the LDAP directory; the object contains the UID and GID.

OrangeFS is designed to use OpenLDAP client libraries, which are available for most distributions. The

OrangeFS server can communicate with an OpenLDAP server or a standard LDAP server from another organization.

For more information on LDAP see http://openldap.org.

Planning for LDAP Identity Mapping

First, identify which users will be allowed to use OrangeFS. These users will require user certificates and must have a user object in the LDAP directory. Information on creating users in LDAP is provided below.

You might be able to leverage an existing LDAP directory. Use the information below to evaluate how

existing LDAP user objects can be utilized.

The next step is to identify a string to be uniquely associated with each user. The most obvious is the

login name, the first field of /etc/passwd, with which users log in. However, if you have existing

LDAP users, use their naming attribute values (often the “CN” or “UID” attribute). The description

field of /etc/passwd could also be used. Any string will work as long as it is unique to each user.

This value is the “user name”.

Determine the naming attribute for user objects in your LDAP directory. When creating a new LDAP directory, Common Name (CN) is a good choice.

http://openldap.org/


25

Next, determine where in LDAP the users will be, or are, stored. LDAP directories are hierarchical

trees, where objects are identified by distinguished names (DNs). A DN consists of segments in the

form “attribute=value”, separated by commas. The DN “ou=Users,dc=acme,dc=com” indicates

an organizational unit (OU) named Users under the acme domain context (DC), which in turn is in the com DC. Objects containing other objects are called containers; some typical container classes are domain contexts (DC), organizations (O) and organizational units (OU). Often the DNS name of an organization is used to form the domain contexts at the root of the directory, for example acme.com

becomes “dc=acme,dc=com”.

Determine the DN of the container that contains all the users to enable for OrangeFS. In some cases the users are in multiple containers; if so, select the container at the “highest” point that contains all subcontainers with users. For example if users are in both “ou=Engineering,ou=Users...” and “ou=Sales,ou=Users...”, make a note of “ou=Users” as the container. Also note whether the users are in one container or multiple containers.

Finally, you must know where the UID and GID values are stored in LDAP. Objects in LDAP have named attributes, which can have one or more values. The default attributes that store the UID and GID are uidNumber and gidNumber. If you are using the OpenLDAP server, use the schema

file nis.schema to enable these attributes. (See the OpenLDAP documentation for more information.)

The list below summarizes information needed to configure OrangeFS for LDAP identity mapping.

1. Which users to enable for OrangeFS

2. A user name to uniquely identify each user. For existing LDAP installations, this should correspond to the naming attribute of the existing user objects (often “CN” or “UID”).

3. The naming attribute used for user objects in LDAP, often Common Name (CN) or UID.

4. The DN of the LDAP container where user objects are stored. Users can be stored in one container or multiple containers.

5. The names of the UID- and GID-storing attributes, usually uidNumber and gidNumber.

Planning for LDAP Binding

"Binding" means connecting and authenticating to an LDAP server. You must have the following information to bind to your LDAP server:

URI(s) for the LDAP server(s). These URIs are in form “ldap[s]://hostname[:port]”. Using

“ldap” specifies a plaintext connection, and “ldaps” specifies a secure (usually SSL) connection.

The default port is 389 for plaintext, and 636 for secure. You can have multiple LDAP servers for the same directory—specify any of these.

DN of a binding user. The user must have sufficient rights to search for users in their specified

container, and to read their uidNumber and gidNumber attributes. This value is optional, as anonymous binds are possible. The administrator must ensure that anonymous binds do not have excess rights.

Password of the binding user. This value can be stored in a protected file for additional security.

This value is optional, as users are not required to have passwords.

Because the password is not encrypted, a user should be created for OrangeFS usage with only the rights described above.

Server Configuration File Settings

The LDAP settings are specified in the OrangeFS configuration file, which is identical for each server.

The <LDAP> tag within the <Security> tag contains the settings:


26

<Defaults>

. . .

<Security>

. . .

<LDAP>

[Hosts {list of LDAP URIs}]

[BindDN {DN}]

[BindPassword {password} or {file:path}]

[SearchMode “CN” or “DN”]

[SearchRoot {DN}]

[SearchClass {Class name}]

[SearchAttr {Attrname}]

[SearchScope “onelevel” or “subtree”]

[UIDAttr {Attrname}]

[GIDAttr {Attrname}]

[SearchTimeout {timeout (secs)}]

</LDAP>

</Security>

. . .

</Defaults>

The settings are defined below.

Setting Default

Hosts: a list of LDAP URIs separated by spaces, for example

“ldaps://myhost.org”.

“ldaps://localhost”.

BindDN: an LDAP DN specifying the user that will connect to LDAP will bind anonymously

BindPassword: the password for the binding user, or the string “file:”

followed by a path to a file from which to read the password.

no password

SearchMode: “CN” or “DN”. See below for more information. “CN”

SearchRoot: the DN of the container with the user objects.

Note You must specify this value if you are using an OpenLDAP server.

the root of the directory

SearchClass: the object class of the user objects. “inetOrgPerson”

SearchAttr: the naming attribute to match against the certificate CN. “CN”

SearchScope: “onelevel” or “subtree”. Whether to search only the

SearchRoot container (“onelevel”) or that container and all child

containers (“subtree”).

“subtree”

UIDAttr: the name of the UID-storing attribute. “uidNumber”

GIDAttr: the name of the GID-storing attribute. “gidNumber”

SearchTimeout: timeout in seconds for LDAP searches. “15”

You should have noted these values during “Planning for LDAP Binding” described above.

Searching LDAP for Identities

The OrangeFS server searches LDAP for the user object based on the user certificate’s subject DN.

If the SearchMode is “CN”, the CN (common name) of the certificate subject is used. It must match

an object meeting these criteria:

1. It is in or under the SearchRoot container (depending on SearchMode).

http://www.omnibond.com/orangefs/docs/v_2_9/admin_Certificate-Based_Security.htm#Planning_for_LDAP_Identity_Mapping


27

2. It has an object class equal to the SearchClass

3. It has its SearchAttr attribute matching the certificate CN. The search filter used is:

(&(objectClass={SearchClass})({SearchAttr}={Certificate CN}))

The UID and GID will be retrieved from the UIDAttr and GIDAttr attributes of the object. This UID

and GID will be used for subsequent file system operations. If this search fails, an error will be printed

to the server log and “operation not permitted” returned to the client.

If the SearchMode is “DN”, the certificate subject DN must match the LDAP user object DN exactly

(case-insensitive). In this mode, SearchRoot, SearchClass, SearchAttr and SearchScopeare

not used.

OrangeFS will retry the connection if it can’t contact the LDAP server. It will try different servers on

the URI list.

LDAP and System Identities

You can specify that an LDAP user object have a different UID/GID from its corresponding system

user. For example, the system user “jsmith” can have UID/GID 500/100, but the LDAP user

corresponding to “jsmith” might have UID/GID 550/500. However, OrangeFS utilities will still show

the system login name associated with the OrangeFS UID/GID. In our example, OrangeFS utilities

display files as owned by system UID 550 rather than “jsmith”. If you are using nsswitch (Name

Service Switch) with LDAP you will not have this conflict. Otherwise, it is not recommended that the identities have mismatching UID/GIDs.

Creating a New LDAP Directory

The examples/certs directory included in the distribution contains scripts and files that can be used

to create a new OpenLDAP directory.

The script pvfs2-ldap-create-dir.sh will create a new OpenLDAP directory and add some basic

objects. Usage of the script is:

./pvfs2-ldap-create-dir.sh [-p {prefix}] [-a {admin dn}] [-s {suffix dn}] [-w

{admin password}]

prefix: base directory for OpenLDAP installation, default /usr/local

admin dn: DN of LDAP administrator; should end with suffix DN, default cn=admin,{suffix

dn}

suffix dn: base (topmost) DN of directory; default based on hostname, for

example hostname acme.com would give dc=acme,dc=com

admin password: LDAP administrator password, default “ldappwd”.

The script will create the new LDAP directory and add two organizational units, named "Users" and

"Groups." A user object for the system root account will be created with a random password. See

“Adding Users to LDAP (page 28)” below for information on changing the password.

Important The directory created is not secure. User passwords are stored in plaintext, and

SSL/TLS security is not enabled. The directory should only be used for testing, or as a starting point for a secure directory.

Consult the OpenLDAP documentation for information on securing the directory.

These statements in the OrangeFS configuration file will configure this directory.


28

<Defaults>

. . .

<Security>

. . .

<LDAP>

Hosts ldap://{hostname}

BindDN {admin dn}

BindPassword {admin password}

SearchRoot cn=Users,{suffix dn}

SearchScope onelevel

</LDAP>

</Security>

. . .

</Defaults>

Substitute the values in braces for the values used when creating the LDAP directory. All unspecified values are equal to the defaults.

Adding Users to LDAP

The ldapadd utility is used to add objects, including users, to an LDAP directory. LDAP utilities use

LDIF files to describe objects. Consult the LDIF RFC (http://www.ietf.org/rfc/rfc2849.txt) for more

information on the LDIF file format.

In examples/certs, the script pvfs2-ldap-add-user.sh will create a user based on the

information for that user in /etc/passwd:

./pvfs2-ldap-add-user.sh [-D {admin dn}] [-w {admin pw}] {logon name} {container

dn}

The script will create a user with the CN equal to the logon name, located in the specified container.

The uidNumber, gidNumber, displayName, homeDirectory and login shell attributes will be set to

correspond to the system account fields (displayName corresponds to description). A random

password will be created.

To change a user password, the ldapmodify utility is used. A wrapper script is provided

in examples/certs:

./pvfs2-ldap-set-pass.sh [-D {admin dn}] [-w {admin pw}] {user dn} {password}

For example:

./pvfs2-ldap-set-pass.sh -D cn=admin,dc=acme,dc=com -w ldappwd

cn=jsmith,ou=users,dc=acme,dc=com ‘sEcr3t!’

The script will store the password in LDAP in an encrypted format, using the slappasswd utility.

http://www.ietf.org/rfc/rfc2849.txt


29

Create OrangeFS Configuration File

Procedure

The OrangeFS installation directory on the Build system will need an OrangeFS configuration file. You will enter basic information for this file in a

program called pvfs2-genconfig. Once the configuration file has been

created, you might need to make additional modifications (regarding security, for example) for the initial deployment.

Note After standard installation, consult the Administration Guide for

details on the options and values available for fine tuning the configuration file.

To create the OrangeFS configuration file, follow these steps:

1. Using pvfs2-genconfig, the configuration file will be automatically generated as follows:

/opt/orangefs/bin/pvfs2-genconfig /opt/orangefs/etc/orangefs-server.conf

The program presents a series of prompts to enter the required settings for your OrangeFS configuration file.

2. Answer all the prompts to generate the configuration file in the etc directory. Following is a list of

possible entries for these prompts:

Option/Setting Default/Example Value Description

Protocol tcp Protocol choices are tcp, gm, mx,

ib and portals. See Preview

System Requirements (page 5)

for more details. Default is tcp.

For multi-homed configurations, separate multiple protocols with

commas, for example ib, tcp

(tcp) Port number

3334 TCP/IP port number that each OrangeFS server will listen on.

Default is 3334.

(gm) Port number

6 GM port number (in the range of 0 to 7) that each OrangeFS server

will listen on. Default is 6.

(mx) Board number

0 MX board number (in the range of 0 to 4) that each server will listen

on. Default is 0.

pvfs2-genconfig assumes that

all servers will use the same board number.


30


Endpoint number

3 MX endpoint (in the range of 0 to 7) that each server will listen on.

Default is 3.


all servers will use the same endpoint number.

(ib) Port number

3335 TCP/IP port that each server will listen on for IB communications.

Default is 3335.


all servers will use the same port

number.

(portals) Portal

index

5 Portal index that each server will

listen on for portals

communications. Default is 5.


all servers will use the same portal index.

Data Directory /opt/orangefs/storage/data Full path + directory name where each OrangeFS server will store its data.

Metadata Directory /opt/orangefs/storage/meta Full path + directory name where

each OrangeFS server will store its metadata.

Log Directory /var/log/orangefs-server.log Full path + file name where each server will write its log messages.

Server Host Names Default: localhost

Example: ofs{1-4}

Hostname of each OrangeFS data server (server on which data directory is located). This should

be the value returned by the

hostname command. Syntax is

node1, node2, ... or

node{#-#,#,#}.

Data/Metadata Allowed yes Enter yes to keep metadata

directory on same server as data

directory. If you enter no, you

are prompted to enter additional hostnames.


31


Security Options Options that display for security are based on the security mode you chose when you built OrangeFS. The three security

modes are default, key-based or certificate-based.

Default NA NA If you select this security mode, you will not be prompted with any more security options.

Key Server key file

location

/opt/orangefs/etc/orangefs-

serverkey.pem Full path + file name where each server will store its public server

key.

Keystore location

/opt/orangefs/etc/orangefs-keystore Full path + file name where each server will store its keystore.

Certificate CA certificate file

/opt/orangefs/etc/orangefs-ca-

cert.pem

Full path + file name where each server will store a copy of the CA certificate.

CA private key file

/opt/orangefs/etc/orangefs-ca-cert-

key.pem Full path + file name where each server will store its private key.

User

certificate root DN

C=US, O=OrangeFS The distinguished name (DN) for

any existing user certificates in your LDAP setup.

User certificate

expiration

365 Enter the number of days a user certificate is in effect (before

expiration)

LDAP host list

ldap://localhost Enter the LDAP host or list of hosts. Syntax is ldap[s]://host[:port],...

LDAP bind user DN

cn=admin,dc=acme,dc=com Enter the LDAP bind user's DN. By default, will bind anonymously.

LDAP bind password

password or

passwd:/opt/orangefs/etc/ldappwd.txt

LDAP bind password or

passwd:file_path to specify

text file with password.

LDAP search mode

CN Enter either CN or DN for the

LDAP search mode.


32


LDAP search root

ou=users,dc=acme,dc=com Enter the LDAP search root object's DN.

LDAP search class

inetOrgPerson Enter the name of the LDAP search class.

LDAP

search attribute

cn Enter the LDAP search attribute.

LDAP

search scope

subtree Enter either onelevel or

subtree for the LDAP search

scope.

LDAP UID

attribute

uidNumber Enter the LDAP UID attribute.

LDAP GID attribute

gidNumber Enter the LDAP GID attribute.

Verify Server List n Asks (y/n) if you want to

redisplay the server hostnames you entered.

Note Standard installation, as configured above, places file system storage directories inside

the OrangeFS installation directory under opt for portability. These directories can be

located elsewhere for system optimization and larger space allocations. For detailed information on all options in the OrangeFS configuration file, see the Administration Guide.

When you are finished running pvfs2-genconfig, the OrangeFS configuration file is added to

/opt/orangefs/etc.

The configuration file is a simple text file that can be opened and modified manually. While

pvfs2-genconfig will query you about the most important options, default values are assigned

to many additional options. You can consider changes for most of these defaults later after installation, when you can reference the Administration Guide for performance-tuning and optimization.

Results (Build and Configure)

At the end of the Build and Configure step, the build system will include:

An installation directory (/opt/orangefs)

A configuration file (/opt/orangefs/etc/orangefs-server.conf) to be used by all servers

associated with this installation.

If you chose key-based security mode, the build system will also include:

A temporary directory where all keys and the keystore file were generated.


33

A copy of the keystore file in the installation directory (opt/orangefs/etc/keystore).

If you chose certificate-based security mode, the build system will also include a CA certificate and

key (orangefs-ca-cert.pem and orangefs-ca-cert-key.pem in /opt/orangefs/etc).

Installation Directory

Following is a top-level list of the orangefs installation directory:

/opt/orangefs $ ls

bin include lib sbin share etc log

For file listings of all directories and subdirectories, see Directory/File Listing (page 127).

Configuration File

Following is a sample OrangeFS configuration file:


34

/opt/orangefs/etc $ cat orangefs-server.conf

<Defaults>

UnexpectedRequests 50

EventLogging none

EnableTracing no

LogStamp datetime

BMIModules bmi_tcp

FlowModules flowproto_multiqueue

PerfUpdateInterval 1000

ServerJobBMITimeoutSecs 30

ServerJobFlowTimeoutSecs 30

ClientJobBMITimeoutSecs 300

ClientJobFlowTimeoutSecs 300

ClientRetryLimit 5

ClientRetryDelayMilliSecs 2000

PrecreateBatchSize 0,32,512,32,32,32,0

PrecreateLowThreshold 0,16,256,16,16,16,0

DataStorageSpace /opt/orangefs/storage/data

MetadataStorageSpace /opt/orangefs/storage/meta

LogFile /var/log/orangefs-server.log

</Defaults>

<Aliases>

Alias tweeks tcp://tweeks:3334

</Aliases>

<Filesystem>

Name orangefs

ID 1600781381

RootHandle 1048576

FileStuffing yes

DistrDirServersInitial 1

DistrDirServersMax 1

DistrDirSplitSize 100

<MetaHandleRanges>

Range tweeks 3-4611686018427387904

</MetaHandleRanges>

<DataHandleRanges>

Range tweeks 4611686018427387905-9223372036854775806

</DataHandleRanges>

<StorageHints>

TroveSyncMeta yes

TroveSyncData no

TroveMethod alt-aio

</StorageHints>

</Filesystem>

If you enabled key- or certificate-based security, a <Security> context will also be in the

configuration file.

Here is an example <Security> context for key-based security:


35

<Defaults>

. . .

<Security>

ServerKey /opt/orangefs/etc/orangefs-serverkey.pem

Keystore /opt/orangefs/etc/keystore

</Security>

. . .

</Defaults>

. . .

Here is an example <Security> context for certificate-based security:

<Defaults>

. . .

<Security>

CAFile /opt/orangefs/etc/orangefs-ca-cert.pem

ServerKey /opt/orangefs/etc/orangefs-ca-cert-key.pem

<LDAP>

Hosts ldap://ldap01.acme.com

BindDN cn=ofsadmin,dc=acme,dc=com

BindPassword file:/opt/orangefs/etc/ldappw.txt

SearchRoot ou=OrangeFS-Users,dc=acme,dc=com

SearchMode CN

SearchClass inetOrgPerson

SearchAttr CN

SearchScope subtree

UIDAttr uidNumber

GIDAttr gidNumber

SearchTimeout 10

</LDAP>

</Security>

. . .

</Defaults>

. . .

The <Security> context can also be specified in a <ServerOptions> context for different settings

on each server.

For more information, see OrangeFS Configuration File in the Administration Guide.

OrangeFS Installation Instructions (2.9) Add Servers

36

Add Servers

You add servers by copying the OrangeFS installation directory from the Build system to each Server

system designated in your OrangeFS solution. You must also consider additional setup and configuration

tasks for security. Then you can start up each server you have added.

The OrangeFS servers make up your actual file system. The servers provide the space across which all data and metadata are distributed and managed for optimal storage and retrieval.

Adding servers includes the following topics:

Copy OrangeFS Installation Directory (page 36)

Set up Security (page 37)

Run (page 38)

Review Results (page 39)

Copy OrangeFS Installation Directory (Server Systems)

Begin your deployment by copying the OrangeFS installation directory from the Build system to each Server system designated in your OrangeFS solution. These are the servers already identified in the OrangeFS configuration file.

System Requirements

Any system that functions as an OrangeFS server requires a supported distribution of Linux.

Note For more information on supported distributions, see Preview System Requirements (page 5).

Procedure

To add the required software to an OrangeFS server, copy the /opt/orangefs directory from the

Build system:

scp –r /opt/orangefs hostname:/opt

where...

hostname = host name of the Server system


37

Important OrangeFS is currently not compatible with SELinux, integrated into many Linux

distributions, so be sure to disable it on all your Linux installations. If it is not disabled, you

will get a "permission denied" error when you try to run OrangeFS.

To disable SELinux, use the following command:

echo 0 > /seLinux/enforce

To prevent SELinux from loading at boot time, edit /etc/seLinux/config and set the


SELINUX=disabled

The command for disabling SELinux can vary, depending on your Linux version.

Set up Security (Server Systems)

To use key-based security mode, you must copy the private keys you generated on the Build system to each of your Server systems. In future versions of OrangeFS, security will be simplified.

Note Neither the default nor certificate-based modes of security require any additional setup on

Server systems. For these modes, see the next topic, Run (page 38).

Procedure

Copying Keys Manually

To add a private key to an individual Server, copy the private key file from /opt/ofs_keys on the

Build system to the Server system:

scp –r /opt/ofs_keys/orangefs-serverkey-hostname.pem

hostname:/opt/orangefs/etc/orangefs-serverkey.pem

where...

hostname = host name of the Server system

Note The above command line format assumes you generated your keys according to

instructions in Set Up Security (page 21) under the Build and Configure (page 18) step.

Copying Keys to Many Systems

You can use a script file (provided with OrangeFS) to copy all of your private keys with one command if both the following statements are true:

1. You have already copied the OrangeFS installation directory to all of your designated servers and any additional Linux systems on which you plan to use an OrangeFS client interface.

2. You generated your security keys during the Build and Configure (page 18) step, using the

script provided with OrangeFS (pvfs2-gen-keys.sh).

Note For more information about client interfaces, see Add Clients (page 40).

If both the above statements are true, you can add private keys to all your Linux-based OrangeFS systems as follows:


38

1. Change Directory (cd) to the /opt/ofs_keys directory on the Build system:

cd /opt/ofs_keys

2. If you followed the security setup instructions under the earlier Build and Configure step, the script file should be in the current directory and you can skip this step. Otherwise, copy the script from the OrangeFS source directory as follows:

cp /tmp/src/orangefs-version/examples/keys/pvfs2-dist-keys.sh

3. With the script named pvfs2-dist-keys.sh, use the following command format to copy private

keys to all OrangeFS Linux systems:

./pvfs2-dist-keys.sh orangefs_install

where...

orangefs_install = the location of the OrangeFS installation directory

Example: /opt/orangefs

Example of full command:

./pvfs2-dist-keys.sh /opt/orangefs

If the above example is used, the script examines the key filenames to determine the hostname for each target server or client, then secure-copies (scp) each key accordingly to the

/opt/orangefs/etc directory of the relevant system.

Note The script assumes that the specified OrangeFS installation directory already exists on all of the targeted systems. The above example uses the default location for the

instructions in this Installation Guide.

Run (Server Systems)

You will use the OrangeFS server daemon (pvfs2-server) on each server to

initialize the storage directories and start the OrangeFS server process.

Procedure

Running the server involves two tasks:

Initializing the working directories on each server for storage space

Starting the server process

The first task must be performed once on each server. Thereafter, you can start and stop the server process with a single command. Both tasks are accomplished with a command line statement that

includes the OrangeFS server daemon (pvfs2-server), located in the OrangeFS installation directory

under sbin.


39

Initialize the Storage Directories

To initialize the storage directories, run the following command on each server:

/opt/orangefs/sbin/pvfs2-server -f -a hostname /opt/orangefs/etc/orangefs-

server.conf

The -f option indicates that the file system storage directories should be initialized. This command

creates the storage directories using the locations provided in the orangefs-server.conf file. The

storage directories created will be /opt/orangefs/storage/{data,meta} with additional

subdirectories under both storage locations.

Notes Because each server has different data based on the handle ranges provided with the

orangefs-server.conf file, do not copy one set of databases to each server. You must

create them separately on each server. The storage space on each OrangeFS server must be initialized only once.

You can change the locations for storage directories by manually adding the

<ServerOptions> section in OrangeFS configuration file. With this method, you can

specify unique directory locations for each server. For detailed information on all options in

the OrangeFS configuration file, see the Administration Guide.

Start the Server Process

To start the server process, enter the following command:

/opt/orangefs/sbin/pvfs2-server -a hostname /opt/orangefs/etc/orangefs-server.conf

Starting the Server Process Automatically

To avoid repeating this command each time you reboot an OrangeFS server, you can place the

statement in the appropriate system file(s) for automatic execution. For more information, see

Automating System Startup (page 130).

Stopping the Server Process

To stop the server process, enter the following command:

killall pvfs2-server

Results (Add Servers)

At the end of the Add Servers step, each Server system will include an OrangeFS installation directory

(/opt/orangefs).

If you are using key-based security mode, each Server system will also include its own private key in

/opt/orangefs/etc.

OrangeFS Installation Instructions (2.9) Add Clients

40

Add Clients

Client access to OrangeFS is flexible, with support for a variety of operating environments and interfaces.

Depending on the Client Interface you select, your Client system might run Linux, Windows, MacOS X or even Apache (web-based), as shown here:

Client Interface Client System (Operating Environment)

Kernel Module Linux

Direct Interface Linux (Kernel bypass)

FUSE Linux and MacOS X

ROMIO (MPI-IO) Linux

Windows Client Windows

Apache WebDAV / S3 Multiplatform Web Access (OS-Independent)

Generally, the requirements for these client solutions can be addressed separately, as their instructions assume the file system servers are already installed and running.

This topic further summarizes client options in two parts:

Client_Interface_Matrix (page 40)

Client_Architecture_Diagram (page 42)

Client Interface Matrix

Click on any client interface for installation information.

Client Interface

Description OS Typical Uses Advantages

Kernel

Module (page 44)

Enables access to OrangeFS through the native Linux operating environment.

Linux For Linux users who wish to access OrangeFS as a mounted file system, using standard tools like ls, cp and rm.

Supports standard out-of-the-box Linux kernels.


41

Client Interface


Direct

Interface (page 49)

Provides program libraries that allow developers to call standard functions

(open, close, read, write, etc.) that communicate with OrangeFS servers directly, bypassing the Linux kernel.

Linux For advanced users who can benefit from higher access speeds or targeted

programming access to OrangeFS.

Bypasses the Linux Kernel for improved performance. Interoperability

between application programs.

Provides three levels of access in varying performance versus ease-of-use

combinations.

FUSE (page

52) Allows access to OrangeFS file systems through FUSE (Filesystem in

Userspace), which provides its own kernel module/driver to mount a file system.

Linux For FUSE interface users who want to use OrangeFS for their file system

storage.

Adds performance advantages of OrangeFS to a FUSE front end.

Note Primarily for Mac users. Linux users will achieve better results with

the Kernel Module and Direct Interface.

ROMIO

(MPI-IO) (page 56)

ROMIO is an implementation of the MPI-IO protocol that includes support for OrangeFS.

Linux Access to OrangeFS in programs and operations optimized for parallel computing.

Any MPI Library implementation that works with ROMIO (such as MPICH and OpenMPI) can also work with OrangeFS.

Windows

Client (page 59)

Enables access to OrangeFS through Microsoft Windows environment.

Windows Public disk storage for Windows-based cluster.

Native, transparent access to OrangeFS from Windows.

Hadoop

Client (page 93)

Enables MapReduce, the processing engine for

Hadoop, to replace its standard file system (HDFS) with OrangeFS.

Java/Linux For Hadoop-based, data-intensive,

distributed applications.

Improves MapReduce performance and

provides more ways to leverage data with the OrangeFS feature set.


42

Client Interface


WebDAV

Apache

Module (page 80)

Part of OrangeFS Web Pack (separate download). Allows any WebDAV client access to

OrangeFS via an Apache server.

Apache Access OrangeFS data via HTTP.

Native WebDAV access to OrangeFS.

S3

Apache

Module (page 80)

Part of OrangeFS Web Pack (separate download). Allows any

S3 client access to OrangeFS via an Apache server.

Apache For using S3 services in a private cloud, free from storage

and bandwidth costs.

Numerous client tools already exist.

Client Architecture Diagram

The following diagram depicts the primary OrangeFS components that enable each client interface to connect to the file system.


43

pvfs2tab File

Each client must know where to access OrangeFS resources. The pvfs2tab file, similar to the /etc/fstab file in Linux, provides clients with this access information. It involves creating a file at a designated path, which will function as the gateway to your OrangeFS installation.

1. Determine the URL of the OrangeFS server you will access.

You can retrieve this information from the orangefs-server.conf file. For example, the first

URL listed in that file can be extracted with the following command:

grep "Alias " /opt/orangefs/etc/orangefs-server.conf | awk '{ print $3 }' | head

-n 1

The format to use for server URL is protocol://hostname:port.

Example: tcp://server1:3334

2. Create a file named pvfs2tab in the system's /etc directory that tells the system how to access

OrangeFS:

echo "tcp://server1:3334/orangefs /mnt/orangefs pvfs2 defaults,noauto 0 0" >>

/etc/pvfs2tab

Note In the above example, tcp: is the network protocol, //server1 is the server

providing access to the configuration file, and 3334 is the number of the TCP/IP port

on which the OrangeFS servers communicate, which was determined in step 1;

/mnt/orangefs is the path you use to access these files. You can think of

/mnt/orangefs as the root directory of the OrangeFS file system.

3. You must also assign read-access to the new file:

chmod a+r /etc/pvfs2tab

4. If you want to use an alternative file path instead of the standard location of /etc/pvfs2tab, you can set the PVFS2TAB_FILE environment variable to the desired path.


44

Kernel Module

The Kernel Module enables access to OrangeFS through the native Linux interface. You must designate

one or more systems as Linux clients, where you will copy the OrangeFS installation directory from the build system.

This section includes the following topics:

Copy OrangeFS Install Directory (Linux Clients) (page 44)

Set up Security (Linux Clients) (page 45)

Run (Linux Clients) (page 48)

Copy OrangeFS Installation Directory (Kernel Module)

You begin by copying the OrangeFS installation directory from the Build system to one or more systems designated as Linux clients.

If a Linux client is also a server, you might have already copied the installation directory.

System Requirements

Any system that functions as an OrangeFS Linux client requires a supported base installation of Linux.

Note For more details, see System Requirements (page 5).

Root Access Required

Adding a Linux client to OrangeFS requires root access on the client system.

Steps

To add an OrangeFS Linux client, follow these steps:

1. Copy the /opt/orangefs directory from the build system to the client system:

scp -r hostname:/opt/orangefs /opt

where...

hostname = host name of the build system

2. Create a directory in the system's /mnt directory through which the client will mount OrangeFS:

mkdir /mnt/orangefs

Note If a client is also a server, you might have already copied the installation directory.


45

Set up Security (Kernel Module)

After you copy the OrangeFS installation directory, you must perform additional setup and configuration for the key-based and certificate-based security modes. In future versions of OrangeFS, security will be simplified.

For key-based security, most of this work can be done once on the build system, then copied later to your servers and clients.

This topic includes sections for setting up two types of security:

Key-Based Security (page 45)

Certificate-Based Security (page 46)

Key-Based Security

Each client has its own key pair, consisting of a private key and a public key that are cryptographically

related. The private key is kept secret while the public key can be distributed. A file used by the servers known as the keystore contains public keys for all servers and clients in the OrangeFS system. When a client sends a request to the server, it submits a credential object which is signed by its private key. The server verifies the signature using the known public key of the client.

Note All OrangeFS clients and servers must be built for the same security mode (key-based in

this case) to interoperate.

Generating Client Private Keys

Like servers, all client systems must have a key pair. Because you need to build the keystore file for

the servers, you should create the client private keys on a single server—typically the one you used to

create the server private keys. You can then distribute them to the clients.

The openssl command used is the same as for the server, although for performance reasons the size

of the key (in bits) is less:

openssl genrsa -out pvfs2-clientkey.pem 1024

The size of the client private key, 1024 bits, is usually half the size of the server keys (default 2048).

This file is typically stored in the etc directory under the OrangeFS installation directory,

/opt/orangefs/etc by default.

Keys for multiple clients can be generated in a temporary directory and distributed to the client systems in a similar fashion as the server keys.

Configure Client for Key-Based Security

On OrangeFS client systems, the private key should be readable only by root:

chmod 600 /opt/orangefs/etc/pvfs2-clientkey.pem

The default private key location is pvfs2-clientkey.pem in the etc directory under the OrangeFS

installation directory, for example /opt/orangefs/etc/pvfs2-clientkey.pem. You can override

this location by using the --keypath parameter when running pvfs2-client. Example:

/opt/orangefs/sbin/pvfs2-client --keypath \

/usr/local/orangefs/etc/pvfs2-clientkey.pem


46

Copy Files to Client

The script pvfs2-dist-keys.sh distributes private keys and the keystore to multiple systems using

scp. The keys should have been generated with the pvfs2-gen-keys.sh script as described in the

Administration Guide.

Note The script requires one argument: the installation directory of OrangeFS which must be the same on all systems. This directory must exist prior to executing the script.

An example using the default location:

./pvfs2-dist-keys.sh /opt/orangefs

The script examines the key filenames to determine the hostname of the target server or client. For

example, the server file orangefs-serverkey-orangefs-server01.pem will cause the script to

execute this command, given /opt/orangefs as the installation directory:

scp orangefs-serverkey-orangefs-server01.pem orangefs-

server01:/opt/orangefs/etc/orangefs-serverkey.pem

Generate a client private key as instructed in the Administration Guide and append its public key to the keystore. Distribute the private key to the client system and the keystore to all servers.

To remove a client, edit the keystore file and remove the hostname identifier (for example

“C:client01”) and the public key that follows. Distribute this updated keystore file to all servers.

Currently in key-based security when a client is added to (or removed from) your OrangeFS installation, all servers must be stopped and restarted. The keystore is read only at server startup, so you would generally add clients during a maintenance period. Certificate-Based Security can be used if a more dynamic system is needed.

Certificate-Based Security

Note All OrangeFS clients and servers must be built for the same security mode (certificates, in

this case) to interoperate.

Prior to configuring your client(s) for certificate-based security, you must configure your servers and create a CA certificate. See Building OrangeFS for Certificate-Based Security for steps to take before configuring clients.

Then, install OpenSSL client libraries to the client system if necessary. (Consult your OS distribution documentation for more information.)

User Certificate Application

A client application, pvfs2-get-user-cert, is installed to allow users to request and receive a user

certificate with no intervention from the administrator.

You must configure the client system to connect to a running OrangeFS server; the file pvfs2tab,

located in /opt/orangefs/etc by default, contains the necessary configuration information. (See

pvfs2tab File (page 43) for more information on pvfs2tab.)

The requesting user must have an identity (user account) in the LDAP directory before requesting a

certificate. See Configuring LDAP for Identity Mapping for more information.

The usage of pvfs2-get-user-cert is:

pvfs2-get-user-cert [user name]

If the optional user name is not supplied, the user name of the currently-active user account will be used. The user will be prompted for their LDAP directory password. Once this is entered correctly, the

http://www.omnibond.com/orangefs/docs/v_2_9/admin_Certificate-Based_Security.htm#Configuring_LDAP_for_Identity_Mapping


47

user certificate and private key are stored as ~/.pvfs2-cert.pem and ~/.pvfs2-cert-key.pem,

respectively.

Obtaining a User Certificate Manually

If you do not want users to use the pvfs2-get-user-cert application, they can create a certificate

request, which an administrator can use to generate a certificate.

Creating a User Certificate Request

A certificate request is a file indicating what values should be in the requested certificate. A user can generate the request and submit the file to the administrator for signing by the CA certificate. In a production environment, it is not secure for users to sign their own certificates.

To generate a certificate request, execute this command:

openssl req -newkey rsa:1024 -config pvfs2-user.cnf -keyout pvfs2-cert-key.pem -

nodes -out pvfs2-cert-req.pem

Note pvfs2-user.cnf is in the examples/certs directory.

You can use different file names. The user will be prompted to enter subject values, which should follow some organization-defined naming scheme.

Note The common name of the certificate subject will be used for UID/GID-mapping later, so

take note of it.

The user can then submit (for example via email) the certificate request (but not the private key) to the administrator for signing.

A script named pvfs2-cert-req.sh is in examples/certs for this step. It takes a name as an

optional parameter (default “pvfs2”):

./pvfs2-cert-req.sh pvfs2

Signing a User Certificate Request

The administrator will sign the certificate request with the CA private key. Execute this command:

openssl x509 -req -in pvfs2-cert-req.pem -CA orangefs-ca-cert.pem -CAkey orangefs-

ca-cert-key.pem -days 365 -out pvfs2-cert.pem

The file names should correspond with file names used in prior steps. Return the resulting certificate

file (pvfs2-cert.pem above) to the user.

A script named pvfs2-sign-cert.sh is in examples/certs. It takes the cert name and the CA

name as optional parameters (defaults “pvfs2” and “orangefs” respectively):

./pvfs2-cert-sign.sh pvfs2 orangefs

The files pvfs2-cert.pem and pvfs2-cert-key.pem can then be sent to the user (for example via

email).

Storing the User Certificate

The user can now store the certificate and private key files. The default file names used by OrangeFS

are ~/.pvfs2-cert.pem for the certificate file and ~/.pvfs2-cert-key.pem for the key file. Note

the “.” preceding both names, which marks them hidden. The private key and certificate should have permissions revoked for other users:


48

mv pvfs2-cert.pem~/.pvfs2-cert.pem

mv pvfs2-cert-key.pem~/.pvfs2-cert-key.pem

chmod 600~/.pvfs2-cert*.pem

These locations can be overridden with the PVFS2CERT_FILE and PVFS2KEY_FILE environment

variables. These variables are used when accessing OrangeFS through a client application (sysint--

for example pvfs2-ls) or library (usrint); they are not used if OrangeFS is mounted through

the kernel module.

Run (Kernel Module)

Running each Linux client involves three administrative tasks:

Inserting the OrangeFS kernel module into the local kernel

Starting the client process

Mounting OrangeFS (via one of its servers)

Run

Running the Linux client involves three tasks:

Inserting the OrangeFS kernel module into the local kernel

Starting the client process

Mounting OrangeFS (via one of its servers)

Generally, all three tasks need to be completed once each time the system is rebooted.

Insert the Kernel Module

The kernel module (pvfs2.ko) for the Linux client resides in the OrangeFS installation directory

several directory layers deep. To insert the module without specifying a long path, include this find

statement:

insmod ‘find /opt/orangefs -name pvfs2.ko‘

Start the Client Process

The OrangeFS Linux client is a daemon (pvfs2-client) that interfaces with the kernel and runs

continuously in the background. To start the client, enter the following command:

/opt/orangefs/sbin/pvfs2-client -p /opt/orangefs/sbin/pvfs2-client-core

Mount OrangeFS

You will mount OrangeFS through the server URL you retrieved earlier. To mount, enter the following command:

mount -t pvfs2 tcp://server1:3334/orangefs /mnt/orangefs

Note In the above example, tcp: is the network protocol, //server1 is the server providing

access to the configuration file, and 3334 is the number of the TCP/IP port on which the

OrangeFS servers communicate; /mnt/orangefs is the path you use to access these files.

You can think of /mnt/orangefs as the root directory of the OrangeFS file system.


49

Direct Interface

The Direct Interface allows you to access OrangeFS in an environment similar to Linux (POSIX-based);

however, the Direct Interface (also known by usrint, the system folder in which it is stored)

bypasses the Linux Kernel for a more direct and better performing path to OrangeFS. It provides high performance access for programs that are not written for MPI.

Important The OrangeFS Direct Interface using Global Configuration will work only on systems configured with shared C libraries.

The Direct Interface is included with the OrangeFS standard installation, accessed by copying appropriate files to a client location and activating it with configuration statements.

This topic is organized into two sections:

Understanding_the_Interface_Levels (page 49)

Configuring_the_Direct_Interface (page 50)

Note To learn about System Calls for the Direct Interface, see System Calls in the Administration Guide.

Understanding the Interface Levels

The Direct Interface offers three levels of access, so you must configure your access based on the level that works best for your needs. The following illustration shows the three interface levels.

Descriptions of the three levels:

Level Library Description

1 System

Call Library

The first and lowest level is an API with OrangeFS-specific functions that can be

substituted for each of the basic POSIX defined I/O related system calls. Essentially, each POSIX system call is replicated in the API. What makes this API different is that each function ONLY works with files in the OrangeFS file systems.


50

Level Library Description

2 POSIX Library

The next layer is a POSIX system call interposition library. Each of the same POSIX system calls represented in the lower layer are provided in this API, this time with the same interface syntax as Linux POSIX. Rather than calling the Linux kernel directly, each call is checked to see if it refers to an OrangeFS file,

and if so the call is made to the corresponding function in the lower level API.

Thus a call to open() will call pvfs_open() if the path refers to an OrangeFS

file; otherwise it will call the Linux open system call. This API is more convenient, though slightly less efficient, than the lower level one.

3 C Library Finally, many programmers prefer to use the C library interface rather than the system call interface to file I/O, in part because it provides I/O buffering and a richer set of interface options. Any C calls are implemented using the POSIX calls, and so their implementation can, in theory, be linked from the C library, and use

the OrangeFS POSIX interposition API.

Virtually all modern Linux systems use shared libraries for the C library. Shared

libraries tend to link all of the various functions at various levels into a single

shared object that is loaded dynamically. Thus, if you call fopen() using the

standard shared C library, there is no means to get that function to call the

OrangeFS pvfs_open() function. For this reason, OrangeFS provides its own

implementation of these functions in an OrangeFS C Library interposition API. These functions are identical to those in the standard C library implementation, except that they call the OrangeFS functions, and, in some cases, can be optimized for specific OrangeFS features.

Configuring the Direct Interface

This section explains two methods for configuring the Direct Interface.

Program_Configuration (page

50) Use this method to specify an individual program to run through the OrangeFS Direct Interface.

Global_Configuration (page

51) Use this method to specify that all programs will run through the OrangeFS Direct Interface.

Program Configuration

Programs, and higher level libraries, written to any of the three library levels included in the Direct

Interface should link to the appropriate OrangeFS replacement library (liborangefsposix or

liborangefs) to directly access the OrangeFS file system. The command for this configuration also

determines whether to use a shared or static version of the library.

To link a program with the replacement library, include the following command when compiling the program:

gcc -o program program_source -Lorangefs_lib_path -rep_lib

where...

program = the name of your program, including the path

program_source = the name of your program source code, including the path

orangefs_lib_path = path to lib directory in the OrangeFS installation directory


51

rep_lib = one of the following options:

If your program is written to: Enter this option: To use this replacement library:

C Library or POSIX Library -lorangefsposix liborangefsposix

OrangeFS System Call Library -lorangefs liborangefs

Example command line:

gcc -o /programs/foo /programs/foo.c -L/opt/orangefs/lib -lorangefsposix

Global Configuration

Programs not specifically recompiled to use OrangeFS can still be redirected to do so by preloading the

shared version of the appropriate OrangeFS replacement library (libofs and/or libpvfs2). You

must configure the source to build the shared library before compiling OrangeFS.

Assuming the shared libraries are installed, set the following environment variables:

export OFS_LIB_PATH=orangefs_lib_path

export LD_LIBRARY_PATH=$OFS_LIB_PATH:$LD_LIBRARY_PATH

export LD_PRELOAD=$OFS_LIB_PATH/rep_shared_library

where...

orangefs_lib_path = path to lib directory (in the OrangeFS installation directory)

Example: /opt/orangefs/lib

rep_shared_lib = one of the following replacement library files:

To redirect programs written to: Use these replacement library files:

C Library or POSIX Library libofs.so and libpvfs2.so

OrangeFS System Call Library libpvfs2.so

Example: LD_PRELOAD=$OFS_LIB_PATH/libpvfs2.so

Notes The global configuration method does not work if you use the static version of libc.

Ensure that your system's /etc/ld.so.preload includes libdl, libssl, libcrypto

and libpthreads preloaded through /etc/ld.so.preload. Most Linux systems will

already include this.

If this configuration method is used in the shell, every program (including such commands

as ls, vi and cp) will redirect through the OrangeFS libraries. You can set these variables

in a script to affect only the desired commands. If all users on a system want the shared libraries preloaded, the system administrator can

edit the file /etc/ld.so.preload and list the libraries there.


52

FUSE

Filesystem in Userspace (FUSE) is a loadable kernel module for UNIX-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code. File system code is run in user space while the FUSE module provides only a bridge to the actual kernel interfaces.

The OrangeFS client interface for FUSE enables access to an OrangeFS file system from a Mac.

Note While FUSE can run on both Linux and Mac, Linux users of OrangeFS will achieve better results with the Linux Kernel Module and Direct Interface.

Setting up a FUSE client involves four main steps on your Mac system:

Install FUSE (page 52)

Install OrangeFS (page 52)

Mount an OrangeFS Server (page 53)

Set up Security (page 55)

Install FUSE

The recommended FUSE distribution for the Mac is Fuse4x.

Fuse4x can be downloaded using Apple's port mechanism or from the Web at

http://fuse4x.github.io/.

To get fuse4x using the port command:

port install fuse4x

Note At the time of this Installation Guide’s initial release, Fuse4x had been recently tested on OrangeFS using the Darwin Kernel Version 11.4.2.

Fuse4x documentation on its website will guide you through the installation process.

Install OrangeFS

OrangeFS must be downloaded and built in the OS X environment on your Mac system.

Important Copying the OrangeFS installation directory from your Linux Build system to a Mac system will not work. You must build it separately.

Prerequisites

Prior to installing OrangeFS, you must install gcc, flex, bison, make, and openssl-devel as

described in Additional Linux Software for the Build System (page 7).

Procedure

To build OrangeFS on your Mac system, follow these steps:

1. Go to www.orangefs.org. If your OrangeFS filesystem is using the latest released version, select

orangefs-<version>

where...

<version> = version number of the OrangeFS distribution release

Example: orangefs-2.9.

If your OrangeFS filesystem is using an older version, select the New releases in the Previous

http://fuse4x.github.io/

http://www.orangefs.org/download/

http://www.orangefs.org/downloads/old-releases/


53

Releases section on the OrangeFS downloads page. select your release then select the source link

to find the tar ball.

Notes The OrangeFS client and servers MUST be using the same version.

For systems using releases older than 2.8.2-1, the tar balls can be found by selecting

the Previous releases link; however, these releases have not been recently tested.

Use at your own discretion. If using Safari to download, the tar ball will be automatically unzipped, producing an

orangefs-<version>.tar file. In Firefox, you will download a zipped tar ball,

orangefs-<version>.tar.gz.

2. Change directory (cd) to

/Users/username/Downloads,

where...

username is the Mac username of the person creating this interface.

Locate one of the following tar balls:

orangefs-<version>.tar or orangefs-<version>.tar.gz

and extract the OrangeFS source files using one of the following commands:

Unzipped:

tar -xf orangefs-<version>.tar

Zipped:

tar -xzf orangefs-<version>.tar.gz

Then, change your working directory (cd) to orangefs-<version>.

-xzf orangefs-<version>.tar.gz

cd orangefs-<version>

3. Build a Makefile for OrangeFS that includes the installation location and four other options as follows:

./configure --prefix=/opt/orangefs --disable-server --disable-usrint --

disable-opt --enable-fuse

4. Continue with the standard Linux commands to build and run an executable program:

make

make install

This will create the OrangeFS installation directory in /opt/orangefs. Within that directory, the

binary you need to run FUSE, pvfs2fuse, will be located in the bin directory.

Mount an OrangeFS Filesystem

Assuming you have network access to the OrangeFS filesystem, you must first create a mount point on your Mac.

ftp://ftp.parl.clemson.edu/pub/pvfs2/old


54

Note Confirm access to any of the servers using the ping command.

To mount an OrangeFS Server:

1. Create a directory as the mount point. This directory can be anywhere on your Mac where you have create permissions:

mkdir /mnt/orangefs

2. The FUSE client requires an OrangeFS filesystem specification defined as

URL/<filesystem name>

where...

URL = any ONE of the OrangeFS servers that manages your filesystem, found with the

filesystem name in the OrangeFS server conf file

Example: orangefs-server.conf.

The URL value for each server in the filesystem is listed in the <Aliases> section, while the

filesystem name is listed in the <Filesystem> section.

<Aliases>

clemson1 tcp://server1:3334

tiger1 tcp://server2:3334

</Aliases>

<Filesystem>

Name <filesystem name>

...

</Filesystem>

The filesystem spec in this case is one of two choices:

tcp://server1:3334/<filesystem name>

tcp://server2:3334/<filesystem name>

3. Now you are ready to mount an OrangeFS filesystem, as follows:

/opt/orangefs/bin/pvfs2fuse /mnt/orangefs -o

fs_spec=tcp://server1:3334/<filesystem name>

Notes In the above example, tcp://server1:3334 is the URL of only one of the OrangeFS

servers managing the given filesystem, determined in Step 2; /mnt/orangefs is the

mount point created in Step 1.

If 'root' issues the pvfs2fuse command, then all users of your Mac can access the

filesystem. However, if <username> issues the command, only <username> has

access.

Once the mount is successful, you can access your OrangeFS installation using common

commands like ls and cp. For example:

ls /mnt/orangefs


55

Set up Security

Using Default Security

By default, OrangeFS uses User and Group IDs to enforce file permissions. Files created using your Mac will be stored with your Mac UID and primary GID, so only you have access to your files. However, if you created files using one of the other OrangeFS clients, you might not have access to your files from the Mac, unless your Mac UID (or GID) happens to match the UID (or GID) in these other environments.

To alleviate this problem, you or your Mac administrator can create User IDs having the same UIDs

and GIDs that match across platforms. We suggest that you do NOT change an existing user's UID or primary GID. Instead, we recommend you create a new ID having the appropriate UID/GID values.

Below is an example (Darwin Kernel Version 11.4.2) that creates a Mac user called "orangefs", with a specific UID and GID from the command line:

$ sudo dscl . create /Users/orangefs uid 500

$ sudo dscl . create /Users/orangefs gid 5005

$ sudo dscl . create /Users/orangefs shell /bin/bash

$ sudo dscl . create /Users/orangefs home /Users/orangefs

$ sudo dscl . create /Users/orangefs realname "orangefs"

$ sudo dscl . create /Groups/orangefs gid 5005

$ sudo dscl . create /Groups/orangefs passwd \*

To create an "orangefs" group and set its GID to 5005:

$ sudo dscl . create /Groups/orangefs gid 5005

$ sudo dscl . create /Groups/orangefs passwd '*'

To add a user to this group:

$ sudo dscl . merge /Groups/orangefs users <username>

Using Key Security

If your OrangeFS file system is using key security, then the OrangeFS FUSE client must be built with

key security enabled. Add --enable-security-key to the "configure" command:

./configure --prefix=/opt/orangefs --disable-server --disable-usrint --disable-opt

--enable-fuse --enable-security-key

Your OrangeFS system administrator will typically create a public/private key pair for your Mac and will give you the private key to store on your machine. By storing the private key as

<prefix>/etc/pvfs2-clientkey.pem

or, as in the above configure command,

/opt/orangefs/etc/pvfs2-clientkey.pem

pvfs2fuse will automatically find and use this key. If you store the key with a different name or in a

different location, you must first define the PVFS2KEY_FILE environment variable before issuing the

pvfs2fuse command:

$ export PVFS2KEY_FILE=<path-to-key>/<key filename>


56

Your OrangeFS system administrator must create the public/private key pair using the hostname of

your Mac. To determine the hostname, first ensure that you can ping at least one of the OrangeFS

server machines. Then, issue the hostname command to get the value needed by the system

administrator to create the correct public/private key pair.

Notes The system administrator must add the public key to the OrangeFS keystore, copy the keystore to each server machine, then restart the servers before you will have access to

the file system. See Setting up Key-Based Security Mode for more information. If your Mac has a non-static IP address in your environment, you will have to regenerate a new public/private key pair each time the address changes. The default security using UID and GID for file permissions is used in addition to the

public/private key pair. See Using Default Security (page 55) above.

ROMIO MPI Interface

ROMIO is a particular implementation of the MPI-IO protocol, the open standard for data transfer to and from MPI.

MPI, also an open standard, was created for researchers who needed a message-passing interface

optimized for high performance parallel computing.

Different working implementations for MPI, also called MPI libraries, exist. Two popular MPI libraries are MPICH from Argonne National Laboratory and Open MPI from a consortium of users including Oak Ridge National Laboratory.

Different implementations of MPI-IO also exist. The ROMIO implementation includes support for OrangeFS. Therefore, any MPI library implementation that works with ROMIO, such as MPICH and Open MPI, can also work with OrangeFS.

Setting up a ROMIO client involves one step, with options for configuring Open MPI or MPICH:

Configuring for Linux (page 56)

Install OrangeFS 2.9 with MPICH 3.0.4 (page 57)

Install OrangeFS 2.9 with Open MPI 1.6.5 (page 58)

Configuring for Linux

Both MPICH and Open MPI are packaged with ROMIO. Configuring either of these MPI implementations to access OrangeFS involves two areas:

Adding OrangeFS installation files

Linking programs to OrangeFS

Adding OrangeFS Installation Files

To add the OrangeFS installation files to the MPI client system, Change Directory (cd) to /opt on the

client and copy the /opt/orangefs directory from the build system:

scp -r hostname:/opt/orangefs /opt

where...


Linking Programs to OrangeFS

When you run your Open MPI or MPICH applications, link to OrangeFS by including the -lpvfs2

option.

For example, to run a program called mytest, you would follow these steps:


57

1. Compile and link the program to the pvfs2 library in your OrangeFS installation:

cc mytest.c -o mytest -L /opt/orangefs/lib -L /mpich2_install_dir/lib -I

/opt/orangefs/include -I /mpich2_install_dir/include -lpvfs2 -lmpich

where...

mpich2_install_dir = the name and path of your MPICH installation directory

Other Operating Environments

ROMIO can also run on Windows and Mac. Those platforms are less efficient for the high performance parallel computing that most ROMIO users seek in OrangeFS, so the above instructions focus on Linux client implementations only.

To connect to OrangeFS from a Windows environment, consider using the Windows Client (page

59) developed specifically for OrangeFS.

To connect to OrangeFS from a Mac environment, consider using FUSE (page 52).

Install OrangeFS 2.9 with MPICH 3.0.4

Notes You must install OrangeFS on your storage nodes and the OrangeFS system must be online

prior to performing these steps. If you have not completed this step, see the Installation

Guide (page 1) for instructions to complete this step before proceeding.

For instructions on how to use MPICH, see MPICH User's Guide.

Secure Shell (SSH)—without Passphrase

You must configure all clients to support secure shell connections via SSH without passing a

passphrase. For more information, see Generating SSH Keys for Passwordless Login, an article

from the Hortonworks Knowledgebase.

Prior to configuring MPICH, ensure that you have built shared libraries for OrangeFS:

1. Run the same ./configure command you used when you installed OrangeFS, but add the following additional option:

--enable-shared

To configure OrangeFS to work with MPICH 3.0.4, complete the following steps:

1. Run the following commands to remove references to methods included in MPICH that will cause

errors during the ./configure stage.

Note You will not be able to use the MPI/IO functions IReadContig and IWriteContig.

sed -i s/ADIOI_PVFS2_IReadContig/NULL/

src/mpi/romio/adio/ad_pvfs2/ad_pvfs2.c

sed -i s/ADIOI_PVFS2_IWriteContig/NULL/

src/mpi/romio/adio/ad_pvfs2/ad_pvfs2.c

2. Compile MPICH with --enable-shared option:

./configure --prefix=/opt/mpich-3.0.4 --enable-romio --enable-shared --with-

pvfs2=/opt/orangefs --with-file-system=pvfs2

where...

/opt/orangefs = the location of your OrangeFS installation

http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4-userguide.pdf

http://hortonworks.com/kb/generating-ssh-keys-for-passwordless-login/


58

/opt/mpich-3.0.4 = the location of your MPICH installation

Note You can remove the --prefix command to install to /usr/local

3. Make and install freshly compiled MPICH 3.0.4 with OrangeFS Support

·sudo make all install

4. Set LD_LIBRARY_PATH to point to the MPICH libs

export LD_LIBRARY_PATH=/opt/mpich-3.0.4/lib:$LD_LIBRARY_PATH

where...

opt/mpich-3.0.4 = the location of your MPICH installation

Install OrangeFS 2.9 with Open MPI 1.6.5

Notes You must install OrangeFS on your storage nodes and the OrangeFS system must be online

prior to performing these steps. If you have not completed this step, see the Installation

Guide (page 1) for instructions to complete this step before proceeding.

For instructions on how to use Open MPI, see Open MPI: Open Source High

Performance Computing.


You must configure all clients to support secure shell connections via SSH without passing a



Prior to configuring MPICH, ensure that you have built shared libraries for OrangeFS:

1. Run the same ./configure command you used when you installed OrangeFS, but add the following

additional option:

--enable-shared

To configure OrangeFS to work with Open MPI 1.6.5, complete the following steps.

1. Patch the Open MPI 1.6.5 source to support OrangeFS:

Patch the Open MPI installation using the openmpi-1.6.5-romio.patch file. This patch is

available on the OrangeFS

patch -p0 < openmpi-1.6.5-romio.patch

2. Run the following commands to remove references to methods included in Open MPI that will

cause errors during the ./configure stage.

Note You will not be able to use the MPI-IO functions IReadContig and IWriteContig.

sed -e 's/ADIOI_PVFS2_IReadContig/NULL/' \

-i ompi/mca/io/romio/romio/adio/ad_pvfs2/ad_pvfs2.c

sed -e 's/ADIOI_PVFS2_IWriteContig/NULL/' \

-i ompi/mca/io/romio/romio/adio/ad_pvfs2/ad_pvfs2.c

3. Compile Open MPI with the --enable-shared and --with-pic options:

http://www.open-mpi.org/

http://www.open-mpi.org/



59

./configure --prefix=/opt/openmpi-1.6.5 --enable-shared --with-pic --with-io-

romio-flags="--with-pvfs2=/opt/orangefs --with-file-system=pvfs2"

where...


/opt/openmpi-1.6.5 = the location of your Open MPI installation

Note You can remove the --prefix command to install to /usr/local

4. Make and install freshly compiled Open MPI 1.6.5 with OrangeFS Support

·sudo make all install

5. Set LD_LIBRARY_PATH to point to the Open MPI libs

export LD_LIBRARY_PATH=/opt/openmpi-1.6.5/lib:$LD_LIBRARY_PATH

where...


/opt/openmpi-1.6.5 = the location of your Open MPI installation

Windows Client Interface

The Windows Client provides native access to OrangeFS/PVFS2 file systems for desktops and servers

using the Microsoft Windows operating system.

The Windows Client harnesses the power and speed of OrangeFS from the Windows platform,

including scale-out storage access and high performance parallel computing with full programmatic access via standard parallel programming APIs.

Options for authentication and user mapping include LDAP and X.509 certificates. The Client, which

runs as a standard Windows service, supports Windows Vista, Windows 7, Windows 8, and Windows Server 2008 R2 (all editions) and Windows Server 2012 (all editions; Server Core installation not currently supported); x86 and x64.

This Client enables you to view files though Windows Explorer and the Command Prompt, or you can

access files programmatically through standard function calls, for example, fopen() in C.

The following topics guide you through the installation, operation and configuration of the Windows Client:

Preparing for Installation (page 60)

What to Expect (page 62)

Running the Enterprise Installer (page 62)

Manual Installation (page 64)

Uninstalling the Windows Client (page 65)

Using the Client (page 66)

Client Administration (page 67)


60

Preparing for Installation (Windows Client)

Important Please read this section before beginning installation of the Windows Client.

Client Requirements

Operating systems:

Windows Vista

Windows 7

Windows 8

Windows Server 2008 or Windows Server 2008 R2 (all editions; Server Core installation not currently supported)

Windows Server 2012 (all editions; Server Core installation not currently supported)

Hardware:

30MB disk space

Other requirements dependent on usage; minimum requirements very low

Other:

You must assign a drive letter during installation.

It is best to run the installer as an administrative user (Administrator, for example).

OrangeFS Requirements

To connect to the OrangeFS server during installation, you must specify its URI. You must know the host name and port number (default is 3334). The format for this entry is provided later in the instructions.

Important The OrangeFS installation accessed by the Windows Client must be configured for TCP

network protocol.

File System Security Mode

An OrangeFS file system operates in one of three security modes: default, key-based or certificate-based. All servers and clients must operate in the chosen mode. The “security-mode” option of the Windows Client configuration file (see “Windows Client Interface > Client Administration”) should be

set to select the correct mode. See Preview Security (page 10) for more information.

Authentication Configuration

During installation you have four mode options for user mapping:

list

certificate

ldap

The following table summarizes these options:

User Mode Option

Description Best For... Installation Input More to do after installation?

list Directly matches one Windows ID with one OrangeFS UID and primary GID.

Simple, smaller installations, trial runs, etc.

Enter one Windows user ID and the OrangeFS (Linux/UNIX-based) UID and primary GID for mapping.

All but first user must be entered manually in the

orangefs.cfg file

after installation.


61

User Mode Option

Description Best For... Installation Input More to do after installation?

certificate Maps user digital certificate to OrangeFS UID/GID. Our recommended

setup is for grid computing, which requires CA, proxy and user certificates.

Important You

must install and configure your certificates

before installation. A recommended method for doing

this is included in these instructions.

Scientific, large cluster, research, etc.

Specify the Windows prefix directory of your user and proxy certificates. This

might be either the user's profile directory or a custom directory, which you must enter:

c:\users

or cert-dir-prefix

If your certificates were properly installed and configured before

installation, nothing else should be required.

ldap Maps user(s) on an

LDAP tree to OrangeFS UIDs/GIDs.

Windows

with Active Directory or eDirectory.

LDAP inputs, acount

to sign in, etc.

If all inputs are

entered, nothing else should be required.

server This mode is only used with certificate

security mode (see above). Identity information is stored for each user in a client-side certificate. Then the server,

rather than the client, maps this information to an OrangeFS identity using LDAP.

Installations that require

per-user security, particularly those that use LDAP for user

information.

Non (configured post-installation)

Run orangefs-

get-user-cert for

each user (see Using

the orangefs-get-

user-cert App (page

72))

For more information on user mapping, see Client Administration (page 67).


62

What to Expect

When you complete the Enterprise installation process, if you have provided all the necessary inputs, the last panel in the installer offers the option to start the client. If you select this option, your system mounts the OrangeFS server you specified, and the Windows Client starts. If you are not ready to start the client, you can do it manually later.

The installation program creates two new directories on your Windows system:

New Directory Description

C:\OrangeFS\Client OrangeFS client software, including the client executable

(orangefs-client.exe), the orangefs-get-user-cert.exe

app, and two configuration files (orangefs.cfg, orangefstab).

C:\Program Files\Dokan

(enterprise installation)

or

C:\Dokan (manual installation)

The Dokan Library, an open source set of files used by the OrangeFS client to mount an OrangeFS file system as a virtual

drive.

The installation program adds a few settings to your Windows Registry. These settings are automatically removed if you uninstall the client.

Running the Enterprise Installer

To install the OrangeFS Windows Client, you need the self-extracting installation program. Two versions are available, depending on your system’s processor type (32-bit or 64-bit).

Download and run orangefs-client-version-win32.exe or orangefs-client-version-

win64.exe

where...

version = version number of the executable

Example: orangefs-client-2.8.5.3-win64.exe

Notes At this time, you cannot run the 32-bit installer on a 64-bit OS.

It is best to run the installer as an administrative user (Administrator, for example).

After running the executable, follow these steps:

1. When the installation program's Welcome dialog displays, click Next.

The next dialog prompts you for an installation location.

2. Use the default or select a different location and click Next.

3. Click Install to install the client.

The next dialog prompts you for a file system URI, a mount point and user mapping mode.

4. Complete the dialog as follows:


63

File System

URI

Enter the DNS name/IP address and port number of an OrangeFS file system

server in a URI format: tcp://hostname:port/FS_name

Example: tcp://server1.com:3334/orangefs

The default port number is 3334.

Mount Point Click the drop-down button to select a drive letter (E: to Z:) for the file system.

Select Auto to use the first available drive letter (starting with E:).

User

Mapping

select List, Certificate, or LDAP. This corresponds with the mode of user

mapping (described in the next step). If you are not sure, select List, as the

settings can be changed later.

Important The certificates must first be generated and placed in appropriate locations to support the Windows client before you can select the

certificate mode. If you still need to complete this process, close the installation program and restart it when you have completed the

certificate generation.

Click Next to continue.

5. Depending on the mode of user mapping you chose in step 5, a dialog with one of the following titles prompts you for more information:

List Map

Add User

If you selected list mode, enter one Windows user ID and the OrangeFS

(Linux/UNIX-based) UID and primary GID for mapping. You will manually add additional users to the configuration file after the installation.

Certificate

User

Mapping

If you selected certificate mode, enter the Windows prefix directory of

your user and proxy certificates. The default is the user profile directory

(C:\Users). You can also enter another location, for the prefix directory.

Setup Type If you selected ldap mode, a dialog provides three choices for your LDAP

implementation (Microsoft Active Directory, Novell eDirectory or

Custom). Select one and click Next.

In the next dialog, enter the LDAP values required by OrangeFS.

Note Depending on your LDAP selection, some of the text fields that display in the dialog might already have entries.

For complete details on user mapping see Client Administration (page 67).

6. Click Next to continue.

The final dialog displays with an option to start the OrangeFS service when you exit the

installation program.

7. Do one of the following:

● Select the check box for Start the OrangeFS services if you want the Windows Client to

mount the OrangeFS file system.

● Leave the check box deselected if your configuration is not complete. You can manually start

the service later.

8. Click Finish to complete the installation.


64

Manual Installation

Important The Client connects to a running OrangeFS server. If you have not yet installed the

OrangeFS server components, consult the documentation and install the server (page 36)

before installing the Windows Client.

Follow the instructions below to install the OrangeFS Windows Client:

1. Download the ZIP file associated with your system type (64- or 32-bit):

a. For 64-bit systems, download orangefs-windows-client-version#-win64.zip.

b. For 32-bit systems, download orangefs-windows-client-version#-win32.zip.

where...

version# is the OrangeFS version, for example, 2.8.*.

2. Extract the ZIP file to any directory, where the OrangeFS and Dokan directories will be created.

Open a Command Prompt from Start | All Programs | Accessories to complete the following steps.

Install the Dokan driver:

1. Change directory (cd) to the Dokan\DokanLibrary directory.

2. Copy dokan.dll to the System32 directory:

copy dokan.dll c:\windows\system32

3. Copy dokan.sys to the system Drivers directory:

copy dokan.sys c:\windows\system32\drivers

4. Install the driver using dokanctl.exe:

dokanctl /i d

5. Restart your system.

Install the Dokan Mounter service:

1. Change directory (cd) to the Dokan\DokanLibrary directory.

2. Install the service using dokanctl.exe:

dokanctl /i s

Install the OrangeFS Client service:

1. Change directory (cd) to the OrangeFS\Client directory.

2. Install the service using orangefs-client.exe:

orangefs-client -installService

3. Configure the OrangeFS Client by creating the orangefstab and orangefs.cfg files in

OrangeFS\Client, following the instructions in the Configuration (page 67) section of the

Windows Client documentation.

4. Start the Dokan Mounter and OrangeFS Client services using the Services Administrative Tool

(Start | Control Panel | Administrative Tools | Services).

Your OrangeFS file system should appear as a Removable Drive.


65

For troubleshooting, open the Event Log Administrative Tool and consult the Application Log. For

additional help, consult the documentation.

Uninstalling the Windows client

Uninstalling the Enterprise Installation

To uninstall the Windows client, follow these steps:

1. From the Windows Start Menu, select Control Panel, then Programs and Features.

2. Locate and select the OrangeFS Client item, and click the Uninstall button above.

3. Follow the uninstaller steps to remove the Client.

4. Remove configuration files under C:\OrangeFS\Client (by default) and the

C:\OrangeFS\Client directories.

Uninstalling a Manual Installation

Follow the instructions below to uninstall the OrangeFS Windows Client:

Stop the Dokan Mounter and OrangeFS Client services.

Remove the OrangeFS Client service:

Change directory (cd) to the OrangeFS\Client directory.

Remove the service using orangefs-client.exe:

orangefs-client -removeService

Remove the Dokan Mounter service:

Change directory (cd) to the Dokan\DokanLibrary directory.

Remove the service using dokanctl.exe:

dokanctl /r s

Remove the Dokan driver:

Change directory (cd) to the Dokan\DokanLibrary directory.

Remove the driver using dokanctl.exe:

dokanctl /r d

Restart your system

Remove Dokan system files:

Remove dokan.dll:

del c:\windows\system32\dokan.dll

Remove dokan.sys:

del c:\windows\system32\drivers\dokan.sys

Remove application files:

Remove the Dokan directory:

rd Dokan /s

Remove the OrangeFS directory:

rd OrangeFS /s


66

Using the Client

Information in this topic includes:

Interfacing with OrangeFS (page 66)

Running the Client (page 66)

Understanding Security (page 66)

Getting/Generating New User-Mapping Certificates (page 67)

Interfacing with OrangeFS

When the Windows Client is running on your computer, the OrangeFS file system appears as a

removable drive at the drive letter (E:-Z:). This drive letter, specified during installation, is a setting

in the configuration file that can be changed. For more information, see Client Administration (page

67).

You can interact with files and directories in the file system like local files. For example, they can be viewed in Windows Explorer, listed in the Command Prompt or accessed using program API functions,

such as fopen.

Note Currently the Client can mount only one OrangeFS file system at a time.

Running the Client

You must start two Windows Services to run the OrangeFS Windows Client:

DokanMounter

OrangeFS Client

You can access these services in the Windows Services utility. To open the Services utility, navigate to the Control Panel and click Administrative Tools | Services. You should see the DokanMounter and OrangeFS Client services included in the console listing.

To start (or stop) a service, right-click the service and select the desired action.

You should start the DokanMounter service first. This service is tied to the Dokan Library, which is the

third-party software included with your installation. DokanMounter enables the Windows Client to mount the file system transparently.

The two services are configured to start automatically any time the system is restarted. To change this setting, right-click the service and select Properties.

Note If you need to stop the Windows Client service, you do not normally have to stop the

DokanMounter service.

Understanding Security

First you must set the Client to operate using the File System security mode used by the servers. Do this by setting the “security-mode” option in the configuration file to “default”, “key” or “certificate”,

with “default” being used if no option is specified. (For more information, see Client Administration

(page 67).)

You can configure the file as read-only on Windows to remove owner write permissions.

Note The default permissions mask can be changed with the new-file-perms and new-dir-

perms configuration file keywords. Form more information, see Client Administration

(page 67).

Level of security will also depend on the user mapping configuration of your Windows Client. The three types of user mapping are


67

List Directly matches one Windows ID with one OrangeFS

UID and primary GID.

Certificate Maps user digital certificate to OrangeFS UID/GID. Our recommended configuration is for grid computing which requires CA, proxy and user certificates.

LDAP Maps user(s) on an LDAP tree, such as Active Directory or eDirectory, to OrangeFS UIDs/GIDs.

Server Used only when the security mode is “certificate,” this mode features client-side certificates for each user and server-side identity mapping with LDAP.

For more information, see Client Administration (page 67).

Getting (or Generating) New User-Mapping Certificates

Note This task only applies if your Windows Client is using certificates mode for user mapping.

If your Windows Client is configured for certificate mapping, this will likely involve three types of certificates (CA, proxy, user). Usually, your administrator creates and installs these certificates. However, since all certificates have expiration dates, you might need new ones regenerated from time to time while using the Windows Client.

Depending on your setup, you might need to request new certificates from your administrator, or the administrator might provide you with instructions for doing it yourself.

Of the three types of certificates mentioned earlier, the proxy certificate must generally be renewed

more often than the other two. Depending on your administrative policies, the time before a proxy certificate expires can average anywhere from 6 hours to two weeks.

Client Administration


Configuration (page 67)

Configuring the Client Security Mode (page 71)

User Mapping (page 73)

Installing and Using Globus Toolkit (page 77) (certificates only)

Troubleshooting (page 79)

Source Code (page 80)

Configuration

Two configuration files exist for the OrangeFS Client:

orangefstab

orangefs.cfg

Both text files are located in the installation directory (C:\OrangeFS\Client) by default and can be

edited.

The orangefstab file contains only one line entry, which is the URI address of the OrangeFS server

to be mounted for Windows Client access.

The orangefs.cfg file can contain a wide range of settings, including:

The drive letter on your Windows system that is associated with OrangeFS


68

The user mapping option your client is configured for (list, certificate, ldap)

Various additional settings for each of the user mapping options

Debug settings for logging and troubleshooting

Note Because the configuration files can be altered to change security information, only

administrative users should have access to change them. For security information, see your Windows documentation.

Working with the orangefstab File

The orangefstab file uses the same format as Linux/UNIX mtab (mounted file system table) files.

Here is a sample line entry in orangefstab:

tcp://orangefs.acme.com:3334/orangefs /mnt/orangefs pvfs2 defaults,noauto 0 0

Since only one file system can be mounted, only one line can be used.

The first field is a URI that specifies an OrangeFS file system server. The format is:

tcp://hostname:port/fs_name

where...

hostname = OrangeFS server host name

port = port number

fs_name = OrangeFS installation name

TCP is the only protocol supported on Windows. The default port is 3334. The file system name

can be determined from the server configuration file (default is orangefs).

The second field is the internal UNIX-style mount point. This value should be the same for all

clients (Windows or Linux/UNIX). The other fields should be left as-is above.

Working with the orangefs.cfg File

Most of the Windows Client configuration information is contained in orangefs.cfg, a text file that

contains lines in the form:

keyword option_value

You can specify comments using the # character:

# This is a comment.

Keyword: mount

The first essential keyword is mount. It specifies the drive letter associated with the mounted

OrangeFS server.

Example:

mount O:

This example will mount the file system on O: drive. (You must include the colon.) If you do not

use the mount keyword, the first alphabetically available drive, starting with E:, is used by

default.


69

Keyword: user-mode

The user-mode keyword sets the user mapping mode. The Client will not start if it is not included in

the file. The option value must be list, certificate or LDAP.

Example:

user-mode list

Note The user-mode keyword is at the top level of a hierarchy of keywords for configuring user

mapping, discussed in more detail in the next section.

Keywords: new-file-perms, new-dir-perms

The new-file-perms and new-dir-perms keywords change the initial permissions mask of newly

created files and directories. If these keywords are not present, the default permissions mask is 755

(rwxr-xr-x).

Note For more information about the permissions mask, see the Linux/UNIX chmod man page.

The keywords are used with an octal integer value representing the permissions mask.

Examples:

new-file-perms 644

new-dir-perms 700

The first example will cause new files to be created with “rw-r--r--“ permissions.

The second will create directories with “rwx------“ permissions.

Note While you can set the “sticky bit” in OrangeFS, it has no effect.

Important Ensure that the file owners always have read permissions to their own files (mask

400), and read and execute permissions to their own directories (mask 500). Otherwise,

they cannot read these files and directories after creation.

Keywords: debug, debug-file, debug-stderr

The debug, debug-file and debug-stderr keywords log detailed debugging information. If you

specify the debug keyword by itself, client-related messages are recorded in orangefs.log in the

installation directory (C:\OrangeFS\Client by default). You can change the name and location of

the log file by using the debug-file keyword.

Example:

debug-file C:\Temp\myfile.log

You can also use any of the debugging flags available with OrangeFS. For a list of these flags, see the

OrangeFS system documentation. The client flag is win_client.

Example:

debug win_client io msgpair

In this example, you would log debugging information about the client, I/O and message pair operations.


70

The debug-stderr keyword is used with no option value and prints debugging messages to the

console. This keyword is useful only if orangefs-client.exe is running as a normal executable (not

as a service).

Keywords Table

Following is a list of all keywords available for use in the orangefs.cfg file.

Keyword Description

mount Sets the Windows drive letter to represent the OrangeFS file system.

user-mode Sets the authentication/security mode used to map Windows user accounts with OrangeFS user accounts. Three possible option values:

list This mode directly matches one Windows ID with one OrangeFS UID and primary GID.

certificate This mode maps digital certificates to OrangeFS UID/GID

ldap This mode enables Windows user ID to be looked up in an identity directory that supports LDAP. Examples: Active Directory, eDirectory.

user Used only when value for user-mode keyword is list.

Specifies a user. A separate line entry with this keyword is required for each user.

Each time it is used, you must enter it in a line that occurs below the user-mode

keyword line.

ca-path Used only when value for user-mode keyword is certificate.

Sets path to file for CA (Certificate Authority) certificate. If you use this keyword,

you must enter it in a line that occurs below the user-mode keyword line.

cert-dir-

prefix Used only when value for user-mode keyword is certificate.

Sets the location of your user and proxy certificates if the user's default profile directory is not being used. If you use this keyword, you must enter it in a line that

occurs below the user-mode keyword line. The option value is the alternative path.

ldap-host Used only when value for user-mode keyword is ldap.

Sets the host computer that is running ldap. If you use this keyword, you must

enter it in a line that occurs below the user-mode keyword line.

ldap-bind-

dn Used only when value for user-mode keyword is ldap.

Sets a user DN to bind to. If you use this keyword, you must enter it in a line that

occurs below the user-mode keyword line.

ldap-bind-

password Used only when value for user-mode keyword is ldap.

Sets a user password. If you use this keyword, you must enter it in a line that

occurs below the user-mode keyword line.


71

Keyword Description

ldap-

search-

root

Used only when value for user-mode keyword is ldap.

Specifies the DN of the directory container object where searches should begin. If

you use this keyword, you must enter it in a line that occurs below the user-mode

keyword line.

ldap-

search-

class


Specifies object class that the user object must be. If you use this keyword, you

must enter it in a line that occurs below the user-mode keyword line.

ldap-

search-

scope


Sets the scope of user searches. If you use this keyword, you must enter it in a line

that occurs below the user-mode keyword line.

ldap-

naming-

attr


Sets the attribute on the user object that must exactly match the Windows user ID.

If you use this keyword, you must enter it in a line that occurs below the user-

mode keyword line.

ldap-uid-

attr Used only when value for user-mode keyword is ldap.

Specifies the attributes with store the OrangeFS UID. If you use this keyword, you


ldap-gid-

attr Used only when value for user-mode keyword is ldap.

Specifies the attributes with store the OrangeFS GID. If you use this keyword, you


new-file-

perms Specifies the permissions mask that new OrangeFS files will have.

new-dir-

perms Specifies the permissions mask that new OrangeFS directories will have.

debug Specifies for all client-related messages to be logged in orangefs.log.

debug-file Sets a custom name and location of the log file to be used for debugging (in place of

orangefs.log).

debug-

stderr Sets all debugging messages to print to console. Works only when the executable,

orangefs-client, is running (rather than the service).

Configuring the Client Security Mode

An OrangeFS installation operates in one of three security modes: default, key- and certificate-based

(see Preview Security (page 10)). You must configure the Client’s security mode to match that of the

servers or it cannot access the file system.


72

Note The “certificate” security mode is distinct from the “certificate” user-mapping mode.

The security mode is specified by the security-mode keyword in orangefs.cfg (see above). Its

value is one of default, key and certificate. (The default value is default.)

Configuring the Client for Default Security

This mode offers checking file object permissions (i.e. read, write and execute) against the owner's identity but does not prevent user impersonation. You may use any user-mapping mode with this security mode, which requires no further configuration.

Configuring the Client for Key-Based Security

In this mode, each client and server has a key pair consisting of a public key and a private key. Public keys are stored in a server-side file known as the keystore, while each client or server stores its

private key in a protected file. See Set Up Security (page 21) for instructions for generating key pairs.

The generated private key for the client should be transferred to the client’s local file system.

In key mode, the key-file keyword in orangefs.cfg must be present and must specify the

absolute path to the private key file.

Example:

key-file C:\OrangeFS\Client\orangefs-key.pem

Note You should protect this file using Windows security so that it cannot be accessed by

non-administrative accounts.

The client’s public key must be stored in the keystore on each server as normal.

Configuring the Client for Certificate-Based Security

In this mode, each user has a certificate which stores identifying information and their public key. A private key corresponds to the public key, with the certificate and private key being stored in separate files on the local (or user-shared) file system.

The key-file and cert-file keywords in orangefs.cfg specify absolute paths to these files.

However, because each pair of files is user-specific, use the %USERNAME% token to specify how the

path is formed.

Example:

key-file C:\Users\%USERNAME%\orangefs-cert-key.pem

cert-file C:\Users\%USERNAME%\orangefs-cert.pem

Additionally, a certificate and private key must be obtained for the SYSTEM user which performs basic

OrangeFS operations such as retrieving the disk space. You must place these files in the same

directory as orangefs-client.exe, C:\OrangeFS\Client by default. (See Using the orangefs-

get-user-cert App (page 72) below.)

Users can generate their own certificates and private keys using the orangefs-get-user-cert app.

Users must not be able to read other users’ private keys, so you should protect them using Windows security. Users who do not have a certificate receive an “access denied” message when attempting to access OrangeFS.

User-mapping is done on the server for this mode, so the user-mapping mode must be “server”.

Using the orangefs-get-user-cert App

An OrangeFS installation may have a dynamic pool of users who need access to files. Having an administrator generate credentials for every user request would be a needless waste of time. Instead,


73

users can use the orangefs-get-user-cert app to create a private key and retrieve a certificate

from the server.

First, users must have and know their OrangeFS user name and password; these correspond to their identity stored in the server-side LDAP directory. It is convenient to make them match their Windows credentials, but not necessary.

Then they may run the orangefs-get-user-cert app from the client. The executable is in

C:\OrangeFS\Client. You may create a shortcut to it.

Users will first be prompted for their OrangeFS user name, with their Windows user name given as a default. Then they’re prompted for their password. If these are entered correctly, their credential files

are stored in the directory specified by orangefs.cfg (typically their profile directory).

Additionally, an administrator can create a certificate for the SYSTEM user by specifying the -s option

when running the app. The OrangeFS user name is typically root (UID 0), but can be any OrangeFS user.

Below is the full usage of the app:

orangefs-get-user-cert [-h|--help] [options...] [username]

Option Description

-s

--system

generates files for

the SYSTEM user

-c path

--certfile=path

full path for certificate file storage (overrides

orangefs.cfg)

-k path

--keyfile=path

full path for private key file

storage (overrides

orangefs.cfg)

-x days

--

expiration=days

expiration time

days (default set in server configuration file)

User Mapping

The Windows Client maps Windows user IDs to OrangeFS Linux/UNIX-based UIDs for authentication.

The user-mode keyword in orangefs.cfg specifies the type of user mapping. There are three

modes of user mapping, detailed below.

List Mode

This simple form of mapping allows you to list Windows user IDs and their corresponding OrangeFS

UIDs and primary GIDs. The list is created in orangefs.cfg. Here is the format of each line:

user windows_userid uid:gid

Example:


74

user ofsuser 500:100

A separate line entry with the user keyword is required for each user. Each time you use this

keyword, you must enter it in a line that occurs below the user-mode keyword line.

File operations originating from the specified Windows user ID will be carried out on OrangeFS as the

specified UID.

Certificate Mode

This section includes:

A summary of the currently supported approach to certificate mapping for the Windows Client, including the supported software package for implementing this approach

The three types of certificates that must be in place before configuring the Windows Client for

certificate mapping

The configuration settings for certificate mapping that can be set in orangefs.cfg, either during

installation or manually

Important This topic does not discuss how to create certificates. For details on the mechanics of

generating certificates, see Notes on Installing and Using Globus Toolkit (page 77) later

in this topic.

Certificates for Grid-Computing

With OrangeFS, certificates for user mapping and security are often associated with grid computing. Therefore, the OrangeFS team chose to support the certificate generation capabilities of Globus Toolkit (an open source utilities package for grid computing) in its early implementation of the Windows

Client.

Specifically, the Globus Toolkit components used to generate certificates for the Windows Client are MyProxy and SimpleCA.

If you select the certificate mode for user mapping, the certificates must already have been

generated and placed in their appropriate locations. For more information on meeting these certificate

requirements for Windows Client, see Notes on Installing and Using Globus Toolkit (page 77) below.

Future releases of the Windows Client will address alternatives to Globus Toolkit. Until then, if you wish to implement a certificate solution other than the one used here, please contact Technical Support.

Certificate Requirements

The Client uses X.509 certificates to identify users. The certificates contain the UID and GID to be

used on the OrangeFS server. Because OrangeFS currently expects trusted clients, the certificates do not provide true security. However, they will limit the actions of typical users, such as preventing deleting files they do not own. Note that support for untrusted clients will be added to OrangeFS in an upcoming release.

Three types of certificates must be in place for the Windows Client:

CA (certificate authority)

Proxy

User

The following table describes each type.


75

Type Example Name Default Location on Windows Client

Default Location on Globus Toolkit System

CA cacert.pem C:\OrangeFS\Client\CA home/.globus/cacert.pem

...where home is the home directory of the

user who installed SimpleCA (typically

root).

Proxy cert.0 C:\Users\userid /tmp/x509up_u250

...for UID 250

User cert.1,

cert.2, ...

C:\Users\userid home/.globus/cacert.pem

...where home is the home directory of the

user who installed SimpleCA (typically

root).

Configuration File

Two configuration file keywords are associated with the certificate mode for user mapping: ca-

path and cert-dir-prefix.

To store the CA certificate in a non-default location on the Windows Client, you can add a line entry to

orangefs.cfg that begins with the ca-path keyword, followed by the custom path.

Example:

ca-path C:\Certificates\OrangeFS\CA

To store the user and proxy certificates in a non-default location on the Windows Client, you can add a

line entry to orangefs.cfg that begins with the cert-dir-prefix keyword, followed by a prefix

directory path to be placed in front of the certificate user directory.

Example:

cert-dir-prefix C:\Certificates\OrangeFS

When the Client attempts to locate the proxy and user certificates for a user, it will append the

userid as a directory name to the cert-dir-prefix. Using the above example, the certificates for

user bsmith would be placed in C:\Certificates\OrangeFS\bsmith\ using the cert-dir-

prefix above.

LDAP Mode

LDAP (Lightweight Directory Access Protocol) mapping allows the Windows user ID to be looked up in an identity directory that supports LDAP. LDAP directory examples include Microsoft Windows Active

Directory and Novell* eDirectory. Consult your directory documentation for information on LDAP.

LDAP options for the Windows Client are specified in orangefs.cfg. All keywords described in this

section must occur below user-mode ldap line entry.

Connecting over LDAP

First you must specify the host computer running LDAP. This is done with the ldap-host keyword in

the following format:

ldap-host ldap[s]://hostname:port

If ldaps is specified, a secure connection is used; otherwise, the connection is plain text. The default

secure port is 636, and the default plain text port is 389, but you can alter the port as shown above.

Example:


76

ldap-host ldaps://myldaphost.acme.com:1636

You can bind to the directory anonymously if it allows, or you can specify a user and password with

the ldap-bind-dn and ldap-bind-password keywords:

ldap-bind-dn bind_user_dn (login) ldap-bind-password password

Example:

ldap-bind-dn cn=orangefs-user,ou=special,o=acme

ldap-bind-password S3crt!

Because the password is stored in plain text in the configuration file, give the binding user minimal

rights to the directory. For more information, see LDAP Security (page 77) below.

Search Options

The Windows Client will search LDAP for the Windows user ID making the file system request. The search options specify how the directory is searched.

First, the ldap-search-root keyword specifies the DN of the directory container object where the

search should begin.

Example:

ldap-search-root ou=cluster-users,o=acme

The ldap-search-scope keyword can be either onelevel or subtree. If onelevel is specified,

only the object specified with ldap-search-root is searched—no descendant objects (sub-

containers) are searched. If subtree is specified, the object specified with ldap-search-root is

searched along with all descendant objects. The default is onelevel.

Example:

ldap-search-root subtree

The Client will form an LDAP search string in the following form:

(&(objectClass=ldap-search-class)(ldap-naming-attr=windows_userid))

The ldap-search-class keyword specifies the required object class of the user object. Typical

values are User or inetOrgPerson.

Example:

ldap-search-class User

The ldap-naming-attr keyword indicates the attribute on the user object that must exactly match

the Windows user ID. Consult your documentation to determine if the comparison is case-sensitive

(typically it is not). Typical values might be cn or name.

Example:

ldap-naming-attr cn


77

Attribute Options

The ldap-uid-attr and ldap-gid-attr keywords specify the attributes which store the OrangeFS

UID and primary GID, respectively. The Windows Client retrieves these values for use on the file system.

Example:

ldap-uid-attr uidNumber

ldap-gid-attr gidNumber

LDAP Security

Because the LDAP binding password is stored as plain text, give the binding user minimal rights to the LDAP directory. Alternatively, minimal rights can be given to users who bind anonymously—no password is stored in this case. Here are rights to consider:

Rights to search objects in the search root and below

Rights to read the object class, naming attribute, UID attribute and GID attribute from searchable

objects

No write/delete/administrator rights

For performance, UID/GID credentials are cached for a time after lookup. If you need to revoke rights, you must restart the OrangeFS Client service.

You should also use an encrypted connection to LDAP if possible, by specifying ldaps in the host URI.

Notes on Installing Globus Toolkit

This section provides supplementary information about Globus Toolkit. The information applies only to

Windows Clients that use the certificate mode for user mapping.

With OrangeFS, certificates for user mapping and security are often associated with grid computing. Therefore, the OrangeFS team chose to support the certificate generation capabilities of Globus Toolkit (an open source utilities package for grid computing) in its early implementation of the Windows Client.

Note Future releases will accommodate alternatives to the Globus Toolkit approach. Until then, if

you wish to implement a certificate solution other than the one described here, please contact Technical Support.

Whether you are new to Globus Toolkit or you have already installed it for certificate generation, the guidelines and suggestions in this section ensure optimal certificate configuration for the Windows Client.

Introduction

The Client can use X.509 certificates to identify users. The certificates contain the UID and GID to be used on the OrangeFS server. Because OrangeFS currently expects trusted clients, the certificates do

not provide true security. However, they will restrict the actions of typical users, such as deleting files they do not own. Note that support for untrusted clients will be added to OrangeFS in an upcoming release.

Identifying Certificate Format

The certificate that identifies the OrangeFS user is called the identifying certificate. It is a proxy certificate, which allows authorization on behalf of an “end entity,” in this case, a user. This user is

represented by a user certificate.

Proxy certificates contain authorization information in a data field known as a policy. For the Client,

the policy is a UTF-8 string in the form uid/gid. For example, with OpenSSL, the proxy specification

for UID 250 and primary GID 100 would be as follows:


78

language=id-ppl-anyLanguage

pathlen=0

policy=text:250/100

More information on generating this certificate is provided below.

Certificates and Validation

The identifying certificate is useful only if it can be validated against its signing certificate. The signing certificate might also require validation against the certificate that signed it, and so on, forming a certificate chain. Ultimately, the chain must end at the trusted, self-signed certificate of a certificate authority (CA).

Installing Globus Toolkit

Install Globus Toolkit on one of the OrangeFS servers or another Linux system that shares the same user information (UIDs/GIDs).

Installation instructions for Globus Toolkit can be obtained at

http://www.globus.org/toolkit/docs/latest-stable/. The Quickstart instructions will provide a

default configuration for MyProxy, including a CA called SimpleCA.

Many different security options can be configured. For example, a third-party certificate authority can be used. As long as the identifying certificate follows the format above, the client will accept the certificate.

Locating the CA Certificate

If SimpleCA is being used, the default CA certificate is home/.globus/cacert.pem, where home is

the home directory of the user who installed SimpleCA, typically root. If a third-party CA is being

used, the certificate will be located in an implementation-dependent location. The security administrator of the grid should be able to locate the file.

The CA certificate must be copied to the Windows Client system after the Client is installed. For the file

location, see Client Certificate Locations (page 79) below.

Using Grid-Based Certification

To use grid-based certification, the user must first have a user certificate. To obtain this certificate,

the user runs grid-cert-request to generate a certificate request file. At that time, the user

specifies the certificate pass phrase. This file is then delivered to the CA organization, where a human agent will review the request and return a user certificate signed by the CA certificate. The certificate

will be stored in home/.globus/usercert.pem, where home is the home directory of the user who

installed SimpleCA, typically root. If the grid installation is using SimpleCA, the certificate request

can be processed by a local administrator using the grid-ca-sign command.

The grid-proxy-init command can then be used to obtain a proxy certificate. Create a file (cert-

policy, for example) to contain the policy text, which is formatted uid/gid. For example, the file

would contain 250/100 for a user with UID 250 and GID 100. The grid-proxy-init command can

be used to generate the proxy certificate with the example cert-policy file, as follows:

grid-proxy-init -policy cert-policy -pl id-ppl-anyLanguage

When the user enters the certificate pass phrase, the proxy certificate is generated.

To simplify this command, the OrangeFS installation package includes the script Tools\pvfs2-

grid-proxy-init.sh. This will generate the policy file and run grid-proxy-init.

The resulting proxy certificate is stored by default at /tmp/x509up_uuid. Example for UID 250:

/tmp/x509up_u250

http://www.globus.org/toolkit/docs/latest-stable/


79

Transfer this certificate to the Windows Client system, along with the user certificate. For the file

location, see Client Certificate Locations (page 79) below. The proxy certificate must be renamed

cert.0, and the user certificate cert.1.

Delegating Identities for Clusters

The use of identifying proxy certificates allows the identity of the user to be separated from the actual Windows user ID making a file system request. This ability is useful for clusters.

For example, when a user with a Windows user ID of JSmith executes a job on a cluster node, the

job scheduler uses Windows user ID ClusterUser.

The system administrator would set the certificate directory prefix to C:\ClusterWork. A directory

called ClusterUser would be created under ClusterWork. The job scheduler would transfer

certificates to the C:\ClusterWork\ClusterUser directory. When ClusterUser makes file system

requests, it will use the certificates of JSmith, so requests on the file system will use the UID of

JSmith. When a different user uses the node, that user’s certificates will be used.

Certificate Expiration and Renewal

For performance, the Client caches the OrangeFS user identity (UID/GID) until the proxy certificate expires.

By default, Globus Toolkit proxy certificates expire after 12 hours. If jobs requiring more time are expected, a means for the user to renew the certificate should be provided.

One way to do this is to have the user to run grid-proxy-init again. This will overwrite the current proxy. Then the new proxy certificate can be transferred to the Client system (overwriting the current

certificate) without interrupting the current job.

Client Certificate Locations

The certificates are stored as PEM-format files on the Windows Client system. The identifying

certificate’s name is cert.0. Because the identifying certificate is associated with a Windows user, it

is stored in its user’s profile directory by default. On most systems this is C:\Users\.

Example: C:\Users\jsmith

Alternatively, you can specify a certificate prefix directory in the client configuration file,

C:\OrangeFS\Client\orangefs.cfg by default. Use the cert-dir-prefix keyword to specify

this directory. The user’s userid will be appended as a directory name to the prefix directory.

Example configuration file line entry:

cert-dir-prefix M:\OrangeFS Users

For user jsmith, the identifying certificate will be M:\OrangeFS Users\jsmith\cert.0.

The identifying certificate must be verified by its end-entity (sometimes called a user) certificate. Place

this certificate in the same directory as the identifying certificate, with the name cert.1. Additional

intermediate certificates can be placed in the same directory with names cert.2, cert.3, and so on.

The CA certificate is placed in the OrangeFS CA directory with the name cacert.pem. By default this

is C:\OrangeFS\Client\CA\cacert.pem. This path can be changed in the configuration file using

the ca-path directive in the configuration file.

Example:

ca-path M:\OrangeFS Certificates\orangefs-cacert.pem

Troubleshooting

To troubleshoot problems, check the Application Event Log in the Event Viewer utility. You can also

turn on detailed debugging (see Working_With_The_orangefs.cfg_File (page 68)).


80

Startup errors are logged to the Windows Event Log.

The configuration file has some strict requirements, so the Windows Client will log an error to Event Log and exit if there is a problem. The event message should give an exact explanation of the problem with the configuration file. Correct the problem and restart the OrangeFS Client service.

ensure network connectivity is available between the Client system and the server(s) hosting OrangeFS. Check firewall settings and network access lists.

For information about the debug and related keywords, see Configuration (page 67). You can use the

generated file orangefs.log to diagnose problems. A file named service.log is also created in

the installation directory when debugging is enabled and can provide more detail on startup errors.

Note that many debug messages are low-level and require extensive knowledge of OrangeFS/PVFS2 to interpret. For more information, consult the OrangeFS and PVFS2 system documentation.

Free and commercial support is available at http://www.orangefs.org.

Source Code

The OrangeFS team intends to provide all source code needed for building the Client.

Currently, a source code package is available at http://www.orangefs.org. (The Windows package is

separate from the Linux/UNIX package.) Build instructions will be released at a later date.

Web Pack Clients

OrangeFS Web Pack Clients

OrangeFS Web Pack provides native WebDAV and S3 client access to OrangeFS through tight integration with Apache Web Server. Clients are implemented as standard Apache Modules, so

installation and configuration are independent of the file system.

Installation tasks are organized into these categories:

Modifying OrangeFS (page 81)

(if necessary)

Ensure the PVFS Library is:

A shared library

Available to Apache and the Web Pack software

Modifying Apache (page 82) (if

necessary)

Ensure that four needed software features/packages are included in your installation of Apache Web Server.

Installing OrangeFS Web

Pack (page 82)

Download, extract and build the Web Pack modules.

Setting up OrangeFS Web

Pack for Security (page 84)

Designate authority to create credentials for security.

Editing httpd.conf (page 85) Add line entries to the Apache configuration file, depending on the setup you wish to support for WebDAV and/or S3.

Using WebDAV Clients (page 89)

and S3 Clients (page 90)

Overview of clients supported by the Web Pack.




81

Modifying OrangeFS (if necessary)

Before you install the Web Pack, do the following regarding the PVFS Library in your OrangeFS installation:

Ensure the PVFS Library is available as a shared library (page 81)

Ensure that your Apache and Web Pack Software can locate the PVFS Library. (page 81)

Ensure the PVFS Library is a Shared Library

1. The clients in the Web Pack require the shared version of the OrangeFS PVFS Library

(lib/libpvfs2.so).

2. By default, only the static version of this library (lib/libpvfs2.a) is created during installation,

so you will probably have to add the shared library by building a new Makefile.

3. To build the new Makefile:

a. Change Directory (cd) to the directory where you extracted the original OrangeFS tar file.

b. Run the same ./configure command you used when you installed OrangeFS, but add the

following additional option:

i. --enable-shared

c. After entering the updated ./configure command, run make and make install.

Notes For more information about these commands, see 1 Build OrangeFS (page 18).

For a record of the options included in your original ./configure command, check the

contents of the config.log file, located in the same directory.

Ensure the PVFS Library can be Located

Installing the Web Pack will add two Apache Modules (the WebDav and S3 clients). When either

module is loaded via LoadModule, Apache effectively becomes an OrangeFS client. This means

Apache must be able to find the PVFS Library.

If you installed your PVFS Libary where Apache can't find it, add the location either to your

/etc/ld.so.conf file or a separate .conf file you create in the /etc/ld.so.conf.d directory.

For example, if your OrangeFS installation was located in /opt/orangefs, you could create a file

called /etc/ld.so.conf.d/orangefs.conf containing the following line entry:

/opt/orangefs/lib

After you add the location, run ldconfig or reboot the system.

Notes If you are an individual user who has installed OrangeFS in a local directory, you also can

edit the LD_LIBRARY_PATH shell variable to include this location.

If you want to limit the modification to Apache Web Server only, you can add the following

line entry to Apache's environment variables (envvars) script: export

LD_LIBRARY_PATH


82

Modifying Apache (if necessary)

Note If you installed Apache with a package manager, this step is unnecessary.

If you compiled Apache Web Server from sources, it is likely you must build a new Makefile before

you install the Web Pack.

The ./configure statement you use must include the following options:

Option Description

--enable-so Allows Apache to load modules at start-up time

--enable-dav Enables WebDAV

--enable-dbd Enables Apache DBD Framework (the OrangeFS WebDAV client uses

Berkeley DB)

--with-included-

apr Uses the APR libraries provided with your distribution

To add the above items by building a new Makefile:

1. Change Directory (cd) to the directory where you extracted your original Apache Web Server

installation files.

2. Run the same ./configure command used when you installed Apache, but add the arguments:

--enable-so --enable-dav --enable-dbd --with-included-apr

3. After entering the updated ./configure command, run make and make install.

Note For a record of the options you included in your original ./configure command, check

the contents of the config.log file, located in the same directory.

Installing OrangeFS Web Pack


System Requirements (page 82)

Steps for Installing the Web Pack (page 83)

System Requirements

Apache Module Dependencies

In addition to a base installation of Linux with Apache Web Server, the OrangeFS Web Pack clients

require up to four more Linux software packages. The names for these packages vary from one Linux distribution to another. For example, following are the package names you would require on a system

running RHEL:

Package Name for RHEL Needed for WebDAV Needed for S3

apr-devel

httpd-devel


83

Package Name for RHEL Needed for WebDAV Needed for S3

apxs (or apxs2)

libxml2-devel

The method for installing these packages also varies from one Linux distribution to another. For example, to automatically install the required packages on a system running RHEL, enter the following command:

yum install apr-devel httpd-devel libxml2-devel

Note You do not need to specify apxs in the above command because it is included with httpd-

devel.

PVFS Library

The clients in the Web Pack require the shared version of the OrangeFS PVFS Library

(lib/libpvfs2.so). See Modifying OrangeFS (page 81) for more information about making this

library available.

Steps

To install the OrangeFS Web Pack, follow these steps:

1. Obtain the OrangeFS source. If this is the same machine the OrangeFS server is running on, this has already been done. If following our instructions, it is in /tmp/src. If the OrangeFS server is not on this machine, you must download but not compile the source as described in the OrangeFS

installation instructions (page 1).

2. Change Directory (cd) to /tmp/src, and extract the compressed tar file, then change directory

into the webpack source directory.

tar -xzf orangefs-2.9.tar.gz

cd orangefs-2.9/src/client/webpack

Following is a sample listing of initial files in the orangefs-webpack-2.9 download directory:

# ls

aclocal.m4 AUTHORS autom4te.cache/ ChangeLog

config.guess*

config.h.in config.sub* configure* configure.ac

COPYING

d.admin/ d.authn/ d.dav/ d.s3/

depcomp*

INSTALL install-

sh* ltmain.sh@ m4/ Makefile.am

Makefile.in missing* NEWS pvfsinit.sh

README

3. Build a Makefile for the Web Pack that includes the installation location and the modules you wish to compile. Listed below are several module options with descriptions:

--enable-admin Enables the mod_orangefs_admin module.

--enable-authn Enables the mod_authn_orangefs module.

--enable-dav Enables the mod_dav_orangefs module.

--enable-s3 Enables the mod_orangefs_s3 module.


84

For example:

./configure --enable-dav --enable-s3

If you installed the Apache Module dependencies using your OS package manager, you should be able to run this command without specifying them.

If any of the dependencies are not found when you run ./configure, you will see error

messages about how to add them.

Following is an example of running ./configure with arguments to specify where to find some of

the dependencies:

./configure --enable-dav --enable-s3 \

--with-apxs=/usr/local/dav/bin/apxs \

--with-pvfs2-config=/o2.8.5/install/bin/pvfs2-config \

--with-pvfs2-source=`pwd`/../../..

4. Continue with the standard Linux commands to build and run the Makefile:

make && make install

Notes When you enter make install, apxs will cause LoadModule statements for each of

the Web Pack modules to be added to httpd.conf.

The modules will also require some line entries to be manually added to httpd.conf,

as explained in Editing httpd.conf (page 85).

Setting up OrangeFS Web Pack for Security

OrangeFS Web Pack processes run as a service user such as daemon on Linux. (See your Apache

documentation regarding the User and Group configuration file settings.) As the Apache HTTP

server may operate on behalf of any user, it must be able to generate credentials for any user rather than its active user.

If your OrangeFS installation uses certificate-based security, you must create an orangefs-

service-users file to specify which service users are authorized to create credentials for other

users. The default location for this file is in the etc directory under the OrangeFS installation directory

(/opt/orangefs by default). You can override this location by specifying the PVFS2_SERVICEFILE

environment variable prior to running the Apache HTTP server.

The file contains the usernames of service users, one per line. Follow these steps to create the file:

1. Edit a new file:

vi /opt/orangefs/etc/orangefs-service-users

2. Add service users, one per line:

daemon

bin

3. Save and close the file.

4. Important: set the file accessible by root only:


85

chown root:root /opt/orangefs/etc/orangefs-service-users

chmod 600 /opt/orangefs/etc/orangefs-service-users

These service users will be able to generate credentials immediately, so there is no need to restart

pvfs2-server.

Editing httpd.conf

This topic discusses various line entries to consider for httpd.conf in the following areas:

Introduction (page 85)

Configuring WebDAV (page 85)

Configuring S3 (page 87)

Introduction

An OrangeFS client must initialize the file system during startup. Apache can have multiple file system modules installed, so the module that will start the file system must be configured. The global

directive PVFSInit must be set to the name of the module which will initialize OrangeFS. This can be

completely arbitrary and is normally set to the first module that was set up.

For example, the following command sets PVFSInit to mod_dav_orangefs:

PVFSInit mod_dav_orangefs

Configuring WebDAV

WebDAV configuration in the http.conf file involves two main areas:

Basic Support of WebDAV (page 85)

Authentication and Authorization (page 86)

Basic Support of WebDAV

There are a total four custom line entries that can be added to the httpd.conf file in association

with the WebDAV component of the OrangeFS Web Pack.

Line Entry Description

PVFSInit The global apache directive PVFSInit should be set to the name of

the module that will perform the OrangeFS initialization procedure

(either mod_dav_orangefs, mod_orangefs_s3,

mod_orangefs_admin, mod_authn_orangefs).

DAV mod_dav_orangefs Enables Apache's mod_dav in combination with the OrangeFS file

system repository.

DAVpvfsCertPath

/path/to/certificates This option is required only if the certificate security mode is used,

when the OrangeFS Authentication (mod_authn_orangefs) module

must be enabled. The authentication module is also set up to put certificates in this directory.

PutBufSize byte_value This directory/location configuration directive causes

mod_dav_orangefs to write blocks of the size indicated (in bytes)

when HTTP PUT is encountered. The default is one megabyte

(1048576).


86


ReadBufSize

byte_value This directory/location configuration directive comes into play in two

situations: COPY and GET. When an HTTP COPY causes a copy of a file

to be made, the source file is both read and written in ReadBufSize

blocks. When an HTTP GET for a file is encountered, the file is read in

ReadBufSize blocks. The default is one megabyte (1048576).

If, for some reason, you don't need authentication and authorization, you can configure Apache to access your OrangeFS file system via the WebDAV module as follows:

<Location /mnt/ornagefs>

DAV mod_dav_orangefs

</Location>

In the above example, only the files and directories accessible to the UID/GID used by Apache will be

available. DavPVFSInit Off is needed if you load more than one OrangeFS Apache Module.

Authentication and Authorization

An OrangeFS file system is filled with files and directories, all of which have normal Linux/UNIX permissions, UIDs, and GIDs.

For security reasons, Apache typically runs as an unprivileged user (daemon/UID = 2, for example). When it loads the OrangeFS modules, Apache is an OrangeFS client; an OrangeFS client’s credentials cue normally the UID and GID of the client process.

An unprivileged OrangeFS client normally uses the default settings for its credentials. Apache is a special case because it services requests from many users.

Two separate issues must be configured: authentication by a WebDAV client to Apache, and authentication by Apache to OrangeFS.

In default security mode, Apache needs only the username from itself and the UID from the system

database or LDAP.

In key-based security mode, Apache operates as in default security mode, except that an additional

configuration is required to give Apache permission to impersonate the other users.

In certificate-based security mode, Apache must be configured with the OrangeFS authentication module so it can obtain user certificates.

Two sample methods follow for setting up authentication and authorization:

Local Method (page 86)

LDAP Method (page 87)

Local Method

You can attain local authentication and authorization with the WebDAV module as follows:

Use normal Apache auth, preferably in digest mode, since basic puts clear-text (encoded with

base64) passwords on the wire.

If you use https, it could be safe to use basic auth, since the whole stream on the wire is

protected.

The local machine password must also have an htpasswd/htdigest file whose symbolic user

names match the ones in the local database. In addition to other ways you could obtain such a file, one can be made from the local password database with all the administrative/non-OrangeFS users

removed. Passwords encrypted with SHA-512 are probably best, since the htpasswd/htdigest

file is readable by Apache (though it shouldn't be exposed), but digest mode requires you to use MD5.


87

Unless it is configured otherwise, the local password database is the files /etc/passwd,

/etc/group, and /etc/shadow on Linux.

Given this setup, here is an example of the supporting line entries to include in the http.conf file:

<Location /pvfsmnt>


AuthType Digest

AuthName "Digest Auth Test"

AuthDigestDomain /pvfsmnt

AuthUserFile /usr/local/http/digest.file

AuthDigestProvider file

Require valid-user

</Location>

LDAP Method

A standard LDAP configuration on Apache will work for authentication and authorization with the

WebDAV module, provided you have an LDAP tree to contain your users with the appropriate UID/GID attributes.

In the following example, ldapsearch has been used to verify the existence of a simple LDAP tree

suitable for this purpose:

# ldapsearch -x -b 'dc=omnibond,dc=com' '(objectclass=person)'

dn: uid=luser1,ou=users,dc=omnibond,dc=com

uid: luser1

uidNumber: 1001

gidNumber: 1001

userPassword:: e1NTSEF9TndEUEdHQ0xmNHFGaSs3a0oydlFCd2NISDlxTVk1Ujk=

Given this setup, here is an example of the supporting line entries to include in the http.conf file:

<Location /pvfsmnt>


AuthType Basic

AuthName "ldap auth"

AuthBasicProvider ldap

AuthzLDAPAuthoritative off

AuthLDAPURL

"ldap://valkyrie.omnibond.com/ou=users,dc=omnibond,dc=com?uid,uidNumber,gidNumber

??(objectClass=*)"

Require valid-user

</Location>

The above example defines an LDAP tree residing in the domain named valkyrie.omnibond.com.

Configuring S3

Note S3 is no longer supported in any security mode due to difficulties with the S3 protocol and

the security protocol. S3 is still supported in default security mode.

The content you add to httpd.conf for S3 depends on the specific client you implement.

Here is a list of possible line entries that can be added for the S3 component of the OrangeFS Web Pack.


88


SetHandler

orangefs_s3 Enables the OrangeFS S3 Apache module (mod_orangefs_s3).

BucketRoot

/path Sets the physical file path to find S3 buckets.

AuthType AWS Instructs Apache to use the AWS authentication extension, implemented by the

mod_orangefs_s3 module. This method sends a signed authentication header,

which is validated by mod_orangefs_s3. This is required for the S3 protocol.

AWSAccount

userid

password uid

gid

This directive defines an account to be used by mod_orangefs_s3. If the uid or

gid operands are omitted, mod_orangefs_s3 will use the underlying operating

system to retrieve them.

For example, the implementation of the s3cmd client described in the later topic, S3 Clients (page 90),

could be supported by the following line entries in httpd.conf:

Listen 81

<VirtualHost *:81>

SetHandler orangefs_s3

BucketRoot /orangefsmnt/s3

<Location />

AuthType AWS

AWSAccount username1 cleartextpassword 400 500


Require valid-user

</Location>

</VirtualHost>

This example causes Apache to listen for s3 connections on port 81. All the buckets will be stored in

the OrangeFS file system mounted at /orangefsmnt in a directory named s3. The example shows

valid users being listed directly in the httpd.conf file, but they can also be in LDAP. Username1's

UID is 400 and GID is 500.

Configuring OrangeFS Authentication

The authentication module is necessary only in certificate-based security mode. It generates a

temporary certificate for any user connecting through the web, so the web server can communicate with the filesystem on the user's behalf.

Line entries necessary for OrangeFS Authentication configuration include:


AuthOrangeFSCertPath The path where user keys and certificates are stored. The web server must be able to write here.

AuthOrangeFSCertValidity Set this to the desired certificate validity in minutes. Defaults to 1 hour.

AuthOrangeFSMountPoint The mount point of the filesystem consulted for user information.


89

OrangeFS Authentication Setup

You must specify a directory where the authentication module will store the certificates. If the web

server is installed into /var/www, an appropriate directory might be /var/www/certs, but you can

select a custom location for your installation.

Below is an example of OrangeFS Authentication configuration commands.

AuthOrangeFSCertPath /var/www/certs

AuthOrangeFSCertValidity 30

AuthOrangeFSMountPoint /mnt/orangefs

Important Remember to ensure that the /var/www/certs directory exists and that the Apache

web server can write to it.

OrangeFS Key-based Security

If the Apache web server is not running as root and OrangeFS is in key-based security mode, the user

which it runs as (typically apache2 or www) must have permission to generate credentials for the

entire file system.

The file /opt/orangefs/etc/orangefs-service-users should contain apache2 if the apache2

user requires permission to generate credentials for the entire file system.

Note See Administrating Key-Based Security in the Administration Guide on orangefs-

service-users for more information.

WebDAV Clients

Support for the WebDAV client protocol is ubiquitous in applications and environments where Web access is required.

Client Examples

Following are examples for accessing a WebDAV-enabled OrangeFS installation located at

http://ofs.omnibond.com:

In Windows, enter the following at the command prompt:

net use W: http://ofs.omnibond.com/pvfsmnt

In Mac OS, enter the following at a shell prompt:

mount -t webdav http://ofs.omnibond.com/pvfsmnt /Volumes/ofs

In Linux, ensure that the davfs2 package is installed and enter the following command:

mount -t davfs http://ofs.omnibond.com/pvfsmnt /mnt

In a Web browser, enter the following location in the address line:

http://ofs.omnibond.com/pvfsmnt

Testing WebDAV

To test compliance for a WebDAV client, you can use the testing suite known as litmus, available

here.

http://www.webdav.org/neon/litmus/


90

S3 Clients

The OrangeFS S3 Client enables Amazon Simple Storage Service (S3) client tools to access OrangeFS/PVFS2 file systems.

For more information on Amazon S3, visit the following website:

http://aws.amazon.com/s3/

Note S3 does not work in any security mode. The S3 protocol and OrangeFS protocols are not compatible.

S3 Client Implementation Example

S3cmd (http://s3tools.org/s3cmd) is a popular command-line client you can use to demonstrate

functionality of the mod_orangefs_s3 module.

The remainder of this topic describes the following setup considerations for using s3cmd with OrangeFS:

Sample configuration file (page 90)

Additions to httpd.conf (page 90)

Resolving your S3 bucket with DNS (page 91)

Command examples (page 91)

Support exceptions (page 92)

Sample Configuration File

The s3cmd program requires a configuration file. Here is an example:

$ cat /home/s3user/.s3cfg

access_key = username1

acl_public = False

bucket_location = US

debug_syncmatch = False

default_mime_type = binary/octet-stream

delete_removed = False

dry_run = False

encrypt = False

force = False

guess_mime_type = False

host_base = s3server.dns.name:81

host_bucket = %(bucket)s.s3server.dns.name:81

service_path = /

human_readable_sizes = False

preserve_attrs = True

proxy_host =

proxy_port = 0

recv_chunk = 4096

secret_key = cleartextpassword

send_chunk = 4096

use_https = False

#verbosity = DEBUG

Additions to httpd.conf

This example implementation of the s3cmd client includes the following line entries in httpd.conf:

http://aws.amazon.com/s3/

http://s3tools.org/s3cmd


91

Listen 81

<VirtualHost *:81>

SetHandler orangefs_s3

BucketRoot /orangefsmnt/s3

<Location />

AuthType AWS



Require valid-user

</Location>

</VirtualHost>

The above example causes Apache to listen for S3 connections on port 81. All the buckets will be

stored in the OrangeFS file system mounted at /orangefsmnt in a directory named s3. The example

shows valid users being listed directly in the httpd.conf file, but they can also be in LDAP.

Username1's UID is 400 and GID is 500.

Note For more discussion of http.conf, see Editing httpd.conf (page 85).

Resolving your S3 Bucket with DNS

In S3 terminology, a bucket is a location for storing data objects. Your bucket name must be

resolvable in DNS, as explained in the S3 documentation.

One simple way to get your buckets to resolve, at least for testing, is to add them to the /etc/hosts

file on both your Apache Web Server and the computer where your S3 client runs.

For example, if the Apache server running your mod_orangefs_s3 module is s3server.dns.name,

and if s3server.dns.name's IP address is 10.11.12.13, if you want to create and access a bucket

named bucketname, you could add this line to both /etc/hosts files:

10.11.12.13 bucketname.s3server.dns.name

Command Examples

Following are examples of s3cmd commands.

Note If you experience problems, both s3cmd and s3_orangefs have debug modes.

Command Description

s3cmd --help Display the s3cmd Help page.

s3cmd mb s3://bucketname Make a bucket named bucketname.

s3cmd rb s3://bucketname Remove a bucket.

s3cmd ls List all your buckets.

For example:

$ s3cmd ls

2012-05-29 11:24 s3://bucketname1

2012-05-29 11:26 s3://bucketname2

s3cmd ls s3://bucketname List all the objects in a bucket.

http://docs.amazonwebservices.com/AmazonS3/latest/dev/VirtualHosting.html


92

Command Description

s3cmd sync dirname s3://bucketname Synchronize all the objects in a directory named

dirname to a bucket.

For example:

$ find dirname

dirname

dirname/filename1

dirname/dirname2

dirname/filename2

$ s3cmd sync dirname s3://bucketname

dirname/filename1 ->

s3://bucketname/dirname/filename1 [1

of 2]

0 of 0 0% in 0s 0.00 B/s

done

dirname/filename2 ->

s3://bucketname/dirname/filename2 [2

of 2]

0 of 0 0% in 0s 0.00 B/s

done

$ s3cmd ls s3://hubcap

2012-05-29 11:24 0

s3://hubcap/dirname/filename1

2012-05-29 11:24 0

s3://hubcap/dirname/filename2

s3cmd get

s3://bucketname/dirname/filename

localfilename

Get a file from a bucket named. Rename it

localfilename if desired.

s3cmd put /dirname/filename

s3://bucketname Put a file into a bucket.

s3cmd del

s3://bucketname/dirname/filename Delete a file from a bucket.

Support Exceptions

In the current release of the OrangeFS Web Pack, the following bucket features are not supported by

mod_orangefs_s3:

ACL

Policy

Lifecycle

Location

Logging

Notification

Versions

RequestPayment

Versioning

Website


93

Hadoop Client

Apache Hadoop is an open source framework that supports data-intensive distributed applications. Hadoop has many parts, but two are fundamental:

MapReduce is the framework that understands and assigns work to the nodes in a cluster. MapReduce divides the application into many fragments of work, each of which can be executed or re-executed on any node in the cluster.

HDFS (Hadoop File System) spans all the nodes in a Hadoop cluster for data storage.

The OrangeFS Hadoop Client is an HCFS plug-in which allows you to run Apache Hadoop version

1.2.1 and 2.4.1 with the OrangeFS distributed file system replacing Hadoop's HDFS filesystem. Together, these two open source products can perform massive computations on the petabyte scale.

OrangeFS also permits modification of data within the file system.

Notes You may also configure an existing Hadoop cluster using HDFS as the default distributed file system to use OrangeFS as an alternative storage solution.

Other versions of Apache Hadoop 1.x.x and 2.x.x will likely work with the OrangeFS Hadoop Client but have not been fully tested. Brief instructions for building the OrangeFS Hadoop

client for a particular Hadoop release are provided in the setup guides.

Before you begin, read Planning For Hadoop Installation (page 93) for prerequisites and

choices you must make during installation. Advance planning for these considerations ensures a smoother installation process.

This section includes the following topics:

Planning for Installation (page 93)

HPC Setup (page 95)

Traditional Hadoop Setup (page 108)

Planning For Installation

Before you begin installing the OrangeFS Hadoop Client, you must select either an HPC or a traditional

Hadoop storage option to use with the OrangeFS Hadoop Client. You can then preview the system and software requirements before moving on to the appropriate installation topic.

Understanding the Architecture

The basic design that enables MapReduce to work with OrangeFS integrates the OrangeFS Hadoop Client, the OrangeFS Java Native Interface (JNI) Shim, and the OrangeFS Direct Interface (DI).

Apache Hadoop is designed to support file systems other than HDFS through an abstract file system API. An implementation of this API, the OrangeFS Hadoop Client enables MapReduce to interface with the OrangeFS JNI Shim.

The OrangeFS JNI Shim utilizes the JNI, a programming framework that enables Java code running in a Java Virtual Machine to interface with native programs. In this case, the Java code is the OrangeFS Hadoop Client and the native code is the DI, with their interaction facilitated by the OrangeFS JNI

Shim.

The OrangeFS DI is a Linux client interface written in C, which enables POSIX-like and direct system calls to the OrangeFS API, directing operations to OrangeFS data/metadata servers.

Understanding Storage Options

HPC vs. Traditional

You can set up the OrangeFS storage for your OrangeFS Hadoop Client in two ways.

https://www.google.com/url?q=https://wiki.apache.org/hadoop/HCFS&sa=D&sntz=1&usg=AFQjCNHWZmiUahISFtopF-sJ8nUqfYB3MA


94

Setup Option

Description Design

HPC In this setup, Hadoop MapReduce accesses an OrangeFS file system through a single mount

point to any of the OrangeFS servers.

Traditional Hadoop

This setup simulates a traditional Hadoop installation, running a

client and server program

on each server in an OrangeFS cluster. This model represents the colocated compute and storage resources typical of most Hadoop clusters.

Linux Operating System

All server and client systems should use the same distribution of Linux. Guidelines for selecting a

Linux distribution in Preview System Requirements (page 5) also apply to any systems used with the

OrangeFS Hadoop Client.

Note For consistency, all topics about the OrangeFS Hadoop client use RHEL command line syntax wherever distribution-specific commands are required.

Common System Requirements

The HPC and the Traditional Hadoop installation configurations share a number of common requirements.

Preparing the Build System

Many of the instructions will prepare the OrangeFS Hadoop Client build system, including downloading, installing and configuring both Hadoop and OrangeFS. For both configurations, the administrator should select a single node out of the desired pool of nodes which will run MapReduce to act as the build system. Some tasks affect all clients, while others are focused on the single build system. This

will produce a directory of software that must be copied from the build system node to the desired installation directory on each client node.

JDK

The build system requires the Java Development Kit (JDK) to build the OrangeFS Hadoop Client and

OrangeFS JNI Shim.

Note For additional guidance on the appropriate JDK version, consult the Apache Hadoop

recommendations.

Maven

To build the OrangeFS Hadoop Client, Maven must be installed on the build system.

http://wiki.apache.org/hadoop/HadoopJavaVersions



95

Preparing Individual Client Nodes

JRE or JDK

While some Apache Hadoop related projects require the JDK for proper functionality (Sqoop, for example), only the Java Runtime Environment (JRE) is required to run MapReduce on client nodes. Your requirements will determine which one you should install.

Note It might be easiest to install the JDK on all client nodes, which produces no adverse results.

Hadoop Binaries

The Hadoop binaries are required to “run” Hadoop and use the OrangeFS Hadoop Client. You must download, extract, and copy the Hadoop binaries archive to each node.

System Variables

On each client node, you must eventually set the environment variables LD_LIBRARY_PATH,

JNI_LIBRARY_PATH, and PVFS2TAB_FILE to run MapReduce with OrangeFS.

Installation Requirement Differences

Installation instructions are separated into two topics, according to your selected storage option. Following are some of the differences in their content.

HPC Setup

Assumes that OrangeFS servers have already been configured and installed on the storage cluster. For more information on completing that setup process prior to setting up the OrangeFS Hadoop

Client, see the beginning (page 1) of the Installation Guide.

Hadoop was not originally designed to work in a scheduled HPC environment, but you can use a customized version of myHadoop, myHadoop-orangefs, with PBS to support on-demand clusters.

Additional steps are required to incorporate myHadoop-orangefs here. This approach has been

tested with PBS Professional version 12.0.0.x and myHadoop-orangefs version 0.1.

Traditional Hadoop Setup

Assumes that you will configure and install OrangeFS client and server libraries/binaries on all

desired nodes in your cluster.

Hadoop HPC Setup

This topic explains how to install the OrangeFS Hadoop client to run in an HPC environment.

Important Prior to performing these steps, you must install a supported distribution of Linux on all

client systems that will run Hadoop, and you must select one of these nodes to serve as your Hadoop build system. You must install OrangeFS on your storage nodes and the OrangeFS system must be online

prior to performing these steps. If you have not completed this step, see the beginning

(page 1) of the Installation Guide for instructions to complete this step before proceeding.

Setting up the Hadoop client involves four main steps, with two optional steps for testing and for configuring for a scheduled HPC environment.

Install System Software (page 96)

Configure Hadoop to Use OrangeFS (page 99)

Copy Hadoop client build system Software To Other Clients (page 103)

Start Hadoop On Each client (page 104)

Verify Installation (page 106) (Optional but recommended)

Support for Scheduled Environments (page 107) (Optional)


96

Notes Most of the following steps require that you have root permissions.

Root permission is not required when using myHadoop-orangefs in a scheduled

environment (page 107).

Clients and servers MUST be using the same version of the OrangeFS filesystem. For more information about choosing the right setup option and a supported Linux

distribution, see Planning for Installation (page 93).

For consistency, all topics about the OrangeFS Hadoop client use RHEL command line syntax wherever distribution-specific commands are required.

Install System Software


You must configure all Hadoop clients to support secure shell connections via SSH without passing a



Select Hadoop Client Build System

After you have identified the set of nodes which will run Hadoop, select a node to configure as your Hadoop client build system.

Example: node001 from { node001 - node999 }

Install the Java Development Kit (All Client Systems)

Install the Java Development Kit (JDK) on each node, including the Hadoop client build system.

For guidance on the appropriate version to install, consult the following recommendations listed on

the Apache Hadoop Wiki.

For details on installation, consult Oracle’s Installation Guide.

Note Only the Java Runtime Environment is needed to run MapReduce, but the JDK is required on the Hadoop client build system to build the OrangeFS Hadoop Client. For easier installation, we recommend simply installing the JDK on all client nodes.

Maven (Hadoop Client Build System)

To install Maven on the Hadoop Client Build System, follow these steps:

1. Download the stable Maven binary tar.gz file from the Apache Maven Project.

2. Copy the downloaded Maven tar file to the /tmp directory on your Hadoop client build system

using scp or a similar program.

3. Extract contents of archive to /opt:

tar -C /opt -xzf /tmp/apache-maven-version-bin.tar.gz

where...

version = version number of the Maven release

Example: apache-maven-3.2.3-bin.tar.gz

4. Set the following environment variables:



http://www.oracle.com/technetwork/java/javase/downloads/index.html

http://maven.apache.org/download.cgi


97

export M2_HOME=/opt/apache-maven-version

export M2=$M2_HOME/bin

export JAVA_HOME=jdk_path

export PATH=$JAVA_HOME/bin:$M2:$PATH

Note You may set the above environment variables in a terminal but they will not persist once the terminal has been closed or the system rebooted. To make these environment

variables persist, add them to /etc/profile or /etc/profile.d/maven.sh and

reboot your system.

4. Verify your installation of Maven:

mvn --version

Important It is critical to set JAVA_HOME correctly.

Hadoop Binaries (All client systems)

To download and install the Hadoop binaries, follow these steps:

1. Download a supported Apache Hadoop release from http://hadoop.apache.org/releases.html and copy it to the /tmp directory on the Hadoop client build system. The filename should

resemble: hadoop-version-bin.tar.gz for Hadoop 1 and hadoop-version.tar.gz for

Hadoop 2.

Notes At the time these instructions were published, the most recent stable version of Hadoop

binaries tested with OrangeFS is hadoop-2.4.1.

On each client system, you must use the same version of Apache Hadoop binaries as the version of the Hadoop source used on the Hadoop client build system.

2. Extract the Hadoop binaries archive to /opt by running the following command on the Hadoop

client build system:

tar -C /opt -xzf /tmp/hadoop-version.tar.gz

where...

version = version number of the Hadoop distribution release

Example: hadoop-2.4.1.tar.gz

Note After you customize the configuration files, you will copy this directory to each client node.

OrangeFS (Hadoop Client Build System)

Download the OrangeFS tar file and copy it to the Hadoop client build system. Then, extract and build

OrangeFS, ultimately installing in /opt on the Hadoop client build system.


1. Go to http://www.orangefs.org/download/ and download the tar file orangefs-

version.tar.gz, where version = version number of the OrangeFS release.

2. Copy the OrangeFS tar file to the /tmp directory on the Hadoop client build system.

3. On the Hadoop client build system, extract the contents of the archive into the /tmp/src

directory:

http://hadoop.apache.org/releases.html



98

tar -C /tmp/src -xzf /tmp/orangefs-version.tar.gz

4. On the Hadoop client build system, generate a Makefile that enables the OrangeFS Hadoop Client and OrangeFS JNI Shim, using the following commands:

cd /tmp/src/orangefs-version

./prepare

./configure \

--prefix=/opt/orangefs \

--enable-shared \

--disable-server \

--enable-jni \

--with-jdk=jdk_path \

where...

jdk_path = path to jdk

Example: /usr/java/jdk1.6.0_45

5. Continue with the standard Linux commands to build and install OrangeFS to /opt:

make

make install

6. Change directory to the appropriate Hadoop client directory. Pick orangefs-hadoop1 if you are building a client for Apache Hadoop 1.x.x or orangefs-hadoop2 if you are building a client for Apache Hadoop 2.x.x:

cd /tmp/src/orangefs-2.9.0/src/client/hadoop/orangefs-hadoop1

OR


7. A pom.xml file is located in each the above directories and is used by Maven to build the OrangeFS

Hadoop Client jar using a specified version of Hadoop. For orangefs-hadoop1 the default is Apache Hadoop 1.2.1. For orangefs-hadoop2 the default is Apache Hadoop 2.4.1. The specified version of Hadoop is indicated by a property present in the pom.xml file. These pom.xml files may be customized to your liking but do so at your own risk. Build the OrangeFS Hadoop Client jar:

mvn -DskipTests clean package && cp target/orangefs-hadoop1-2.9.0.jar

/opt/orangefs/lib

OR


/opt/orangefs/lib

8. Determine the URL of the OrangeFS server you will name in the OrangeFS tab file.



9. Create a file named pvfs2tab in the Hadoop client build system /etc directory. This file lists the

first OrangeFS server that an OrangeFS client should contact and will eventually be copied to the remaining client nodes.


99

For example:

echo "tcp://server1:3334/orangefs /mnt/orangefs pvfs2 defaults,noauto 0 0" >

/etc/pvfs2tab

Note In the above example, tcp://server1:3334 is the URL of the server to be mounted,

which was determined in step 5.

10. Assign read-access to the new file:


Important If you already have an OrangeFS tab file present on your compute nodes, skip

steps 8-10. These steps for creating a new tab file will overwrite an existing tab file.

Configure Hadoop to Use OrangeFS

On the Hadoop client build system, modify the files contained in the Hadoop default configuration directory to use OrangeFS.

By default, Hadoop 1 looks for its configuration files in its conf directory and Hadoop 2 uses its

etc/hadoop directory, although you can create a separate custom configuration directory. When

running Hadoop using a custom configuration directory, specify the directory as follows:

/opt/hadoop-version/bin/hadoop --config configuration_path

where...

version = selected Hadoop version

Example: hadoop-2.4.1

configuration_path = custom configuration directory

Note We recommend that you keep your Hadoop configuration under version control, using open

source tools such as Subversion or Git.

Instructions for Hadoop 1

Using your preferred text editor, modify the following files in the /opt/hadoop-version/conf to

include the indicated properties.

1. Set the following environment variables in the hadoop-env.sh file:

# The java implementation to use.

export JAVA_HOME=path_to_jdk

where...

path_to_jdk = path of the installed JDK


# Path to OrangeFS libs

export LD_LIBRARY_PATH=/opt/orangefs/lib

export JNI_LIBRARY_PATH=/opt/orangefs/lib

# Extra Java CLASSPATH elements.

export HADOOP_CLASSPATH="$JNI_LIBRARY_PATH/orangefs-jni-

2.9.0.jar:$JNI_LIBRARY_PATH/orangefs-hadoop1-2.9.0.jar"


100

2. Set the following properties in the core-site.xml file:

<property>

<name>fs.default.name</name>

<value>ofs://localhost:3334</value>

</property>

Notes Currently, the OrangeFS tab file provides the OrangeFS client with its server connection information. Thus, the hostname and port number are irrelevant when setting this

property, so localhost is used. Support for multiple OrangeFS filesystems is possible

but complex and is thus outside the scope of this documentation.

The value must begin with ofs:// to differentiate OrangeFS from other file system

prefixes.

The following additional properties may be of some interest:

<property>

<name>fs.ofs.file.buffer.size</name>

<value>4194304</value>

<description>4MB OrangeFS I/O Buffer. The default for OrangeFS is

4MB.</description>

</property>

<property>

<name>fs.ofs.block.size</name>


<description>128MB Block size for OrangeFS files. The default for

OrangeFS is 64 MB.</description>

</property>

3. Set the following properties in the mapred-site.xml file:

<property>

<name>mapred.job.tracker</name>

<value>node001:8021</value>

</property>

Notes The value of this property must be the hostname and port where you will run your JobTracker. In this documentation, the Hadoop build system is set to run as the JobTracker.

Though the JobTracker daemon can be run on any of the other nodes, we recommend dedicating a JobTracker node which will not run an OrangeFS server. The node assigned to run the JobTracker should also be excluded from the slaves configuration file. 8021 is a good choice for the JobTracker port number.

You must disable speculative execution when using Hadoop with OrangeFS:


101

<property>

<name>mapred.map.tasks.speculative.execution</name>

<value>false</value>

<final>true</final>

</property>

<property>

<name>mapred.reduce.tasks.speculative.execution</name>


<final>true</final>

</property>

4. Add hosts to the slaves file.

The slaves file contains a list of hostnames that run mapreduce. Assuming node001 is the

JobTracker, the slaves list would include the remaining clients (running TaskTrackers):

node002

node003

…

node998

node999


Using your preferred text editor, modify the following files in the /opt/hadoop-

version/etc/hadoop directory to include the indicated properties.




where...







if [ "$HADOOP_CLASSPATH" ]; then

export HADOOP_CLASSPATH="$HADOOP_CLASSPATH:$JNI_LIBRARY_PATH/orangefs-

hadoop2-2.9.0.jar:$JNI_LIBRARY_PATH/orangefs-jni-2.9.0.jar"

else

export HADOOP_CLASSPATH="$JNI_LIBRARY_PATH/orangefs-hadoop2-

2.9.0.jar:$JNI_LIBRARY_PATH/orangefs-jni-2.9.0.jar"

fi


<property>



</property>


102

Notes Currently, the OrangeFS tab file provides the OrangeFS client with its server connection

information. Thus, the hostname and port number are irrelevant when setting this

property, so localhost is used. Support for multiple OrangeFS filesystems is possible



prefixes.

The following properties direct Hadoop to use the OrangeFS file system implementation:

<property>

<name>fs.ofs.impl</name>

<value>org.orangefs.hadoop.fs.ofs.OrangeFileSystem</value>

<description>An extension of filesystem for OrangeFS URIs.</description>

</property>

<property>

<name>fs.defaultFS</name>


</property>

<property>

<name>fs.AbstractFileSystem.ofs.impl</name>

<value>org.apache.hadoop.fs.ofs.OrangeFs</value>

<description> The file system for OrangeFS (ofs:) uris.</description>

</property>

The following property represents a comma separated list of OrangeFS installations. This is truly the “Authority” portion of a URI which is used by Hadoop to uniquely identify a path on a

particular system using a particular file system implementation:

<property>

<name>fs.ofs.systems</name>

<value>localhost:3334</value>

</property>

The following property represents the mount location of an OrangeFS client. This value may be a comma separated list indicating multiple OrangeFS mount locations. The same locations must also be valid in your pvfs2tab file. There is a 1-to-1 correspondence between the fs.ofs.systems and

fs.ofs.mntLocations lists. This is how support for more than a single OrangeFS system may be achieved:

<property>

<name>fs.ofs.mntLocations</name>

<value>/mnt/orangefs</value>

<description>Location of OrangeFS mount point.</description>

</property>

Notes When using OrangeFS utilities, the local path /mnt/orangefs/ is effectively the root

directory of OrangeFS. When using Hadoop with OrangeFS, the prefix must be removed with the root directory

declared as ofs://localhost:3334/.



103

<property>




4MB.</description>

</property>

<property>



<description>128MB Block size for OrangeFS files. The default for OrangeFS is

64 MB.</description>

</property>

3. You must disable speculative execution in the mapred-site.xml file when using Hadoop with

OrangeFS:

<property>

<name>mapreduce.map.speculative</name>


<final>true</final>

</property>

<property>

<name>mapreduce.reduce.speculative</name>


<final>true</final>

</property>

4. Customize the following property in yarn-site.xml:

<property>

<description>CLASSPATH for YARN applications. A comma-separated list of

CLASSPATH entries</description>

<name>yarn.application.classpath</name>

<value>

$JNI_LIBRARY_PATH/*,

$HADOOP_CONF_DIR,

$HADOOP_COMMON_HOME/share/hadoop/common/*,

$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,

$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,

$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,

$HADOOP_YARN_HOME/share/hadoop/yarn/*,

$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*

</value>

</property>



ResourceManager, the slaves list would include the remaining clients (running NodeManagers):

node002

node003

…

node998

node999

Copy Hadoop Client Build System Software to Other Clients

Copy the following software from the Hadoop client build system to the remaining client systems:


104

/opt/hadoop-version

/opt/orangefs

/etc/pvfs2tab

Note Hadoop and OrangeFS directories can alternatively be placed in a shared NFS location.

Assuming the Hadoop client build system hostname is node001, with the rest of the client nodes

being node002-node999, use this approach to distribute the binaries via scp.

On the Hadoop client build system, run the following commands, adjusting the placeholders as necessary:

for hostnum in `seq -w 002 999`;

do

scp -r /opt/orangefs node${hostnum}:/opt/

scp -r /opt/hadoop-version node${hostnum}:/opt/

scp /etc/pvfs2tab node${hostnum}:/etc/pvfs2tab

done

where...



node = hostname prefix used throughout your cluster

Start Hadoop on Each Client

Hadoop 1

1. To start the necessary Hadoop daemons (JobTracker and TaskTrackers), run start-mapred.sh

on the node configured to be the JobTracker (also the Hadoop client build system here) with the following command:

/opt/hadoop-version/bin/start-mapred.sh

Notes start-mapred.sh starts only the JobTracker and TaskTrackers (needed for

MapReduce). You do not need to run the HDFS NameNode and DataNodes because OrangeFS Servers handle file metadata and data.

2. On the node running the JobTracker, check for any errors in the Hadoop logs in the

/opt/hadoop-version/logs directory.

3. If your compute nodes are accessible only through a gateway, use SSH tunneling to enable viewing of the JobTracker’s web UI. From your workstation, run:

ssh -L 50030:jobtracker:50030 gateway

where...

jobtracker = hostname of client designated as Hadoop JobTracker

gateway = gateway address

Example: ssh -L 50030:node001:50030 mygateway.mycluster.mydomain

Once authenticated, you can access the web UI from your workstation until the SSH connection terminates due to user exit or timeout.


105

4. To use the JobTracker’s web UI, open the following in a web browser:

http://localhost:50030

5. To stop the JobTracker and TaskTrackers, run stop-mapred.sh:

/opt/hadoop-version/bin/stop-mapred.sh

Hadoop 2

1. To start the necessary Hadoop ResourceManager daemon, from the node configured to be the

ResourceManager in yarn-site.xml (also the Hadoop client build system here), run the

following command:

/opt/hadoop-version/sbin/yarn-daemon.sh start resourcemanager

2. On every slave node issue the following commands to start the necessary daemons:

/opt/hadoop-version/sbin/yarn-daemon.sh start nodemanager

/opt/hadoop-version/sbin/yarn-daemon.sh start proxyserver

/opt/hadoop-version/sbin/mr-jobhistory-daemon.sh start historyserver

Note You do not need to run the HDFS NameNode and DataNodes because OrangeFS Servers handle file metadata and data.

3. If your compute nodes are accessible only through a gateway, use SSH tunneling to enable viewing of the ResourceManager’s web UI. From your workstation, run:

ssh -L 8088:resourcemanager:8088 gateway

where...

resourcemanager = hostname of client designated as Hadoop ResourceManager




4. To use the ResourceManager's web UI, open the following in a web browser:


5. To stop the Hadoop daemons run the following commands:

On the node running the ResourceManager daemon, run:

/opt/hadoop-version/sbin/yarn-daemon.sh stop resourcemanager

On the slave nodes, run:

/opt/hadoop-version/sbin/yarn-daemon.sh stop nodemanager

/opt/hadoop-version/sbin/yarn-daemon.sh stop proxyserver

/opt/hadoop-version/sbin/mr-jobhistory-daemon.sh stop historyserver


106

Verify Installation

Hadoop 1

After starting your Hadoop cluster via start-mapred.sh, run the TestDFSIO example program:

1. Write n x 1000 MB files to OrangeFS.

/opt/hadoop-version/bin/hadoop jar /opt/hadoop-version/hadoop-test-version.jar

TestDFSIO -write -nrFiles n -fileSize mb

where...

n = number of files generated by TestDFSIO. We recommend one file per Map Task, so n will

vary according to the size of your cluster.

mb = number of megabytes written per file

2. List files stored in OrangeFS (through the Hadoop FileSystem interface):

/opt/version/bin/hadoop fs -lsr ofs://localhost:3334/

3. Read n x 1000 MB files stored on OrangeFS:


TestDFSIO -read -nrFiles n -fileSize mb

where...




Notes If you get unexpected TestDFSIO performance numbers, examine your Hadoop

configuration for missing parameters and ensure that all nodes are running properly.

IO rates reported by TestDFSIO reflect the per map task rate. For more information,

see Benchmarking and Stress Testing an Hadoop Cluster With TeraSort,

TestDFSIO & Co.

4. Remove TestDFSIO generated data.

/opt/hadoop-version/bin/hadoop jar /opt/hadoop-version/hadoop-version-test.jar

-clean

Note This command removes the OrangeFS directory /benchmarks/TestDFSIO.

Check Logs Again

To verify that your TestDFSIO tests ran correctly, check your log files again.

Hadoop 2

After starting your Hadoop cluster, run the TestDFSIO example program:


/opt/hadoop-version/bin/hadoop org.apache.hadoop.fs.TestDFSIO -write -nrFiles n

-fileSize mb

http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench#testdfsio



107

where...




2. List files stored in OrangeFS (through the Hadoop FileSystem interface).



/opt/hadoop-version/bin/hadoop org.apache.hadoop.fs.TestDFSIO -read -nrFiles n -

fileSize mb

where...








TestDFSIO & Co.


/opt/hadoop-version/bin/hadoop org.apache.hadoop.fs.TestDFSIO -clean


Check Logs Again

To verify that your TestDFSIO tests ran correctly, check your log files again. You may also want to

use the web UI to examine more information pertaining to the TestDFSIO tests you just ran.

Support for Scheduled Environments

Hadoop MapReduce clusters and jobs can be deployed on-demand in a PBS Professional scheduled cluster using the open source project myHadoop.

myHadoop-orangefs is a fork of this project, developed to enable on-demand Hadoop clusters

running on compute nodes which access a dedicated OrangeFS storage system as the underlying file

system used by MapReduce. For more information, see myHadoop-orangefs.

Note Root permission is not required when using myHadoop-orangefs.



https://github.com/nevdullcode/myHadoop-orangefs


108

Hadoop Traditional Setup

This topic explains how to install OrangeFS and Apache Hadoop 1.2.1 or 2.4.1 across a cluster of nodes, replacing HDFS with OrangeFS configured as the distributed file system, layered across all nodes running MapReduce. In this configuration MapReduce programs and the OrangeFS server program run together on the same machine, rather than dedicating a disjoint set of storage nodes like

the HPC configuration.

Important Prior to performing these steps, you must install a supported distribution of Linux on all client systems that will run Hadoop, and you must select one of these nodes to serve as your Hadoop build system.

If the OrangeFS server setup is unfamiliar, see the beginning (page 1) of the Installation Guide and

review the instructions before proceeding. You also must install the required dependencies of OrangeFS on all nodes before proceeding.

Deploying an Apache Hadoop cluster, configured to use OrangeFS as the distributed file system,

involves six main steps, with one additional recommended step for deployment verification.

Install System Software (page 108)

Configure Hadoop to Use OrangeFS (page 111)

Configure OrangeFS (page 117)

Copy Build System Software to Remaining Nodes (page 117)

Start OrangeFS (page 118)

Start Hadoop (page 119)

Verify Installation (page 120) (Optional but recommended)

Notes Most of the following steps require that you have root permissions.

All nodes MUST be using the same version of the OrangeFS file system. For more information about choosing the right setup option and a supported Linux

distribution, see Planning for Installation (page 93).

Install System Software


You must configure all nodes to support secure shell connections via SSH without passing a



Select Build System

After you have identified the nodes that will run Hadoop with OrangeFS, select a node to configure as your Build System.

Example: node001 from { node001 - node999 }

Install the Java Development Kit (All Client Systems)

Install the Java Development Kit (JDK) on each node, including the build system.

For guidance on the appropriate version to install, consult the following recommendations listed on

the Apache Hadoop Wiki.

For details on installation, consult Oracle’s Installation Guide.

Note Only the Java Runtime Environment is needed to run MapReduce, but the JDK is required

on the Hadoop build system to build the OrangeFS Hadoop Client. For easier installation, we recommend simply installing the JDK on all client nodes.



http://www.oracle.com/technetwork/java/javase/downloads/index.html


109

Maven (Hadoop Build System)

To install Maven on the Hadoop Build System, follow these steps:

1. Download the stable Maven binary tar.gz file from the Apache Maven Project.

2. Copy the downloaded Hadoop source tar file to the /tmp directory on your build system using

scp or a similar program.

3. Extract the contents of the archive to /opt directory on the Hadoop build system.

tar -C /opt -xzf /tmp/apache-maven-version-bin.tar.gz

where...

version = version number of the Maven release

Example: apache-maven-3.2.3-bin.tar.gz

4. Set the following environment variables:

export M2_HOME=/opt/apache-maven-version

export M2=$M2_HOME/bin

export JAVA_HOME=jdk_path

export PATH=$JAVA_HOME/bin:$M2:$PATH

Note You may set the above environment variables in a terminal but they will not persist

once the terminal has been closed or the system rebooted. To make these environment

variables persist add them to /etc/profile or /etc/profile.d/maven.sh and

reboot your system.

5. Verify your installation of Maven:

mvn --version

Important It is critical to set JAVA_HOME correctly.

Hadoop Binaries (All Systems)

1. Download a supported Apache Hadoop release from http://hadoop.apache.org/releases.html and copy it to the /tmp directory on the Hadoop build system. The filename should resemble:

hadoop-version-bin.tar.gz for Hadoop 1 and hadoop-version.tar.gz for Hadoop 2.

Notes The most recent stable version of Hadoop binaries tested with OrangeFS is hadoop-

2.4.1.

On each system, you must use the same version of Apache Hadoop binaries as the version of the Apache Hadoop source used on the build system..

2. Extract the Hadoop binaries archive to /opt by running the following command on the Hadoop

build system:

tar -C /opt -xzf /tmp/hadoop-version.tar.gz

where...


Example: hadoop-2.4.1.tar.gz

http://maven.apache.org/download.cgi

http://hadoop.apache.org/releases.html


110

Note You will customize the Hadoop configuration files present in /opt/hadoop-

version/conf for Hadoop 1 or in /opt/hadoop-version/etc/hadoop for Hadoop

2 prior to distributing the Hadoop binaries directory containing your configuration to all nodes.

OrangeFS (Hadoop Client Build System)

Download the OrangeFS tar file and copy it to the Hadoop build system. Then, extract and build

OrangeFS, ultimately installing in /opt on the Hadoop build system.


1. Go to http://www.orangefs.org/download/ and download the tar file orangefs-

version.tar.gz, where version = version number of the OrangeFS release.

2. Copy the OrangeFS tar file to the /tmp directory on the Hadoop build system.

3. On the Hadoop build system, extract the contents of the archive into the /tmp/src directory:

tar -C /tmp/src -xzf /tmp/orangefs-version.tar.gz

4. On the Hadoop build system, generate a Makefile that enables the OrangeFS Hadoop Client and OrangeFS JNI Shim for OrangeFS, using the following commands:

cd /tmp/src/orangefs-version

./prepare

./configure \

--prefix=/opt/orangefs \

--enable-shared \

--disable-server \

--enable-jni \

--with-jdk=jdk_path \

where...

jdk_path = path to jdk




5. Continue with the standard Linux commands to build and install OrangeFS to /opt:

make

make install

6. Change directory to the appropriate Hadoop client directory. Pick orangefs-hadoop1 if you are

building a client for Apache Hadoop 1.x.x or orangefs-hadoop2 if you are building a client for

Apache Hadoop 2.x.x:


OR


7. A pom.xml file is located in each the above directories and is used by Maven to build the

OrangeFS Hadoop Client jar using a specified version of Hadoop. For orangefs-hadoop1 the



111

default is Apache Hadoop 1.2.1. For orangefs-hadoop2 the default is Apache Hadoop 2.4.1. The

specified version of Hadoop is indicated by a property present in the pom.xml file. These

pom.xml files may be customized to your liking, but do so at your own risk. Build the OrangeFS

Hadoop Client jar:


/opt/orangefs/lib

OR


/opt/orangefs/lib

8. Determine the URL of the node you will name in the OrangeFS tab file.


Example: tcp://node002:3334


/etc/pvfs2tab

9. Create a file named pvfs2tab in the Hadoop build system /etc directory. This file lists the first

OrangeFS server that an OrangeFS client should contact and will eventually be copied to the remaining client nodes, since all nodes in your configuration (including the JobTracker/ResourceManager) act as clients with respect to OrangeFS.

Note We use node002 instead of node001 as the server listed in the pvfs2tab file, since

node001 is dedicated as a JobTracker and is not acting as an OrangeFS server. Select a node that is not the JobTracker.

For example:


/etc/pvfs2tab

Note In the above example, tcp://node002:3334 is the URL of the server to be mounted,

which was determined in step 5.

10. Assign read-access to the new file:


Important If you already have an OrangeFS tab file present on your compute nodes, skip

steps 8-10. These steps for creating a new tab file will overwrite an existing tab file.

Configure Hadoop to Use OrangeFS

On the Hadoop build system, modify the files in the Hadoop default configuration directory to use

OrangeFS. By default, Hadoop 1.x.x looks for its configuration files in its conf directory and Hadoop 2

uses its etc/hadoop directory, although you can create a separate custom configuration directory for

OrangeFS.

When running Hadoop using a custom configuration directory, you must specify the directory as follows:


112

/opt/hadoop-version/bin/hadoop --config configuration_path

where...

version = selected Hadoop version


configuration_path = configuration directory

Note We recommend that you keep your Hadoop configuration under version control, using open

source tools like Subversion or Git.


Using your preferred text editor, modify the following files in the /opt/hadoop-version/conf to

include the indicated properties.




where...







export HADOOP_CLASSPATH="$JNI_LIBRARY_PATH/orangefs-jni-

2.9.0.jar:$JNI_LIBRARY_PATH/orangefs-hadoop1-2.9.0.jar"


<property>



</property>


property, so localhost is used. Support for multiple OrangeFS file systems is possible



prefixes.

The following property directs Hadoop to use the OrangeFS file system implementation:

<property>




</property>


113

The following property represents a comma separated list of OrangeFS installations. This is truly

the “Authority” portion of a URI which is used by Hadoop to uniquely identify a path on a particular system using a particular file system implementation:

<property>



</property>

The following property represents the mount location of an OrangeFS client. This value may be a

comma separated list indicating multiple OrangeFS mount locations. The same locations must also

be valid in your pvfs2tab file. There is a 1-to-1 correspondence between the fs.ofs.systems

and fs.ofs.mntLocations lists. This is how support for more than a single OrangeFS system

may be achieved.:

<property>




</property>

Notes When using OrangeFS utilities the local path /mnt/orangefs/ is effectively the root

directory of OrangeFS. When using Hadoop with OrangeFS, the prefix must be removed, with the root

directory declared as ofs://localhost:3334/.


<property>




4MB.</description>

</property>

<property>





</property>

3. Set the following properties in the mapred-site.xml file:

<property>

<name>mapred.job.tracker</name>

<value>node001:8021</value>

</property>


114

Notes The value of this property must be the hostname and port where you will run your

JobTracker. In this documentation, the Hadoop build system is set to run as the JobTracker.

Though the JobTracker daemon can be run on any of the other nodes, we recommend dedicating a JobTracker node which will not run an OrangeFS server. The node assigned to run the JobTracker should also be excluded from the slaves configuration file. 8021 is a good choice for the JobTracker port number.

You must disable speculative execution when using Hadoop with OrangeFS:

<property>

<name>mapred.map.tasks.speculative.execution</name>


<final>true</final>

</property>

<property>

<name>mapred.reduce.tasks.speculative.execution</name>


<final>true</final>

</property>



ResourceManager, the slaves list would include the remaining clients (running NodeManagers):

node002

node003

…

node998

node999


Using your preferred text editor, modify the following files in the /opt/hadoop-

version/etc/hadoop to include the indicated properties.




where...




115





if [ "$HADOOP_CLASSPATH" ]; then

export HADOOP_CLASSPATH="$HADOOP_CLASSPATH:$JNI_LIBRARY_PATH/orangefs-

hadoop2-2.9.0.jar:$JNI_LIBRARY_PATH/orangefs-jni-2.9.0.jar"

else

export HADOOP_CLASSPATH="$JNI_LIBRARY_PATH/orangefs-hadoop2-

2.9.0.jar:$JNI_LIBRARY_PATH/orangefs-jni-2.9.0.jar"

fi

2. Set the following environment variables in the core-site.xml file:

<property>



</property>


property, so localhost is used. Support for multiple OrangeFS file systems is possible



prefixes.

The following properties direct Hadoop to use the OrangeFS file system implementation:

<property>




</property>

<property>

<name>fs.defaultFS</name>


</property>

<property>

<name>fs.AbstractFileSystem.ofs.impl</name>

<value>org.apache.hadoop.fs.ofs.OrangeFs</value>

<description> The file system for OrangeFS (ofs:) uris.</description>

</property>

The following property represents a comma separated list of OrangeFS installations. This is truly the “Authority” portion of a URI which is used by Hadoop to uniquely identify a path on a particular system using a particular file system implementation:

<property>



</property>

The following property represents the mount location of an OrangeFS client. This value may be a

comma separated list indicating multiple OrangeFS mount locations. The same locations must also

be valid in your pvfs2tab file. There is a 1-to-1 correspondence between the fs.ofs.systems


116

and fs.ofs.mntLocations lists. This is how support for more than a single OrangeFS system

may be achieved.:

<property>




</property>

Notes When using OrangeFS utilities the local path /mnt/orangefs/ is effectively the root

directory of OrangeFS. When using Hadoop with OrangeFS, the prefix must be removed, with the root

directory declared as ofs://localhost:3334/.


<property>




4MB.</description>

</property>

<property>





</property>

3. You must disable speculative execution in the mapred-site.xml file when using Hadoop with

OrangeFS:

<property>

<name>mapreduce.map.speculative</name>


<final>true</final>

</property>

<property>

<name>mapreduce.reduce.speculative</name>


<final>true</final>

</property>

4. Customize the following property in yarn-site.xml:


117

<property>

<description>CLASSPATH for YARN applications. A comma-separated list of

CLASSPATH entries</description>

<name>yarn.application.classpath</name>

<value>

$JNI_LIBRARY_PATH/*,

$HADOOP_CONF_DIR,

$HADOOP_COMMON_HOME/share/hadoop/common/*,

$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,

$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,

$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,

$HADOOP_YARN_HOME/share/hadoop/yarn/*,

$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*

</value>

</property>



JobTracker, the slaves list would include the remaining clients (running TaskTrackers):

node002

node003

…

node998

node999

Configure OrangeFS

By this point you’ve identified the node which will run the JobTracker (or ResourceManager if running Hadoop 2) and the remaining nodes. You will run a Hadoop TaskTracker (or NodeManager et al if

running Hadoop 2) and OrangeFS server on each of these. See Create OrangeFS Configuration File

(page 29) for instructions to generate an OrangeFS configuration file using an OrangeFS utility.

/opt/orangefs/bin/pvfs2-genconfig server_configuration_file_path

where...

server_configuration_file_path = the path of the configuration file generated by pvfs2-

genconfig

Example: /etc/orangefs-server.conf

Copy Hadoop Build System Software to Remaining Nodes

You must copy the following software from the Hadoop build system to the remaining nodes:

/opt/hadoop-version

/opt/orangefs

/etc/pvfs2tab

/etc/orangefs-server.conf

Note The files and directories listed above can also be placed in a shared NFS location.

If the pvfs2tab file resides at a path other than /etc/pvfs2tab, add the following command to the

hadoop-env.sh script you customized:

export PVFS2TAB_FILE=path


118

where...

path = the path to the tab file

Example: /home/$USER/pvfs2tabfile

Assuming the Hadoop build system hostname is node001, with the rest of the client nodes being

node002-node999, use this approach to distribute the binaries via scp.

On the Hadoop build system, run the following commands, adjusting the placeholders as necessary:

for hostnum in `seq -w 002 999`;

do

scp -r /opt/orangefs node${hostnum}:/opt/

scp -r /opt/hadoop-version node${hostnum}:/opt/

scp /etc/pvfs2tab node${hostnum}:/etc/pvfs2tab

scp /etc/orangefs-server.conf node${hostnum}:/etc/orangefs-server.conf

done

where...



node = hostname prefix used throughout your cluster

Start OrangeFS

Initialize OrangeFS

/opt/orangefs/sbin/pvfs2-start-all -c config_file_path -p prefix_path -s f

where...

config_file_path = path to the OrangeFS server configuration file on all OrangeFS servers

prefix_path = path of OrangeFS prefix (installation directory)

Example: /opt/orangefs

Note The above command contains ‘-s f’ which initializes local storage, then exits, on all

servers listed in the specified OrangeFS server configuration file. This command must be run only the first time you start your OrangeFS servers.

Starting all OrangeFS servers

/opt/orangefs/sbin/pvfs2-start-all -c config_file_path -p prefix_path -m

orangefs_mnt_point

where...

orangefs_mnt_point = the local mount point prefix of the OrangeFS root directory specified in

the pvfs2tab file

Example: /mnt/orangefs

Note The above command contains the optional ‘-m orangefs_mnt_point’ option and

argument. This tells the pvfs2-start-all script to ping the OrangeFS servers (file

system) corresponding to the specified mount directory following startup to verify the status of the file system.


119

Stopping all OrangeFS Servers:

To force all OrangeFS servers to exit (go offline), run:

/opt/orangefs/sbin/pvfs2-stop-all -c config_file_path

where...

config_file_path = path to the OrangeFS server configuration file on all OrangeFS servers

Example: /etc/orangefs-server.conf

Start Hadoop

Hadoop 1

1. To start the necessary Hadoop daemons (JobTracker and TaskTrackers), run start-mapred.sh

on the node configured to be the JobTracker (the Hadoop client build system here) with the following command:

/opt/hadoop-version/bin/start-mapred.sh

Notes start-mapred.sh starts the JobTracker and TaskTrackers (needed for MapReduce)

only. You do not have to run the HDFS NameNode and DataNodes since OrangeFS Servers handle file metadata and data.

2. On the node running the JobTracker, check for any errors in the Hadoop logs in the

/opt/hadoop-version/logs directory.

3. If your compute nodes are accessible only through a gateway, use SSH tunneling to enable viewing of the JobTracker’s web UI. From your workstation, run:

ssh -L 50030:jobtracker:50030 gateway

where...

jobtracker = hostname of client designated as Hadoop JobTracker




4. To use the JobTracker’s web UI, open the following in a web browser:


5. To stop the JobTracker and TaskTrackers, run stop-mapred.sh:

/opt/hadoop-version/bin/stop-mapred.sh

Hadoop 2

1. To start the necessary Hadoop ResourceManager daemon, from the node configured to be the

ResourceManager in yarn-site.xml (also the Hadoop client build system here), run the

following command:


120

/opt/hadoop-version/sbin/yarn-daemon.sh start resourcemanager

2. On every slave node issue the following commands to start the necessary daemons:

/opt/hadoop-version/sbin/yarn-daemon.sh start nodemanager

/opt/hadoop-version/sbin/yarn-daemon.sh start proxyserver

/opt/hadoop-version/sbin/mr-jobhistory-daemon.sh start historyserver

Note You do not need to run the HDFS NameNode and DataNodes because OrangeFS

Servers handle file metadata and data.

3. If your compute nodes are accessible only through a gateway, use SSH tunneling to enable viewing of the ResourceManager’s web UI. From your workstation, run:

ssh -L 8088:resourcemanager:8088 gateway

where...

resourcemanager = hostname of client designated as Hadoop JobTracker




4. To use the ResourceManager's web UI, open the following in a web browser:


5. To stop the Hadoop daemons run the following commands:

On the node running the ResourceManager daemon, run:

/opt/hadoop-version/sbin/yarn-daemon.sh stop resourcemanager

On the slave nodes, run:

/opt/hadoop-version/sbin/yarn-daemon.sh stop nodemanager

/opt/hadoop-version/sbin/yarn-daemon.sh stop proxyserver

/opt/hadoop-version/sbin/mr-jobhistory-daemon.sh stop historyserver

Verify Installation

Hadoop 1

After starting your Hadoop cluster via start-mapred.sh, run the TestDFSIO example program:



TestDFSIO -write -nrFiles n -fileSize mb

where...





121





TestDFSIO -read -nrFiles n -fileSize mb

where...








TestDFSIO & Co.


/opt/hadoop-version/bin/hadoop jar /opt/hadoop-version/hadoop-version-test.jar

-clean


Check Logs Again

To verify that your TestDFSIO tests ran correctly, check your log files again.

Hadoop 2

After starting your Hadoop cluster, run the TestDFSIO example program:


/opt/hadoop-version/bin/hadoop org.apache.hadoop.fs.TestDFSIO -write -nrFiles n

-fileSize mb

where...







/opt/hadoop-version/bin/hadoop org.apache.hadoop.fs.TestDFSIO -read -nrFiles n -

fileSize mb




122

where...








TestDFSIO & Co.


/opt/hadoop-version/bin/hadoop org.apache.hadoop.fs.TestDFSIO -clean


Check Logs Again

To verify that your TestDFSIO tests ran correctly, check your log files again. You may also want to

use the web UI to examine more information pertaining to the TestDFSIO tests you just ran.



OrangeFS Installation Instructions (2.9) Other Installation Topics

123

Other Installation Topics

This section provides supplemental information for consideration both during and after the standard OrangeFS installation. Topics include:

Basic Install Example / Quick Start (page 123)

Berkeley DB Version Support (page 126)

Directory/File Listing (page 127)

OrangeFS Configuration File (page 15)

Storage Directory Location (page 129)

Automating System Startup (page 130)

Working With Firewalls (page 130)

Note For troubleshooting, FAQ, and more advanced information, see the Administration Guide.

Basic Installation Example / Quick Start

This topic provides an example of a complete installation of OrangeFS in a single procedure. It can also be used as a Quick Start reference for experienced users who wish to bypass the more detailed and segmented instructions in the earlier topics of this manual.

Note Most of the following steps require that you have root permissions.

Assumptions

The following assumptions apply to this example installation:

Distribution of Linux on all systems is Red Hat Enterprise Linux (RHEL)

Network protocol is tcp/ip

Includes only the Linux (kernel) interface; other clients can be installed later

Uses Default security mode

Build OrangeFS

The system on which you build OrangeFS requires eight additional Linux software packages. Following are the names for these packages on a system running RHEL:

gcc

flex

bison

openssl-devel

db4-devel

kernel-devel

perl

make

To automatically install these packages, enter the following command:


To build OrangeFS, complete the following steps:

1. Download and extract the OrangeFS software:

Download the source from http://www.orangefs.org/download/.

Extract the source tar archive:

tar -xzf orangefs-version.tar.gz

Change Directory (cd) to the extracted directory:



124

cd orangefs-version

Configure the OrangeFS installation location and the path of the system kernel:

./configure --prefix=/opt/orangefs --with-kernel=kernel_path

where...

kernel_source_path = path to kernel source

Example: /usr/src/kernels/2.6.18-194.17.1.el5-x86_64/

2. Build and install the software:

make

make install

make kmod

make kmod_prefix=/opt/orangefs kmod_install

3. Change Directory (cd) to the new installation directory and create the following working

directories:

log storage storage/data storage/meta

Note Standard installation places file system storage directories inside the OrangeFS

installation directory under opt for portability. These directories can be located

elsewhere for purposes of system optimization and larger space allocations. For more

information see Storage Directory Location.

cd /opt/orangefs

mkdir -p log storage/data storage/meta

4. Create a server configuration file by running the automatic file generation program (pvfs2-

genconfig) and answering the prompts.

/opt/orangefs/bin/pvfs2-genconfig /opt/orangefs/etc/orangefs-server.conf

Notes During the pvfs2-genconfig process:

● Use the directories you created in Step 3 for your storage and log file locations.

● Each host you specify must be the value returned by the hostname command.

This places a server configuration file (named orangefs-server.conf in this example) in the

etc directory.

Add Servers

1. To add the required software to an OrangeFS server, Change Directory (cd) to /opt on the Server

system and copy the /opt/orangefs directory from the Build system:

scp -r /opt/orangefs hostname:/opt/

where...



125

2. Initialize the server storage space on each server:

/opt/orangefs/sbin/pvfs2-server -f /opt/orangefs/etc/orangefs-server.conf

3. Start the server processes on each server:

/opt/orangefs/sbin/pvfs2-server /opt/orangefs/etc/orangefs-server.conf

Add Clients (Kernel Module)

1. To add the software required for an OrangeFS Linux client interface, Change Directory (cd) to

/opt on the Client system and copy the /opt/orangefs directory from the Build system:

scp -r /opt/orangefs hostname:/opt/

where...


2. Insert the client kernel module.

This module (pvfs2.ko) resides in the OrangeFS installation directory several directory layers

deep. To insert the module without specifying a long path, include this find statement:

insmod ‘find /opt/orangefs -name pvfs2.ko‘

3. Start the client process on each Client system:

/opt/orangefs/sbin/pvfs2-client

4. Create a directory in the Client system's /mnt directory through which the client will mount

OrangeFS:

mkdir /mnt/orangefs

5. Determine the URL of the OrangeFS server you will mount.

You can retrieve this information from the orangefs-server.conf file. For example, the first

server URL listed in that file can be extracted with the following command:

grep "Alias " /opt/orangefs/etc/orangefs-server.conf | awk '{ print $3 }' | head

-n 1



6. Create a file named pvfs2tab in the Client system's /etc directory that tells the system how to

mount OrangeFS. Assign read access to the file.

echo "tcp://server1:3334/orangefs /mnt/orangefs pvfs2 defaults,noauto 0 0" >>

/etc/pvfs2tab

7. Mount OrangeFS through the server URL you retrieved earlier:

mount -t pvfs2 tcp://server1:3334/orangefs /mnt/orangefs


126

Berkeley DB Version Support

Important If you are installing and using OrangeFS for anything other than evaluation

purposes, all servers in the installation must use Berkeley DB version 4.8.30 or

later.

Background

In all distributions of Linux supported by OrangeFS, Berkeley DB (BDB) is included as a standard package. However, not all of those distributions use version 4.8.30 or later.

Running a version of BDB older than 4.8.30 on an OrangeFS server has been associated with the following:

Files that are missing attribute data

Files that cannot be removed

Therefore, any production server running OrangeFS must use BDB version 4.8.30 or later.

Note The build system in an OrangeFS installation also requires the BDB Development Libraries, which is a separate package not included with the Linux distributions. For more information,

see Preview System Requirements (page 5).

Automatic Notice

Beginning with version 2.8.5 of OrangeFS, a notice during execution of the configure command during installation will remind you about the BDB version requirement. The notice can be ignored, but that

can lead to the issues described above.

Determining Your Version

Determining the installed version of BDB will vary from one distribution of Linux to another. For

example, if you are using RHEL, the following package manager command should provide the version:

yum info db4

If you have db4 utilities installed (there will be db_stat executable in your path), you can also

determine the version of BDB with the following:

db_stat -V

Updating BDB

Since requirements are different for all distributions, the easiest solution is to install a copy of BDB 4.8.30 (or later) in a non-standard location.

OrangeFS Requirements

OrangeFS requires the following in regard to BDB software components:

Build system requires Berkeley DB Libraries and Berkeley DB Development Libraries

Server systems require Berkeley DB libraries (and for the library to be in LD_LIBRARY_PATH)

Linux clients do not require any Berkeley DB software

Getting BDB

The main download page for BDB is

http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html.

Previous releases of Berkeley DB are available at

http://www.oracle.com/technetwork/database/berkeleydb/downloads/index-082944.html.

http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html

http://www.oracle.com/technetwork/database/berkeleydb/downloads/index-082944.html


127

Note OrangeFS has not been thoroughly tested against Berkeley DB 11gR2 (version 5) or later.

Although compilation and basic functionality appear to work correctly, this version has not yet been validated.

Installing BDB In A Non-Standard Location

The following list of commands demonstrates the general steps; however, see Berkeley DB installation

instructions for more information and additional options.

tar -xzf db-4.8.30.tar.gz

cd db-4.8.30

cd build_unix

../dist/configure --prefix=/usr/local/db4.8.30

make

sudo make install

Note The version of BDB built here is not intended as a replacement for the version provided

with your distribution. It is a standalone version of BDB that provides the needed functionality for use with OrangeFS.

Using BDB From A Non-Standard Location With OrangeFS

To use BDB from a non-standard location (one that is not in the default search paths), include the following argument when you configure OrangeFS:

--with-db=<path>

For example, for the BDB installation explained above, the configure statement would be:

./configure --prefix=/opt/orangefs --with-db=/usr/local/db4.8.30

Running OrangeFS With BDB From A Non-Standard Location

For each server running OrangeFS, LD_LIBRARY_PATH must include the path to the non-standard

BDB libraries. One way to achieve this is to set it in the script or current environment.

For example, for the BDB installation explained above, you could enter the following:

export LD_LIBRARY_PATH=/usr/local/db4.8.30/lib:${LD_LIBRARY_PATH}

Sample Installation Directory and File Listing

Following is a sample listing of all the directories and files existing in the OrangeFS installation directory at the completion of Step 1 (Build OrangeFS) in the Installation Instructions.

Notes For more information about individual file contents, see the separate online interface for

Source Code Documentation. For more information about building OrangeFS, see Build

OrangeFS (page 18). For an overview of the installation instructions, see Installation

Concepts (page 2).

Color

Key: Directories are indicated by red text, and files are indicated by green text.


128

Top Level

/opt/orangefs:

bin include lib sbin share etc log

/bin

/opt/orangefs/bin:

getmattr pvfs2-getmattr pvfs2-set-eventmask

karma pvfs2-ln pvfs2-setmattr

pvfs2-change-fsid pvfs2-ls pvfs2-set-mode

pvfs2-check-config pvfs2-lsplus pvfs2-set-sync

pvfs2-check-server pvfs2-migrate-collection pvfs2-showcoll

pvfs2-chmod pvfs2-mkdir pvfs2-stat

pvfs2-chown pvfs2-mkspace pvfs2-statfs

pvfs2-config pvfs2-perf-mon-example pvfs2-touch

pvfs2-config-convert pvfs2-perf-mon-snmp pvfs2-validate

pvfs2-cp pvfs2-perror pvfs2-viewdist

pvfs2-drop-caches pvfs2-ping pvfs2-write

pvfs2-fsck pvfs2-remove-object pvfs2-xattr

pvfs2-fs-dump pvfs2-rm setmattr

pvfs2-genconfig pvfs2-set-debugmask

/include

/opt/orangefs/include:

pvfs2-compat.h pvfs2.h pvfs2-mirror.h pvfs2-types.h

pvfs2-debug.h pvfs2-hint.h pvfs2-request.h pvfs2-usrint.h

pvfs2-encode-stubs.h pvfs2-mgmt.h pvfs2-sysint.h pvfs2-util.h

/lib

libofs.a libpvfs2.a

liborangefs.a modules

liborangefsposix.a

/opt/orangefs/lib/modules:

2.6.32-358.11.1.el6.x86_64

/opt/orangefs/lib/modules/2.6.32-358.11.1.el6.x86_64:

kernel

/opt/orangefs/lib/modules/2.6.32-358.11.1.el6.x86_64/kernel:

fs

/opt/orangefs/lib/modules/2.6.32-358.11.1.el6.x86_64/kernel/fs:

pvfs2

/opt/orangefs/lib/modules/2.6.32-358.11.1.el6.x86_64/kernel/fs/pvfs2:

pvfs2.ko

Notes The name of the subdirectory found under modules will vary depending on the kernel version on the build system.

With the --enable-shared option, additional files will be included. The subdirectory

"modules" is included regardless.


129

/sbin

/opt/orangefs/sbin:

pvfs2-client pvfs2-client-core pvfs2-server pvfs2-start-

all pvfs2-stop-all

/share

/opt/orangefs/share:

man

/opt/orangefs/share/man:

man1 man5

/opt/orangefs/share/man/man1:

pvfs2.1.gz pvfs2-ls.1.gz pvfs2-set-mode.1.gz

pvfs2-cp.1.gz pvfs2-ping.1.gz pvfs2-set-sync.1.gz

pvfs2-fs-dump.1.gz pvfs2-server.1.gz pvfs2-statfs.1.gz

pvfs2-genconfig.1.gz pvfs2-set-debugmask.1.gz

/opt/orangefs/share/man/man5:

pvfs2.conf.5.gz pvfs2tab.5.gz

/etc

/opt/orangefs/etc:

orangefs-server.conf

Storage Directory Location

For a standard installation, once you have initialized the storage directories, the two file system

storage directories (storage/data and storage/meta) are provided inside the OrangeFS

installation directory under opt, primarily for portability. When you copy the installation directory to

your other OrangeFS servers, the storage directories are included. You do not have to recreate them.

However, these directories can be located elsewhere on each OrangeFS server for system optimization and larger space allocations. The directory locations can even differ for each server if you add a

<ServerOptions> context for each server in the OrangeFS configuration file.

After Installation

You can change the locations for storage directories by manually editing the <ServerOptions>

section in OrangeFS configuration file. See the Administration Guide FAQ for more information.

When you use this method, you can set unique directory locations for each server.

For detailed information on all options in the OrangeFS configuration file, see the Administration

Guide.


130

Automating System Startup

The installation instructions call for manually entered commands to start and run OrangeFS processes, including:

Starting the server daemon on an OrangeFS server

Inserting the kernel module on an OrangeFS Linux client

Starting the client daemon on an OrangeFS Linux client

Mounting OrangeFS on a Linux client

To avoid repeating one or more of these commands each time you reboot an OrangeFS server or client, you can place the command statements in the appropriate system file(s) for automatic execution.

Start/Stop All Servers

These two shell scripts can be used to start and stop all servers in your OrangeFS file system. They are provided as a basic starting point that can be customized for your installation.

These two scripts are located in the following directories:

/opt/orangefs/sbin/pvfs2-start-all

and

/opt/orangefs/sbin/pvfs2-stop-all

For more information on shell scripts, see the Administration Guide.

Working With Firewalls

If you are installing OrangeFS on a TCP network that uses a firewall, you must add an iptables rule to

allow access to the port you have configured for your OrangeFS servers.

Note For protocols other than TCP and IB, firewalls are not an issue.

To bypass firewalls on a TCP network, you must add an iptables rule to every OrangeFS server in your

installation. The specific steps to accomplish this will vary from one Linux distribution to another.

For example, if you are running Red Hat, you would add the following line to

/etc/sysconfig/iptables:

-A INPUT -m state --state NEW -p tcp -m tcp --dport 3334 -j ACCEPT

This example allows incoming TCP access to port 3334 (the default for OrangeFS servers) to be accepted by iptables.

Date post:	10-Feb-2017
Category:	Documents
Upload:	tranminh
View:	219 times
Download:	3 times

OrangeFS Installation Instructions (2.9)

Documents