Dell C
TeciSCS
Confidential
chnical ISI suppor
nstructiort for IBM
on M AIX
For Authhorized Use O
Only
Documen
Revision: Revision D
nt ID: N/A
A02 Date: Septemmber 2, 2009
iSCSI support for IBM AIX
Disclaimer: This information is provided only for support purposes. EqualLogic does not
develop or maintain iSCSI initiators. Check with your initiator vendor for the latest information.
To connect to PS Series iSCSI volumes from an AIX host, you must be running PS Series
firmware version 2.2.3 or later. For versions of AIX from 5.1 up to 5.2 ML2, the Cisco software
initiator is downloadable from the EqualLogic support web site. For AIX 5.2 ML7, and AIX 5.3
ML3 and higher, IBM is including an iSCSI initiator in the operating system itself.
IBM also makes an HBA card (FC 5714, 1987, 5713, 1986) which can be installed in AIX
systems, based on the QLogic QLA4050 chipset. EqualLogic has not fully tested and qualified
this adapter, and cannot make any guarantees about its specific compatibility with EqualLogic
storage arrays. Please see your IBM reseller for specific compatibility requirements, and review
the following link: IBM 1 Gigabit iSCSI TOE Adapter Installation and Configuration.
AIX 5.1 and 5.2 up to ML2 (32-bit only):
Use the Cisco iSCSI initiator 3.4.5 for IBM AIX (“AIX 5.1/5.2 iSCSI Initiator,” posted
12/08/2004) available from the EqualLogic Support web site. Use is subject to license agreement
acceptance. Be sure to review the initiator release notes for package dependency information,
and the related Knowledge Base article on the EqualLogic Customer Support site titled Using the
AIX iSCSI Initiator. Please also read the README file included in the initiator TAR file, as it
contains detailed information about installing and using the initiator, including dependencies. If
you need to report an issue, contact EqualLogic Customer Support.
AIX 5.2 ML7 (32- and 64-bit):
AIX iSCSI support is now part of the operating system distribution. This requires ML7 and
U803038. When you have installed maintenance level 7, go to Quick Links for AIX Fixes
(http://www.ibm.com/servers/eserver/support/unixservers/aixfixes.html). Under Specific Fixes,
click AIX 5.2 and enter U803038 in the search string field to obtain the correct fix.
AIX 5.3 (32- and 64-bit):
AIX iSCSI support is now part of the operating system distribution. This requires ML3 and
U802416. When you have installed maintenance level 3, go to Quick Links for AIX Fixes
(http://www.ibm.com/servers/eserver/support/unixservers/aixfixes.html). Under Specific Fixes,
click AIX 5.3 and enter U802416 in the search string field to obtain the correct fix.
IBM directly supports the native iSCSI initiator in AIX 5.3. On-line documentation for the
package can be found at the AIX Information Center
(http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.doc/aixbma
n/commadmn/tcp_initiator.htm).
Note that there is no support for AIX 5.2 ML3-ML6. If you need to use AIX 5.2 later than ML2,
please update to ML7 and use the initiator that is included with the operating system.
Configuration Considerations
First and foremost, it’s useful to determine that actual speed that your AIX ethernet intreface has
negotiated with the switch. If the interface has negotiated something other than 1000Mbps Full
Duplex, then your performance is already compromised. To determine what speed your ethernet
interfaces are currently using, issue the following command:
# netstat -v | egrep "STATISTICS|Media Speed"
ETHERNET STATISTICS (ent0) :
Media Speed Selected: Auto negotiation
Media Speed Running: 100 Mbps Full Duplex
ETHERNET STATISTICS (ent1) :
Media Speed Selected: Auto negotiation
Media Speed Running: 1000 Mbps Full Duplex
#
Note that on the example system, there is a built-in 10/100 port, and an added 1000Mbps port.
This command tells us that the built-in port has properly negotiated 100Mbps, and that the
gigabit port has properly negotiated 1000Mps. Using “ifconfig” and “netstat -r”, verify the
addressing and routing information to assure yourself that the gigabit port is the one that will
actually be communicating with the array.
To ensure the best performance, tune network options and interface parameters for maximum
iSCSI I/O throughput on the AIX system. Note that Jumbo Frames should only be enabled if
your switch supports them. The remainder of this guide assumes that your switch is
appropriately configured to enable Jumbo Frames, so you should ignore the Jumbo Frames
information if you are not using them.
Enable the TCP Large Send, TCP send and receive flow control, and Jumbo Frame features of
the AIX Gigabit Ethernet Adapter and the iSCSI Target interface (as appropriate). Use the
following commands to see and set those parameters:
# lsattr -E -l ent1
# chdev -l ent1 -a flow_ctrl=yes
# chdev -l ent1 -a large_send=yes
# chdev -l ent1 -a jumbo_frames=yes
If a user or system service (for example, IP interface) is currently using the ethernet device, you
may see a message similar to the following:
Method error (/usr/lib/methods/ucfgcommo):
0514-062 Cannot perform the requested function because the
specified device is busy.
To configure the ethernet device, you would need to first detach it. For example, if the user is IP
interface en1, then you can use the following command:
# ifconfig en1 detach
Then go back and re-run the chdev commands that failed.
There are networking parameters that should be set in SMIT as well for the interface
communicating with the PS Series array. Starting from “smit tcpip” (which is the
“Communications / TCP/IP” menu): select “Further Configuration / Network
Interfaces / Network Interface Selection / Change”, select the appropriate interface
(en1 in the current example), and make the following settings:
Interface Specific Network Options
('NULL' will unset the option)
rfc1323 [1]
tcp_mssdflt []
tcp_nodelay []
tcp_recvspace [262144]
tcp_sendspace [262144]
Apply the changes, and go back to the “Network Interfaces / Network Interface
Drivers” menu. Select the appropriate interface, and set the “Maximum IP PACKET SIZE” to
9000.
The iSCSI Software Initiator's maximum transfer size is 256KB. Assuming that the system
maximums for tcp_sendspace and tcp_recvspace are set to 262144 bytes, an ifconfig
command used to configure a gigabit Ethernet interface might look like the following:
ifconfig en1 10.1.2.216 mtu 9000 tcp_sendspace 262144 tcp_recvspace 262144
Set the sb_max network option to at least 524288, and preferably 1048576. Use the following
commands to show the current values, and to set sb_max to 1048576:
# no -a
# no -o sb_max
# no -p -o sb_max=1048576
Important Note: If you configure Jumbo Frames on an AIX host, you may find that large data
transfers can freeze your system for minutes at a time. The problem is that the way AIX
implements the “Nagle algorithm” for aggregating small packets, automatically applies to every
packet. With a 1500-byte MTU, this is not a serious impact. But when the packet size goes to
9000 bytes, it can take a long time to fill up the packet, and more often the 200ms Nagle timeout
is hit before the packet is transmitted.
AIX has a tunable parameter to set the threshhold for what is a "large" packet and what is a
"small" packet to the Nagle algorithm. When set to 1900 bytes, it forces the typical 2KB iSCSI
data segment to be considered a “large” packet and so it is sent immediately.
The AIX command to set the Nagle limit to 1900 is:
# no -o tcp_nagle_limit=1900
This is a system-wide parameter.
Configuring Etherchannel Failover on AIX
Under AIX, it is possible to configure a pair of ethernet adapters to provide failover protection.
The mechanism is called "Etherchannel", and the general process is to create a new virtual
etherchannel adapter that is composed of two physical ethernet adapters. Currently, only fail-
over is supported.
The general process for creating and configuring a new Etherchannel device is to identify the
ethernet ports you wish to use, detach them (remove the devices), create the etherchannel device,
assign the primary interface, and assign the backup interface. Then configure the new
etherchannel device as you would a normal ethernet device.
1. Make sure the Ethernet adapters used for Etherchannel all in “detach” state (unconfigured).
For example, the commands below will unconfigure en0 and en1.
# ifconfig en0 down detach
# ifconfig en1 down detach
# rmdev -l en0
# rmdev -l en1
# rmdev -dl ent0
# rmdev -dl ent1
2. Change “Enable Link Polling” value to "yes" on any 100 Mbps interfaces to be included in
the channel:
# smit chgenet
Select the appropriate ethernet interface (entX) device. Set "Enable Link Polling" for the
device to "yes" and activate the change (hit Return).
3. Create an etherchannel:
# smit etherchannel
Select "Add an Etherchannel", and then select your primary interface. This will be the
intreface preferentially used. Select only one interface.
Select "Backup Adapter" by moving the cursor to the correct line and hitting PF4 (or ESC-4),
and then selecting another ethernet interface. This interface will only be used when the main
interface is not operational. Set "Enable Gigabit Ethernet Jumbo Frames" to "yes" if
applicable to your network. Leave the other settings as default.
Press Enter to execute. The system will reply with the newly created ent device name: "ent2
Available".
4. Assign IP to newly created EtherChannel interface:
# smit chinet
Select the new entX device, and configure it as you would have your original primary address.
In general, make sure the EtherChannel Ports on the switch are on the same segment. Special
note for the Dell 5224: avoid Port 21, 22, 23 and 24.
Limitations
Cisco initiator with AIX 5.1 and 5.2 up to ML2:
Refer to the driver release notes and README for installation information.
AIX imposes the following restrictions on storage devices:
AIX does not support dynamic expansion of physical volumes.
To increase the size of the volume group, create another volume on the array group, and
add the new volume to the AIX VG (see Extending a Volume Group).
The interface used for iSCSI traffic must be set to full duplex.
The initiator will automatically discover and login to any visible iSCSI targets on the
array(s) it is configured to use. You should use IP address-based access control on the
storage array to ensure that the initiator does not access any incorrect volumes.
AIX native initiator AIX 5.2 ML7 and 5.3 ML3 and above:
The iSCSI software initiator does not implement iSCSI inbound discovery (For example,
Canonical iSCSI targets). A text file is used to configure each target.
IPv6 is not implemented.
Only CHAP(MD5) can be used to configure Initiator authentication. Target
authentication is not implemented.
The default queue_depth for iSCSI hdisk devices is 1. This should be changed to 128.
To avoid configuration problems and error log entries when you create volume groups
using iSCSI devices, follow these guidelines:
o Configure volume groups that are created using iSCSI devices to be in an inactive
state after reboot. After the iSCSI devices are configured, manually activate the
iSCSI-backed volume groups. Then, mount any associated file systems.
o Volume groups are activated during a different boot phase than the iSCSI
software driver. For this reason, it is not possible to activate iSCSI volume groups
during the boot process. It may be possible to write a custom startup script to
perform these operations later in the boot process (see Startup/Shutdown Scripts
below).
o Do not span volume groups across non-iSCSI devices.
If connectivity to iSCSI target devices is lost, I/O failures occur. To prevent I/O failures
and file system corruption, stop all I/O activity and unmount iSCSI backed file systems
before doing anything that will cause long term loss of connectivity to active iSCSI
targets.
If a loss of connectivity to iSCSI targets occurs while applications are attempting I/O
activities with iSCSI devices, I/O errors will eventually occur. It might not be possible to
unmount iSCSI backed file systems because the underlying iSCSI device stays busy.
Administrator must manually unmount all iSCSI filesystems, and varyoffvg all iSCSI
volume groups before rebooting or shutting down, otherwise the server may hang during
shutdown when it tries to unmount a filesystem after the iSCSI system has stopped (see
Startup/Shutdown Scripts below).
IBM 1 Gigabit iSCSI TOE Adapter::
The default queue_depth for iSCSI hdisk devices is 1. This should be changed to 128.
The "Maximum number of commands to queue to the adapter" parameter for the HBA
ports should be set to be 128 times the number of volumes connected.
MPIO is not supported between the IBM HBA cards and EqualLogic PS Series storage
arrays. If you have multiple HBA ports, you should split your volumes among your ports,
but be careful to only connect once to any volume.
Security considerations
AIX 5.2 ML7 and 5.3 ML3 and above:
The /etc/iscsi/directory file and the /etc/iscsi/targets configuration file are protected from non-
privileged users through file permission and ownership. CHAP secrets are saved in the
/etc/iscsi/targets file as clear text.
Note: Do not to change the original file permission and ownership of these files.
Using the IBM AIX native iSCSI initiator Under AIX 5.2 ML7 and 5.3 ML3 and above
After installing the required maintenance levels and any required fixes, and configuring the
network settings appropriately, you are ready to configure your iSCSI targets. The AIX native
iSCSI controller does not perform device discovery, so you will need to know your iSCSI IQN
target names specifically.
Verify the installation by going into SMIT (also see the SMIT section below) and verify that
there is an iSCSI heading under Devices. Select iSCSI, and iSCSI Protocol Device, and then
Change / Show Characteristics. You may desire to change the iSCSI Initiator Name that was
assigned by default, to better match your internal naming conventions. Please remember that a
legal iSCSI name must begin with “iqn.” or “eul.” To make sure that any name you assign here is
legal.
The first step to getting iSCSI storage on your host is to create an iSCSI volume on your PS
Series array group. On general principle, you would normally assign an IP address-based access
control list to this volume at this time, so that only your desired AIX host can use it. However,
for an initial configuration and verification, leave the volume configured for open access. Once
communication has been verified, then enable the ACL. If you also wish to configure CHAP
authentication, see Limitations above.
To enable and use this iSCSI volume as an hdisk device under AIX, you must add it to the iSCSI
targets file, and reconfigure your iSCSI device. Once the device is visible as an hdisk, then you
can operate on it just as you would a local physical disk (except for boot-time - see Limitations
and Startup/Shutdown Scripts), including creating a new volume group, logical volume, and
filesystem.
Note that the AIX native initiator does not keep connections to the PS Series array group active
when there is no data being transferred. A new volume that has been attached using cfgmgr is
not necessarily going to show an iSCSI connection in the PS Array GUI at any given moment.
When the volume is attached to a volume group and has a filesystem placed on it, then AIX will
maintain the iSCSI connection.
SMIT
Before attempting to configure the iSCSI initiator under AIX, first verify that it is enabled and
operational. The right place to do this is within SMIT. Run "smit iscsi" from a shell prompt,
and select the iSCSI Protocol Device. First, "List All iSCSI Protocol Devices", and verify that
you see this output:
iscsi0 Available iSCSI Protocol Device
Then hit F3 and select "Change / Show Characteristics of an iSCSI Protocol Device". Select
"iscsi0", and verify that you have something like the folllowing output:
Change / Show Characteristics of an iSCSI Protocol Device
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
iSCSI Protocol Device iscsi0
Description iSCSI Protocol Device
Status Available
iSCSI Initiator Name [iqn.localhost.hostid.>
Maximum number of commands to queue to driver [384] +#
Discovery Policy file +
Maximum Targets Allowed [16] +#
Apply change to DATABASE only no +
F1=Help F2=Refresh F3=Cancel F4=List
F5=Reset F6=Command F7=Edit F8=Image
F9=Shell F10=Exit Enter=Do
The important thing to look for is the Discovery Policy, which must be "file" (and not "ODM").
The other settings are fair to leave at the default. The "Maximum number of commands to queue
to driver" is set elsewhere in this guide, the "iSCSI Initiator Name" can be edited as long as the
resulting name is legal, and begins with "iqn.". You may need to adjust the "Maximum Targtes
Allowed" entry to suit your specific needs.
/etc/iscsi/targets
The iSCSI targets file lives at /etc/iscsi/targets, and defines every volume that the AIX host will
attempt to connect to. The format of the file is that each line defines a single target (line breaks
can be escaped) with the following fields:
<IP address of the PS Array> <Port> <IQN name> [“plaintext CHAP secret”]
A sample line might look like this:
172.19.0.100 3260 iqn.2001-05.com.equallogic:6-...-volume-name
You should always use the group address for the PS Series array group, and not any of the
individual eth port addresses. This allows the array to balance its network load normally. The
port will almost always be 3260, which is the standard iSCSI service port number. It is
necessary to know the IQN name for the volume on the PS Array; this information is displayed
in the volume configuration in the array GUI and CLI, and can be copied and pasted as necessary.
Create a line for the target you just created.
Adding drives using cfgmgr
The command to process any changes to the iSCSI configuration and add any new devices is
cfgmgr. To limit the scope to just iSCSI volumes, enter the following command:
# cfgmgr -l iscsi0
This will take a few seconds to operate, and then any new targets that were added since the last
time the command was run will be assigned new hdisk devices. To view all of your hdisk
devices, enter the following command:
# lsdev -C -c disk
hdisk0 Available 1S-08-00-8,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 1S-08-00-3,0 16 Bit LVD SCSI Disk Drive
hdisk2 Available Other iSCSI Disk Drive
#
In this example, the AIX host has two local SCSI devices, and one PS Array volume attached.
hdisk2 is available to be used as a new volume group or as an addition to an existing VG. Note
the limitation stated above: never mix iSCSI and non-iSCSI devices in a single volume group.
Corruption will result.
To determine what iSCSI target a particular hdisk device is connected to, use the following
command, and look for the following two device attributes:
# lsattr -E -l hdisk2
host_addr 172.19.0.100
target_name iqn.2001-05.com.equallogic:6-...-volume-name
These two parameters tell you the IP address that the iSCSI target was logged in through, and the
IQN name of the iSCSI target. It is an EqualLogic convention to place the name of the volume
at the end of the IQN name for every target.
Once the new hdisk device has been identified, you can modify the device properties (see Queue
Depth below), go into SMIT and configure it as a new volume group, and create an LV and a
filesystem on it, or you can manipulate the device from the shell prompt. Managing the AIX
LVM is beyond the scope of this document.
Queue Depth
According to IBM’s documentation on their iSCSI initiator, iSCSI-based disks may benefit from
increasing the queue depth on the drive if the storage array supports the necessary feature sets
(“command tag queueing” and “NACA=1”). The EqualLogic PS Array supports those features
properly, and recommends a queue depth of 128.
Before connecting to your iSCSI volumes, you will need to determine what your queue depth
setting will be for each your connected volumes, so you can set the initiator itself to have a
capacity large enough to service all of the volumes. On the iscsi0 device, there is a "number of
command elements" attribute, which effectively defines the total queue depth for the entire
iscsi0 device tree. The default value is 200. For best performance, this parameter should be
greater than or equal to the sum of all iSCSI hdisk queue_depth settings. For example, if you
have 20 iSCSI devices each with queue_depth=12, the num_cmd_elems parameter should be at
least 240. If it is less than 240 it should still work, though there will be times when the driver will
stall a command waiting for a free command element.
This can be changed at the iSCSI page in smit (run “smitty iscsi” to get to that page), or it can
be set at the command line:
# chdev -l iscsi0 -a num_cmd_elems=384
Note that this parameter can only be changed on the iSCSI initiator when no targets are currently
connected. If you see the following error, either from the command line or in smit, then you will
need to comment out all of your targets in /etc/iscsi/targets and re-run "cfgmgr -l
iscsi0" to clear the connections before you can set it.
Method error (/usr/lib/methods/chggen):
0514-029 Cannot perform the requested function because a
child device of the specified device is not in a correct state.
To change the queue depth for an iSCSI disk device, first confirm the current value, then change
it, then confirm the new value:
# lsattr -E -l hdisk2
queue_depth 1
# chdev -l hdisk2 -a queue_depth=128
hdisk23 changed
# lsattr -E -l hdisk2
queue_depth 128
The change can only be made on a device which is not currently being used – either the device
has not been assigned to a VG, or the VG that it belongs to has been deactivated. If you see this
error:
# chdev -l hdisk2 -a queue_depth=128
Method error (/usr/lib/methods/chgdisk):
0514-062 Cannot perform the requested function because the
specified device is busy.
Then you should unmount any LVs that are on that disk, and go to SMIT and explicitly
deactivate the VG that contains the desired hdisk. Make the change outlined above, and then
reactivate the VG and remount the LVs as necessary.
Note that the hdisk queue_depth parameter cannot be changed within SMIT. SMIT has some
difficulties with iSCSI-based disk parameters, so this change must be done at the command
prompt.
Setting Disk Timeouts
In the event of a short-term network interruption or a controller failover on the PS Array, the
AIX host may stop receiving data from the PS Array for some time. On many operating systems,
it's necessary to configure an extended disk timeout, to have the operating system tolerate much
longer periods of network interruption before failing the I/O operation, or failing the entire disk
device. For every operating system, EqualLogic recommends configuring an operating system to
tolerate up to 120 seconds of communications loss to withstand a full controller failover.
AIX will tolerate at least 120 seconds of communication loss before returning errors to an
application, so it does not require any reconfiguration. Note that an application is still free to
retry its operations even after that period.
There are two places in AIX with parameters which govern how the OS handles timeouts: within
the TCP stack and within the SCSI system. The system-wide tcp_keepidle parameter
determines how long an unclosed TCP session will be kept alive without response. The units for
this are half-seconds, and the system default is 14400 – 2 hours.
The disk-level I/O timeout value is set on each hdisk device separately, and is called
rw_timeout. It defaults to 30 seconds. However, exceeding 30 seconds of I/O timeout only
causes a temporary error to be logged, and no data is lost.
To configure the system-wide tcp_keepidle parameter:
# no -o tcp_keepidle
tcp_keepidle = 14400
# no -o tcp_keepidle=14400
To configure the disk-level I/O timeout parameter:
# lsattr -El hdisk2
rw_timeout 30
# chdev -l hdisk2 -a rw_timeout=120
A temporary I/O error that was recovered appears in the system error log like this (not including
the additional SENSE DATA block):
LABEL: SC_DISK_ERR4
Date/Time: Fri Dec 29 14:35:40 EST 2006
Type: TEMP
Resource Name: hdisk2
Description
DISK OPERATION ERROR
A permanent I/O error that caused an error to be returned to the calling application appears in the
system log like this (not including the additional SENSE DATA block):
LABEL: SC_DISK_ERR3
Date/Time: Fri Dec 29 14:44:20 EST 2006
Type: PERM
Resource Name: hdisk2
Description
DISK OPERATION ERROR
Deleting iSCSI devices
To remove an iSCSI volume from your system, first unmount any applicable filesystems, and
disable the volume group using varyoffvg. You may then also remove the appropriate line from
/etc/iscsi/targets, but note that removing an entry on the iSCSI targets file does not cause AIX to
dismount the volume. It only prevents the AIX host from reattaching to it later.
To actually remove an iSCSI device from your AIX system, you must go to SMIT, to the
Devices / Fixed Disk section, and remove the appropriate volume there. The volume will no
longer be connected to the host until another cfgmgr call is made, but only if the entry in
/etc/iscsi/targets still exists.
Startup/Shutdown Scripts
Since the native AIX initiator does not handle automatically activating iSCSI-based volume
groups or mounting filesystems on them, that process has to be handled manually, or by the use
of a start-up script.
As stated in the Limitations section, you must create your volume groups with the option to not
activate them at boot. Also, you must create your filesystems to not automatically mount at boot.
The iSCSI initiator is not running when the volume group system wants to activate VGs, or when
the automatic mouting of filesystems happens. You must either manually reactivete your VGs
and manually mount your filesystems, or you must create a script to do it for you.
Here is a simple, and not robust, script for connecting an iSCSI volume at startup and
disconnecting it at shutdown. Note that this script does not attempt to detect or handle errors.
The script references a configuration file, by default /etc/iscsi/volumegroups, which should
contain lines declaring each volume group to activate, followed by the mount points for each
filesystem contained within that VG. To use this script, install it as /etc/rc.d/init.d/iscsi,
and enter the following commands:
# chmod 755 /etc/rc.d/init.d/iscsi
# cd /etc/rc.d/rc2.d
# ln -s /etc/rc.d/init.d/iscsi S50iscsi
# ln -s /etc/rc.d/init.d/iscsi K50iscsi
Then place the sample configuration file in /etc/iscsi/volumegroups, and enter the
appropriate information about your iSCSI-based volume groups.
/etc/rc.d/init.d/iscsi
#!/bin/sh
#
# Sample iSCSI Startup Script for AIX 5.3
#
# This file normally resides as /etc/rc.d/init.d/iscsi, and is
# symbolically linked to /etc/rc.d/rc2.d/S50iscsi and K50iscsi.
#
# Copyright 2007 EqualLogic, Inc.
#
# Version history:
# 1.0 - initial pass GWA
ISCSI_VGFILE=/etc/iscsi/volumegroups
case "$1" in
start)
awk '!/^\#/ { if (NF >= 2) { print "Executing \"varyonvg " $1
"\"..." ; system("varyonvg " $1) ; for (i=2; i<=NF; i+=1) {print
"Executing \"mount " $i "\"..." ; system("mount " $i)}}}' $ISCSI_VGFILE
;;
stop)
awk '!/^\#/ { if (NF >= 2) { for (i=2; i<=NF; i+=1) {print
"Executing \"umount " $i "\"..." ; system("umount " $i)} ; print
"Executing \"varyoffvg " $1 "\"..." ; system("varyoffvg " $1)}}'
$ISCSI_VGFILE
;;
esac
/etc/iscsi/volumegroups
# /etc/iscsi/volumegroups
#
# This file defines the volume groups and associated filesystems so
# that they can be activated and mounted properly during startup,
# and also dismounted and deactivated properly during shutdown.
# The AIX native iSCSI initiator does not handle these functions
# automatically. This file is read by /etc/rc.d/init.d/iscsi,
# which is normally linked to /etc/rc.d/rc2.d/S50iscsi and K50iscsi.
#
# Copyright 2007 EqualLogic Inc.
#
# Format: one entry per line. First field is the name of the
# volume group, additional fields are the mount points. Whitespace
# must separate the fields. Blank lines are ignored.
#
# volume-group mountpoint [mountpoints...]
# iscsi-vg2 /test /test2 /test3
Using the Cisco iSCSI initiator for IBM AIX AIX 5.1 and 5.2 up to ML2
Once the initiator and the required fix have been installed, you are ready to begin using iSCSI
with your AIX system. Please follow the directions in the README file that shipped in the
initiator TAR file, as it contains specific instructions, including required modifications ti
/etc/rc.shutdown to properly unmount iSCSI volumes during shutdown.
The three files that are used to configure iSCSI are:
/etc/iscsi.conf discovery portals and configuration parameters are defined
/etc/iscsi_bindings
device numbers are recorded for persistence and consistency
/etc/filesystems.iscsi iSCSI-based filesystems are automatically mounted and unmounted
Note that the queue depth for the iSCSI hdisk devices will default to 1. The administrator can
change this to improve performance – see Queue Depth above. Also note the Setting Disk
Timeouts section above, although no timeout configuration changes are normally needed.
/etc/iscsi.conf
To work properly with an EqualLogic PS Array, the only information you need to put in
iscsi.conf is the discovery address. This is where the initiator will go to get a list of available
iSCSI targets, and will automatically login to them. Add a line to the file like this:
DiscoveryAddress=172.19.0.1
The address supplied should be the group address, not any of the individual eth port addresses on
your array(s). If you have multiple groups, you can specify multiple discovery addresses here,
and the system will attach to available iSCSI targets on all of them.
There are many other commands and directives available in the iscsi.conf file. It is self-
documenting, so if you have a need for other options (CHAP authentication, for instance), please
see the file itself for instructions on further configuration. For simple operation, setting the
discovery address is sufficient. There are a number of other options available, but we do not
recommend configuring them unless you know you have a specific need. We do not require any
of them.
iscsi-reload
Once you have defined your target portal, you need to instruct the initiator to reread its
configuration files and take appropriate actions to use any new resources that become available.
To do this, you execute the command:
# iscsi-reload
This instructs the initiator to read in new information, while preserving any existing connections.
Other commands available are “/etc/iscsi.clean rem” (which unloads the initiator
completely, dropping all connections), and “/etc/rc.iscsi” (which starts the initiator from
scratch). Note that stopping the initiator disconnects from all targets immediately. This should
not be done lightly, as it can cause real problems for your host if there was a process accessing
any data on one of those volumes. However, it may occasionally be necessary to force the
initiator to completely flush its configuration and start over. Generally speaking, a host reboot
may be more appropriate, and certainly safer.
iscsi-ls
Once you have reloaded the initiator, your configured volume(s) should be visible on your
system. The command “iscsi-ls” is used to interrogate the iSCSI system.
# iscsi-ls -l
*****************************************************************************
Cisco iSCSI Driver Version ... 3.3.2 (25-Jul-2003)
*****************************************************************************
NAME : iqn.1987-05.com.equallogic:00...6-volname
TARGET ADDRESS : 172.19.0.100:3260,1
TARGET ID : 0
STATUS : Connected 172.19.0.25:29174<->172.19.0.100:3260
at Wed Dec 28 08:01:24 2005
SESSION : ISID 00023d000001 TSID 02
LUN 0 : hdisk2
DEVICE STATUS : AVAILABLE
DEVICE DETAILS :
VENDOR : EQUALLOGIC
PRODUCT : PS-200E
PRODUCT REVISION: 0073
DISK CAPACITY : BLOCKCOUNT 0x21eb38f, BLOCKLEN 0x200,
DISKSIZE 17366.45 MB
The most important thing that this report tells us the the device name for the iSCSI volume.
Look at the “LUN” line, and note the name there (hdisk2). PS Arrays provide only a single lun
per target, and so each iSCSI volume will appear as a single hdisk device. (For EqualLogic PS
Arrays, you can identify the volume being used by looking at the end of the “NAME” field - in
this case, “volname.”)
/etc/iscsi_bindings
Once you have successfully attached to a volume, the target number is recorded in the file
“/etc/iscsi_bindings.” This ensures that, on subsequent boots when the order of discovery
might have changed, a particular target is assigned the same device node. Iscsi_bindings is a
plain text file and can be edited by hand. Its content will look something like this:
# iSCSI bindings, file format version 1.0.
# NOTE: this file is automatically maintained by the iSCSI daemon.
# You should not need to edit this file under most circumstances.
# If iSCSI targets in this file have been permanently deleted, you
# may wish to delete the bindings for the deleted targets.
#
# Format:
# bus target iSCSI
# id id TargetName
#
0 0 iqn.2001-05.com.equallogic:6-8a0900-ff4000301-7e1ff088ce9430dd-volname
The file will contain one line for each target that it has connected to. The most useful reason for
editing it is to remove old connections that are no longer used. There is a limit of 15 targets, so
if you have attached to and then discarded 15 volumes, your 16th
will fail because the table is full.
You will want to edit this file and remove the entries for the volumes you will no longer be
accessing.
Once you have attached to your iSCSI volume, you can treat it as an ordinary hard drive, with
one exception (see /etc/filesystems.iscsi below).
Once the iSCSI volume has been bound to an hdisk device, you can manipulate it as you would
any other physical disk.
/etc/filesystems.iscsi
To mount your new filesystem at boot time, you cannot simply add it to the system’s
/etc/filesystems. filesystems is scanned before the network system is operational, so it
will always fail to find and mount your iSCSI-based filesystems. To get around this, the iSCSI
system uses its own table, called /etc/filesystems.iscsi. To properly activate an iSCSI-
based volume group and mount its filesystems at boot time, you must use both
/etc/filesystems, to define the mount options, and /etc/filesystems.iscsi to tell the
iSCSI system to activate and mount those filesystems after the iSCSI system has started.
The entry for /etc/filesystems should describe the filesystem normally, as you would for any
direct-attached storage, except that mount should be false, so that AIX doesn’t try to mount it
when it’s mounting direct-attached storage.
/mount-point:
dev = /dev/lv00
vfs = jfs
log = /dev/loglv01
mount = false
options = rw
account = false
Then you need to add a single line to /etc/filesystems.iscsi in order to cause the system to
mount any logical volumes after activating the new volume groups. The entry will simply be the
LV device, which the iSCSI system will use to enable the enclosing volume group, and then
mount the filesystem using the information in /etc/filesystems. The entries will look like:
/dev/lv00
/dev/lveql01
And now your system is configured properly and ready to go. There are many parameters and
operations in the AIX LVM system that are well beyond the scope of this document. This is
intended just to get you started.
iscsiactlun
The iscsiactlun command dynamically rescans the device information for any session that has
already been established. This will not cause the system to discover any new targets, but it will
permit the system to determine any changes to a volume (most importantly, a change in size) that
is already being used.
Mounting a Snapshot
To mount a Snapshot, you can either mount the snapshot in place of the original volume, or
mount both the base volume and the snapshot at the same time.
Mount the snapshot in place of the original volume
Unmount all file system(s) on iSCSI volumes that are on the primary AIX volume group, and
deactivate the primary volume group using “varyoffvg”. Then perform the following:
o From the AIX server, stop the initiator first by running the “/etc/iscsi.clean rem”
command.
o From the PS Series Storage Array, set the volume offline
o From PS Series Storage Array set the snapshot online
o From the AIX server start the initiator by running the # /etc/rc.iscsi command
o Activate the primary volume group.
o Mount any required file systems.
Mount both the base volume and the snapshot at the same time
o On the PS Series group, set the snapshot online
o Perform the following commands from the AIX server to load the new online snapshot: # /sbin/init.d/iscsi-reload
# iscsiactlun
# lspv
# iscsi-ls -l
o Mount any required file systems.
Extending a Volume Group
To extend an AIX volume group:
o Create a new volume in the group.
o Execute the following comands so that the new volume is visible to the system: # iscsi-reload
# iscsiactlun
# lspv
# iscsi-ls -l
o Extend the volume group using the appropriate smit commands.
Stopping and Starting the Initiator Service
If required, use these commands to stop and restart the iSCSI system on the AIX system when
taking a volume offline:
o From the AIX server stop the initiator first by running the “/etc/iscsi.clean rem”
command
o On the PS Series Storage Array set the volume offline
o From the AIX server start the initiator by running the “/etc/rc.iscsi” command
Using the IBM 1 Gigabit iSCSI TOE Adapter
Although EqualLogic has not tested and qualified this initiator, we can provide some guidance
on getting it properly installed on your AIX systems. However, we can make no guarantees
about its specific compatibility with EqualLogic storage arrays.
Please verify that the BIOS and firmware of the TOE card are fully up to date before proceeding.
IBM's documentation for setting up the iSCSI TOE card can be found here.
Configuration of the IBM iSCSI TOE card is performed in SMIT. Each port is a separate SCSI
interface, and must have its own separate configuration. The first port is given the name "ics0",
the second "ics1", and so on. Each port must havre a separate IP address, and will consult a
separate target configuration file. They can, however, use the same CHAP secrets file if
appropriate.
To configure an iSCSI port, run "smit iscsi", choose "iSCSI Adapter", and then "Change /
Show Characteristics of an iSCSI Adapter".
iSCSI Adapter
Move cursor to desired item and press Enter.
List All iSCSI Adapters
Change / Show Characteristics of an iSCSI Adapter
Generate Error Report
Trace iSCSI Adapter
+--------------------------------------------------------------------------+
| iSCSI Adapter |
| |
| Move cursor to desired item and press Enter. |
| |
| ics0 Available 00-09 iSCSI Adapter |
| ics1 Available 09-09 iSCSI Adapter |
| |
| F1=Help F2=Refresh F3=Cancel |
| F8=Image F10=Exit Enter=Do |
F1| /=Find n=Find Next |
F9+--------------------------------------------------------------------------+
Select the particular port you wish to configure, and hit return. You will see the following
configuration page:
Change / Show Characteristics of an iSCSI Adapter
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
iSCSI Adapter ics0
Description iSCSI Adapter
Status Available
Location 00-09
iSCSI Initiator Name []
Maximum number of commands to queue to the adapter [200] +#
Maximum transfer size [0x100000] +
Discovery Filename [/etc/iscsi/targetshw]
Discovery Policy SLP +
Automatic discovery secrets filename []
Adapter IP Address []
Adapter Subnet Mask []
Adapter Gateway Address []
Apply change to DATABASE only no +
F1=Help F2=Refresh F3=Cancel F4=List
F5=Reset F6=Command F7=Edit F8=Image
F9=Shell F10=Exit Enter=Do
Please assign the adapter an iSCSI Initiator Name first. The only requirement for the iSCSI
Initiator Name is that it begin with "iqn.", and be globally unique; however, it's common to
include the hostname and other descriptive information in the name. For instance, "iqn.aix-
server1-port0".
Since we recommend a queue depth of 128 for every iSCSI hdisk device, we recommend setting
the "Maximum number of commands to queue to the adapter" parameter to 128 times the number
of volumes being connected. For instance, if you are connecting to three volumes, set this
parameter to 384.
Although the Discovery Filename defaults to "/etc/iscsi/targetshw" (where "hw" means this
is the targets file for a hardware initiator), because of the possibility of multiple TOE card ports,
we recommending setting it to "/etc/iscsi/targetshw0" for ics0, hw1 for ics1, and so on. If
you are using CHAP, please consult the IBM documentation for configuration, and set the
"Automatic discovery secrets filename" parameter appropriately.
Set the Discovery Policy to "FILE", and assign this HBA an IP address, subnet, and, if
appropriate, a default gateway. Remember that this TOE card is not an ethernet adapter, and will
not provide you general TCP/IP services. So you cannot, for instance, ping the array over your
TOE card, or telnet to it. If you desire, you can certainly configure one of your normal ethernet
interfaces to also be on the same subnet as the iSCSI traffic if you wish to manage your array
from your AIX server.
Your configuration should now look something like this:
Change / Show Characteristics of an iSCSI Adapter
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
iSCSI Adapter ics0
Description iSCSI Adapter
Status Available
Location 00-09
iSCSI Initiator Name [iqn.aix-server1-port0]
Maximum number of commands to queue to the adapter [256] +#
Maximum transfer size [0x100000] +
Discovery Filename [/etc/iscsi/targetshw0]
Discovery Policy file +
Automatic discovery secrets filename []
Adapter IP Address [172.25.6.174]
Adapter Subnet Mask [255.255.255.0]
Adapter Gateway Address []
Apply change to DATABASE only no +
F1=Help F2=Refresh F3=Cancel F4=List
F5=Reset F6=Command F7=Edit F8=Image
F9=Shell F10=Exit Enter=Do
Hit Enter to commit the changes. If the command fails with the following error, then you must
go back to the iSCSI Adapter page and set the "Apply change to DATABASE only" parameter to
"Yes." That will cause the changes to be applied only at reboot. Then you can reboot to get your
iSCSI Adapter operational.
Method error (/usr/lib/methods/chggen):
0514-029 Cannot perform the requested function because a
child device of the specified device is not in a correct state.
It's fine to configure all ports this way before having to restart the AIX server.
Once you have configured your iSCSI Adapters, you must connect them to their respective
targets. The process is identical to connecting the AIX native software initiator to its targets,
with the two exceptions that the device to scan in cfgmgr is "ics0" (or "ics1" and so on),
instead of "iscsi0"; and the configuration file will be /etc/iscsi/targetshw0 (or
/etc/iscsi/targetshw1, and so on).
Please see the /etc/iscsi/targets section for information about writing the targets configuration
file(s); Adding drives using cfgmgr for information about rescanning for targets; Queue Depth
for setting the Queue Depth on an iSCSI-based hdisk device (and make sure you do that before
using the disk in any way); and Setting Disk Timeouts for information about setting appropriate
timeouts on the devices. Note that Deleting iSCSI Devices also applies.
Since the iSCSI Adapter does not depend on any networking services, the devices connected
through your TOE card ports are always available. You do not have to do anything special to get
them activated and mounted during start-up. Treat them like locally direct-attached storage.
Dell Confidential For Authorized Use Only
Document Control and History Information
Revision Change Author/Organization Reason Approved
By/Organization Date
A02 M. Cynewski / GCSS ISO Compliance M. Gauthier / GCSS 9/2/2009
A01 Unknown Initial Release Unknown 4/28/2008