This README is for the Linux Fibre Channel HBA device driver version 8.01.06 for the IBM FAStT FC-2, the IBM FAStT FC2-133 Host Bus Adapters, and the IBM DS4000 FC 4Gbps PCI-X Single and Dual HBA. IMPORTANT: This version of the Linux Fibre Channel HBA device driver should only be used with the 2.6 kernel. Refer to the IBM Support Web site or the Fibre Channel Host Bus Adapter (FC HBA) support CD that is shipped with your FC HBA for the IBM FAStT/DS4000 FC-2, FC2-133 single and dual port and FC 4Gbps single and dual port Host Bus Adapter Installation and User's guide, the latest information, and updated readme file. NOTE TO SERVICE – Reference RETAIN # N/A (C) Copyright International Business Machines Corporation 1999, 2006. All rights reserved. US Government Users Restricted Rights - Use, duplication, or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Note: Before using this information and the product it supports, read the general information in section 12.0 "Trademarks and Notices" in this document. Also read the section 9.0 Limitations before installing this device driver. Products supported: ------------------------------------------------------------------------------ | DS4000 Adapter | Qlogic Adapter | IBM Feature Code | IBM Option P/N | ------------------------------------------------------------------------------ |FAStT FC2 | QLA2310FL-IBM-SP | FC2130 | 19K1246 | |FAStT FC2-133 | QLA2340-IBM-SP | FC2104 | 24P0960 | |DS4000 FC 4Gb-Single | QLA2460-IBM-SP | FC2105 | 39M5894 | |DS4000 FC 4Gb-Dual | QLA2462-IBM-SP | FC2106 | 39M5895 | ------------------------------------------------------------------------------- Last Update: 10/27/2006 ======================================================================= Contents -------- 1.0 OS Support 2.0 Supported Features 3.0 Change History 4.0 Host Adapter configuration 4.1 Update IBM DS4000/FAStT Host Adapter BIOS 4.2 Configure the NOVRAM Setting for the IBM DS4000/FAStT Host Adapter 5.0 Download the IBM DS4000/FAStT Host Bus Adapter driver source code for Linux 6.0 Building a Driver from the Sources Code 6.1 Install the kernel source and header packages 6.2 Install the IBM DS4000/FAStT (Qlogic) driver source code 6.3 Building the device driver 7.0 Loading and Configuring the driver 7.1 Enable more than 1 SCSI device per adapter 7.2 Install SANSurfer 7.3 Modify the module load time options 7.4 Loading the Driver manually 7.5 Loading the Driver using a ramdisk image 7.6 Rebuilding the ramdisk image after configuration changes 8.0 Failover Support 8.1 How to enable the Failover support in the Driver 8.2 Configuration Changes Made via SANSurfer 8.3 Persistent Binding 8.4 Configuration Data 8.5 Hot Add new LUNs 9.0 Limitations 9.1 Proc Filesystem Support 10.0 Driver file Contents 11.0 WEB Sites and Support Phone Number 12.0 Trademarks and Notices 13.0 Disclaimer ======================================================================= 1.0 OS Support -------------- The 8.01.06 device driver was tested with the following Linux versions and kernel versions: ----------------------------------------------------------------- | RedHat 4 IA-32 update 4 | 2.6.9-42.EL UP SMP | | | | |--------------------------------|--------------------------------| | RedHat 4 IA-64 update 4 | 2.6.9-42.EL UP SMP | | | | |--------------------------------|--------------------------------| | RedHat 4 AMD-64/EM64T update 4 | 2.6.9-42.EL UP SMP | | (x86 and x86_64 versions) | | |--------------------------------|--------------------------------| | SuSe SLES 9 SP3 | 2.6.5-7.244 UP SMP | | 32-bit | 2.6.5-7.257 | | | 2.6.5-7.276 | |--------------------------------|--------------------------------| | SuSe SLES 9 SP3 | 2.6.5-7.244 UP SMP | | IA-64 | 2.6.5-7.257 | | | 2.6.5-7.276 | |--------------------------------|--------------------------------| | SuSe SLES 9 AMD-64/EM64T SP3 | 2.6.5-7.244 UP SMP | | (x86 and x86_64 versions) | 2.6.5-7.257 | | | 2.6.5-7.276 | ----------------------------------------------------------------- Earlier or later versions of the vendor kernels or generic kernels have not been tested with this device driver version and may not be supported with this release. The Qlogic device driver in the Linux vendor packages have not been tested and are not supported by IBM. This driver package contains the Qlogic driver source code only. The device driver must be compiled from the source code for your specific Linux installation. This readme file makes an assumption that the user has already configured their storage subsystem properly for a Linux server with storage partitioning enabled and configured with the host type set to LINUX if the driver is configured as LUN failover/failback (fo) driver. If this driver is configured as non-failover/non-failback (non-fo) driver, the host type set to LNXCLVMWARE (in Linux clustering environments) or Linux with a special DS4000 storage manager SMCli script applied to disable the AVT/ADT option (in Linux nonclustering environment) and a multipath driver for LUN failover and failback, like IBM DS4000 Linux RDAC driver, must also be installed. Please refer to the DS4000 Storage Manager host software package for Linux operating system environments for more info on the disable ADT/AVT. Only like type adapters can be configured as a failover pair. An IBM FAStT FC2-133 Host Bus Adapter cannot failover to an IBM DS4000 FC 4Gbps Host Bus Adapter. IBM DS4000 FC 4Gbps Host Adapter Driver uses the qla2400.ko device driver. The IBM FAStT FC-2 and FC2-133 Host Bus Adapters use the qla2300.ko device driver. This readme will use the term IBM DS4000 Host Adapter to refer to the the IBM FAStT FC-2 Host Bus Adapter, the IBM FAStT FC2-133 Host Bus Adapter, and the IBM DS4000 FC 4Gbps Host Bus Adapter. ======================================================================= 2.0 Supported Features ---------------------- * FCAL - direct attach loop * Point-to-point * Fabric support * Initiator mode only * Fault recovery on down loops * Persistent binding * Extended LUN support up to 255 LUNs * FC tape support * Non Failover and Failover capability ======================================================================= 3.0 Change History ------------------- Please refer to change history file ibm_dd_ds4kfc_8.01.06_linux2.6_anycpu.chg. ======================================================================= 4.0 Host Adapter configuration ------------------------------ 4.1 Update IBM DS4000/FAStT Fiber Channel Host Bus Adapter BIOS. --------------------------------------------------------------------- IMPORTANT NOTE: The DS4000 Fibre Channel Host Bus Adapter BIOS must be at the latest level that is available in the DS4000 System Storage support web site for your IBM DS4000 Fibre Channel Host Bus Adapter type. Currently, the latest BIOS version for the IBM FC-2 and FC2-133 HBA is 1.47. Whereas, the BIOS version for the FC 4Gbps single and dual port HBAs is 1.12. Please refer to readme file in the BIOS package of your Fibre Channel HBA for the information on how to update the BIOS and NVRAM settings of your Fibre Channel HBA. IMPORTANT: If you made modifications to FC HBA NVRAM settings, you will have to make the modifications again because the NVRAM will be reset to the new IBM default values. An alternative to using the bootable diskette or the DS4000 FC 4Gbps FC HBA support CD to update the FC HBA BIOS and NVRAM, the QLogic SANsurfer program can be used for this purpose. This program was previously released as IBM FAStT Management Suite Java (FAStT MSJ). Starting with the version 2.0.30b62 release, IBM will not release separately an IBMized version of the Qlogic SANsurfer HBA Manager or SANsurfer PRO program. The terms SANsurfer and SANsurfer will be used interchangeably from the 2.0.30b62 release forward. The flash *.bin and NVRAM *.dat file are included in the web-download BIOS package or in the BIOS directory of the support CD for this purpose. These files are used in conjunction with the QLogic SANsurfer program to update the IBM DS4000 FC HBA Multiboot flash and NVRAM data without the need to reboot the servers to the bootable diskette or to the DS4000 FC 4Gbps FC HBA support CD (use the password config when prompted) However, the server must be rebooted after the Multiboot image flash or NVRAM update via QLogic SANsurfer program to enable it. The SANsurfer program is available at the IBM System Storage™ Disk Storage Systems Technical Support web site: http://www.ibm.com/servers/storage/support/disk/ 3.1.2 EFI Driver Update For IA64 Environment -------------------------------------------- IMPORTANT NOTE: The DS4000 Fibre Channel Host Bus Adapter (FC HBA) BIOS or EFI driver must be at the latest level that is available in the DS4000 System Storage support web site for your IBM DS4000 Fibre Channel Host Bus Adapter type. The FC HBA with IA32 BIOS will function in a IA64 server. However, you will not be able to modify the NVSRAM settings via the adapter Fast!UTIL utility or to configure the adapter for "root boot" or "remote boot". You have to install the appropriate "EFI BIOS" file for your adapter to enable those functions. Currently, the latest BIOS and EFI driver version for the IBM FC-2 and FC2-133 FC HBA is 1.47 and 1.37, respectively. Whereas, the BIOS and EFI driver version for the FC 4Gbps single and dual port FC HBAs is 1.12 and 1.04, respectively. You can not update the IBM DS4000 Fibre Channel Host Bus Adapter (FC HBA) BIOS or EFI driver by booting to a DOS diskette in a IA64 server like you would in an IA32 server. In addition, you will not be able to modify the NVSRAM settings via the adapter Fast!UTIL utility or to configure the adapter for "root boot" or "remote boot". You have to install the appropriate "EFI BIOS" file for your adapter to enable those functions. You can use the efiutil in the efi driver package for your FC HBA to install the EFI BIOS in your adapter and to modify the NVSRAM settings. An alternative to using the bootable diskette or the DS4000 FC 4Gbps FC HBA support CD to update the FC HBA BIOS and NVRAM, the QLogic SANsurfer program can be used for this purpose. This program was previously released as IBM FAStT Management Suite Java (FAStT MSJ). Starting with the version 2.0.30b62 release, IBM will not release separately an IBMized version of the Qlogic SANsurfer HBA Manager or SANsurfer PRO program. The terms FAStT-MSJ and SANsurfer will be used interchangeably from the 2.0.30b62 release forward. The flash *.bin and NVRAM *.dat file are included in the web-download BIOS package or in the BIOS directory of the support CD for this purpose. These files are used in conjunction with the QLogic SANsurfer program to update the IBM DS4000 FC HBA Multiboot flash and NVRAM data without the need to reboot the servers to the bootable diskette or to the DS4000 FC 4Gbps FC HBA support CD (use the password config when prompted) However, the server must be rebooted after the Multiboot image flash or NVRAM update via QLogic SANsurfer program to enable it. The SANsurfer program is available at the IBM System Storage™ Disk Storage Systems Technical Support web site: http://www.ibm.com/servers/storage/support/disk/ 4.2 Configure the NOVRAM settings for the IBM DS4000/FAStT Fibre Channel Host Bus Adapter ------------------------------------------------------------------------- Note: Currently the only method to change the adapter NOVRAM settings in an IA-64 environment are with SANSurfer. All settings, except for the following, should maintain the IBM defaults. - Host Adapter settings Loop reset delay - 8. - Advanced Adapter Settings LUNs per target - 0 Enable Target Reset - Yes Port down retry count - 12 1. As the host boots, press when prompted. 2. After the Fast!Util program loads, the display will depend on whether there are multiple IBM DS4000/FAStT Adapters installed. If there are multiple IBM DS4000/FAStT Adapters, a list of addresses occupied by those Host Adapters will appear. Using the arrow keys, select the desired adapter and press ENTER. The Fast!Util Options menu will then appear. For further information refer to the IBM DS4000/FAStT Host Adapter publication. ======================================================================= 5.0 Download the IBM DS4000/FAStT Host Bus Adapter driver source code for Linux and the latest Qlogic SANSurfer from the IBM Support website. ------------------------------------------------------------------------ Download the IBM DS4000 Host Adapter driver source code for Linux from the IBM System Storage Technical Support website. If prompted "What would you like to do with this file?" choose "Save this file to disk". Insert a blank formatted diskette and download to the diskette directly. This file will be in the .tgz file compression format. Download the latest Qlogic SANSurfer for Linux from the IBM Support website. If prompted "What would you like to do with this file?" choose "Save this file to disk". This file will need to be saved to your hard disk drive. This file will be in the .tgz file compression format. ======================================================================= 6.0 Building a Driver from the Source Code ------------------------------------------ From the source code, you can build a qla2xxx.ko and a qla2200.ko, qla2300.ko, qla2322.ko, qla6312.ko, or qla2400.ko for your host system, and load the driver manually or automatically using a RAMDISK image during system boot time. 6.1 Install the kernel source, header, and ncurses packages ----------------------------------------------------------- 1. Install the kernel-source RPM files for the supported kernel. These files are available from your Linux vendor or your OS installation CD set. Also, be sure that the ncurses packages and the gcc packages have been installed. Kernel installation instructions for different linux versions may vary. Please refer to your linux vendor documentation for procedures to update the kernel source code for your linux version. # rpm -iv kernel-source*.rpm (SLES9) # rpm -iv kernel-syms*.rpm (SLES9) # rpm -iv kernel-devel*.rpm (RH4) # rpm -iv kernel-utils*.rpm # rpm -iv ncurses*.rpm # rpm -iv ncurses-devel*.rpm # rpm -iv gcc*.rpm # rpm -iv libgcc*.rpm Verify that kernel-source, ncurses and gcc RPMS are installed. # rpm -qa | grep kernel # rpm -qa | grep ncurses # rpm -qa | grep gcc 6.2 Install the qlogic driver source code for the IBM DS4000/FAStT Fibre Channel Host Adapters ------------------------------------------------------------------------- 1. Using the adapter driver diskette you created in Section 5, copy the qla2xxx-v8.01.06-dist.tgz file to /qla2xxx-v8.01.06. Follow these steps: # mkdir /qla2xxx-v8.01.06 # cd /qla2xxx-v8.01.06 # mount /mnt/floppy # cp /mnt/floppy/*.tgz . (the period at the end is required) # tar -xvzf *.tgz This will create a folder named "qlogic" and extract out the following files to the "qlogic" folder: drvrinstall Script file to copy driver source files included in the driver source tgz file. qla2xxx-src-.tgz Compressed binary distribution file for driver sources. This file is the same type of driver source tgz file as the ones used for distributing earlier versions of the QLA2XXX drivers. libinstall Script file to install/setup HBA API library. libremove Script file to remove HBA API library. qlapi--rel.tgz Compressed binary distribution file for API library. 2. Execute the following commands: # cd qlogic # ./drvrsetup (this will extract the driver source files to the current directory) # cd qla2xxx-8.01.06 3. Type the following command in to install/setup SNIA API library: # cd qlogic # ./libinstall 4. Type the following command to remove SNIA API library: # cd qlogic # ./libremove 6.3 Building the device driver ----------------------------------------------------------- This section makes extensive use of the build.sh script located in driver source (extras/build.sh). This script currently supports driver compilation (installation and updates) on SLES9 and RedHat4 distributions on four main hardware platforms: (x86, x86_64, ia64 and ppc64). The build.sh script supports for following directives: # ./extras/build.sh Build the driver sources based on the standard 2.6 kernel build environment. # ./extras/build.sh clean Clean driver source directory of all build files (i.e. *.ko, *.o, etc). # ./extras/build.sh new Rebuild the driver sources from scratch. This is essentially a shortcut for: # ./build.sh clean # ./build.sh # ./extras/build.sh install Build and install the driver module files. This command performs the following: 1. Builds the driver .ko files. 2. Copies the .ko files to the appropriate /lib/modules/... directory. 3. Adds the appropriate directive in the modprobe.conf[.local] to remove the qla2xxx_conf module when the qla2xxx modules in unloaded. 4. Updates the newly built qla2xxx_conf.ko module with any previously saved data in /etc/qla2xxx.conf. NOTE: For SLES9 distributions the "modprobe.conf.local" file is used. For RedHat4 distributions the "modprobe.conf" file is used. For Suse Linux Enterprise Server 9 (SLES9) Distribution: You will need to modify the /etc/sysconfig/kernel file to specify which modules will be added during initrd creation. NOTE: Please ensure the conf module is listed before the actual driver module. All "qla" modules should be added to the end of the INITRD_MODULES string. For example: INITRD_MODULES="aic7xxx qla2xxx_conf qla2xxx qla2300" or INITRD_MODULES="aic7xxx qla2xxx_conf qla2xxx qla2400" For RedHat 4 Advanced Server Distribution: You will need to modify the /etc/modprobe.conf file to load the correct device driver modules. Add the following lines to the /etc/modules.conf file depending on the type of adapters you have in your system. alias scsi_hostadapter0 qla2xxx_conf alias scsi_hostadapter1 qla2xxx alias scsi_hostadapter2 qla2400 or alias scsi_hostadapter0 qla2xxx_conf alias scsi_hostadapter1 qla2xxx alias scsi_hostadapter2 qla2300 ======================================================================= 7.0 Load and configure the driver --------------------------------- 7.1 Enable more than 1 scsi device per adapter ---------------------------------------------- Support for multiple LUNs per device is standard in Linux 2.6 kernels. Currently, the maximum number of LUNs that can be scanned for each device is 256. To configure support for multiple LUNs amount other than 256: If the SCSI Mid-Layer is compiled as a module, add the following line to the loadable module configuration file modprobe.conf, found under /etc directory to scan for multiple LUNs(128): options scsi_mod max_luns=128 NOTE: If you have multiple adapters, set max_luns to the largest number of LUNs supported by any one of these adapters. 7.2 Install SANSurfer --------------------- If SANSurfer for Linux is not currently installed you will need to Install it now. SANSurfer and the qlremote agent are needed to configure this device driver and the host adapters for Multi-path failover. See the Qlogic SANSurfer publications for instructions on the installation procedure for SANSurfer and the qlremote agent for Linux. SANSurfer is required for proper multi-path and failover configuration. 7.3 Modify the module load time options --------------------------------------- IMPORTANT: Incorrect changes to these settings may cause driver to malfunction. This device driver has multiple load time options that will need to be added to change driver values without having to recompile the device driver. When connecting to a FAStT Storage Server, you will need to add ql2xsuspendcount=70 and ql2xretrycount=60 to your modprobe.conf.[local] options string as shown later in this section. The two entries are to be added to the file after finishing the Load Balancing procedure. IMPORTANT: Please note that the /etc/modules.conf file has been replaced with /etc/modprobe.conf.local in SLES9 and /etc/modprobe.conf in RH4. Note: For FAStT FC HBA drivers version earlier 7.00.61-fo, the ql2xsuspendcount is set to 40 instead of 70 as indicated. The change was made to accommodate longer delay time of SATA drives. The driver gets its parameters from the command line itself or from modprobe 'option' directive found in the modprobe.conf[.local] file. The parameters are in simple =value format, i.e. ql2xfailover=1. Where is one of the following option parameters: Usage: insmod qla2xxx.ko =value * ql2xfailover - This parameter defines whether failover mode is enable or disable. 0 to disable; 1 to enable. Default: 1. * ql2xsuspendcount - Number of 6-second suspend iterations to perform while a target returns a status. The default is 10 iterations. which is equal to 60 seconds. This value is required to be modified to 40 for the DS4000/FAStT Storage Servers. * max_srbs - Maximum number of simultaneous commands allowed for an HBA. The default value is set to 4096 in the driver source. * extended_error_logging - This parameter defines whether the driver will print verbose logging information. 0 to disable; 1 to enable. Default: 0. * ql2xmaxqdepth - This parameter defines the maximum queue depth reported to Scsi Mid-Level per device. The Queue depth specifies the number of outstanding requests per lun. Default: 32. * ql2xlogintimeout - This parameter defines the Login timeout value in seconds during the initial login. Default: 20 seconds * qlport_down_retry - This parameter defines how long to wait for a port that returns a PORT-DOWN status before returning I/O back to the OS. Default: 0 (use value specified in NVRAM) * ql2xretrycount - This parameter defines the maximum number of Scsi Mid-Level retries allowed per command. Default: 20 (standard mode value), 30 (failover mode value) * displayConfig - This parameter defines whether to display the current configuration. 0 - don't display the configuration; 1 - display the configuration. Default: * Bind - This parameter defines what target persistent binding method to use: 0 - bind by Portname and 1 - bind by PortID. Default: 0 (portname binding) * ConfigRequired - This parameter defines how to bind the devices. 0 - Present all the devices discovered to the OS. 1 - Present only the configured devices (i.e. the device defined in /etc/qla2xxx.conf ) to the OS. Default: 0 * MaxPathsPerDevice - This parameter defines maximum number of paths to a device. Default: 8 (compile time only) * MaxRetriesPerPath - This parameter defines how many retries to perform on the current path before failing over to the next path in the path list. Default: 3 * MaxRetriesPerIo - This parameter defines total retries to do before failing the command and returning to the OS with selection timeout (DID_NO_CONNECT). Default: (MaxRetriesPerPath * MaxPathsPerDevice ) + 1 * QlFailoverNotifyType - This parameter defines type of failover notification mechanism to use when a failover or failback occurs. Certain storage systems require special CDBs to be issued to do failover or failback. Default: 0 (none) * FailbackTime - This parameter defines the delay in seconds before a failback is performed to ensure all paths are available. Default: 5 seconds * RecoveryTime - This parameter defines the recovery time in seconds required before commands can be sent to the restored path. Default: 10 seconds * ql2xautorestore - This parameter enables or disables the logic that restores the previous failed preferred path and/or controller for a given lun. This option toggles the default state. Combine one or more of the following model numbers into an inclusion mask: 0x80 - MSA A/A (auto-restore disabled) 0x20 - HSV111, HSV101, HSV200, HSV210 (auto-restore disabled) 0x10 - DSXXX (auto-restore disabled) 0x04 - HSV110, HSV100 (auto-restore disabled) 0x02 - MSA1000 (auto-restore disabled) 0x01 - XP (auto-restore enabled) Default: 0 * ql2xlbType - This parameter defines the load Balance Method for the driver to use either static or dynamic: 0 (None) - Expose luns on the first active path and make them the preferred path or the first active optimize path and make them the preferred path (storages: MSA A/A and EVA A/A). 1 (Static load balancing) - Distribute and expose the LUNs across the active optimize port(s) or active un-optimize port(s) and HBA(s). 2 (Least outstanding I/O) - send command to the path with the lowest I/O count. 3 (Least Service time) - Send request to the path with the shortest execution time. Default: 0 * ql2xexcludemodel - This parameter excludes device models from being a failover capable target. Combine one or more of the following model numbers into an exclusion mask: 0x80 - MSA A/A 0x20 - HSV111, HSV101, HSV200, HSV210 0x10 - DSXXX 0x04 - HSV110, HSV100 0x02 - MSA1000 0x01 - XP Default: 0 * ql2xtgtemul - Enable/Disable target level grouping emulation. This option is necessary for the GUI to work correctly if the driver is set for lun level grouping of paths by lunid. The following storages uses this method of combining paths: HSV210, DSXXX, HSV110, MSA1000, XP. 1 - Enable target level grouping emulation 0 - Disable target level grouping emulation Default: Enable A comprehensive list of parameters can be found with the following command line: # /sbin/modinfo qla2xxx.ko These values can be changed in the modprobe.conf.local file as part of the options string created by SANSurfer. For example, you could add the following into your options string in the /etc/modprobe.conf.local file. ql2xretrycount=60 and ql2xsuspendcount=70 as shown below. "options qla2xxx ql2xfailover=1 ConfigRequired=1 ql2xretrycount=60 ql2xsuspendcount=70 7.4 Load the Driver Manually using INSMOD or MODPROBE ----------------------------------------------------------- Before loading the driver manually, first build the driver binary from the driver source files as described in sections 6.0. To load the driver directly from the local build directory, type the following in order: # insmod qla2xxx_conf.ko # insmod qla2xxx.ko and either # insmod qla2400.ko (QLA24xx/QLE24xx) oR # insmod qla2300.ko (QLA23xx) To load the driver using modprobe: 1. Install the driver modules (*.ko) files to the appropriate kernel module directory: # ./extras/build.sh install 2. Type the following to load the driver for qla2xxx HBAs: # modprobe -v qla2xxx_conf # modprobe -v qla2400 (qla24XX/qle24XX) OR modprobe -v qla2300 (qla23XX) The modprobe -v qla2400 command will automatically load qla2xxx.ko the component. To unload the driver using modprobe: 1. # modprobe -r qla2400 (qla24XX/qle24XX) OR modprobe -r qla2300 (qla23XX) This will unload qla2400.ko and qla2xxx.ko modules. If there are are additional firmware-loader modules (e.g. qla2300.ko or qla2400.ko, respectively) which depend on qla2xxx.ko, then the qla2xxx.ko will not be unloaded 2. # modprobe -r qla2xxx_conf This will unload qla2xxx_conf.ko Verify that the driver is loaded correctly. # lsmod (This command will display the loaded modules. qla2xxx_conf, qla2xxx and qla2400 should be in the loaded modules listing.) # cd /proc/scsi/qla2xxx # cat 1 (Displays driver information for Adapter 1) 7.5 Loading the Driver using a ramdisk image -------------------------------------------- 1. Linux may detect new adapters during the system boot, if the adapters are configured during the boot process. For Suse Linux Enterprise Server 9 (SLES9) Distribution: You will need to modify the /etc/sysconfig/kernel file to specify which modules will be added during initrd creation. NOTE: Please ensure the conf module is listed before the actual driver module. All "qla" modules should be added to the end of the INITRD_MODULES string. For example: INITRD_MODULES="aic7xxx qla2xxx_conf qla2xxx qla2300" or INITRD_MODULES="aic7xxx qla2xxx_conf qla2xxx qla2400" or INITRD_MODULES=".... qla2xxx_conf qla2xxx qla2300 qla2322 qla6312 qla2400" Notes: qla2xxx_conf: SANsurfer use only qla2xxx: Common module qla2300: For QLA234X qla2322: For QLE236X qla2400: For QLA24XX qla6312: For QLA2XX For RedHat 4 Advanced Server Distribution: You will need to modify the /etc/modprobe.conf file to load the correct device driver modules. Add the following lines to the /etc/modprobe.conf file depending on the type of adapters you have in your system (if not already present). alias scsi_hostadapter0 qla2xxx_conf alias scsi_hostadapter1 qla2xxx alias scsi_hostadapter2 qla2400 or alias scsi_hostadapter0 qla2xxx_conf alias scsi_hostadapter1 qla2xxx alias scsi_hostadapter2 qla2300 2. You will then need to run depmod from the command prompt to update your /lib/modules/KERNEL_VERSION/modules.dep file with the information added in your /etc/modprobe.conf.[local] file. # depmod -a 3. You will now need to load the device driver. To load the driver manually, type the following command: # modprobe -v qla2xxx_conf # modprobe -v qla2400 The modprobe -v qla2400 command will automatically load qla2xxx.ko the component. 4. Type the following command: For RedHat Distribution: # mkinitrd -f NOTE: This step will overwrite the original ramdisk image file if executed within the /boot directory. Specify a unique ramdisk image name to preserve the original ramdisk image. On IA-32 and AMD-64 ------------------- - Copy the newly built file to /boot. On IA-64 -------- - Copy the newly built file to /boot/efi/efi/redhat For SLES9 Distribution: # /sbin/mk_initrd By default, the RAMDISK images created are: /boot/initrd /boot/initrd.suse NOTE: This step will overwrite the original ramdisk image file. To preserve the original ramdisk image specify a unique ramdisk image name as follows: # /sbin/mk_initrd -k -i 5. Configure the boot loader with the new RAMDISK image. For RedHat4: Add "initrd=/boot/" in /etc/grub.conf under one of the kernel entries to use the RAMDISK image. For SLES9: Add "initrd=/boot/" in /boot/grub/menu.lst under one of the kernel entries to use the RAMDISK image. 7. Reboot the system, the qla2xxx and qla2400 driver will be loaded by the ramdisk image at boot time. 8. You will now need to configure your device driver for Multi- path I/O using SANSurfer and the qlremote agent. Refer to the SANSurfer publication and readme.txt for these instructions. 9. After the system is configured for multi-path I/O steps 3 - 7 above need to be repeated so that during system boot the proper driver configuration is loaded. 7.6 Rebuilding the ramdisk image after configuration changes ------------------------------------------------------------ To rebuild your ramdisk image after adding a new LUN or other configuration changes these steps may be followed: 1. To unload the driver using modprobe: 1. # modprobe -r qla2400 This will unload qla2400.ko and qla2xxx.ko modules. 2. # modprobe -r qla2xxx_conf This will unload qla2xxx_conf.ko 2. Delete the options string in /etc/modprobe.conf.[local] that is added by the SANSurfer load-balancing process. This is typically the last line in the /etc/modprobe.conf.[local] file: "options qla2xxx ConfigRequired=1 3. In the terminal windows type: depmod -a 4. Reload the qla2xxx and qla2400 device drivers typing: # modprobe -v qla2xxx_conf # modprobe -v qla2400 The modprobe -v qla2400 command will automatically load qla2xxx.ko the component. 5. Run the SMclient to redistribute the logical volumes. 6. Open a terminal windows and run: qlremote 7. Open another terminal window and run: /usr/SANSurfer 8. Connect to local host, configure LUNs to match the preferred paths shown in IBM DS4000 Storage manager. Running Storage Subsystem->Profile in the Subsystem Management window will display the world-wide port name for each controller. Ensure that the preferred path in the SANSurfer LUN configuration window matches the controller assignment in IBM DS4000 Storage manager. 9. Apply the configuration and exit SANSurfer. 10. Switch to the terminal window that has qlremote running and to stop qlremote. 11. In the terminal windows type: depmod -a 12. Unload the qla2xxx and qla2400 driver and then reload the driver using modprobe so the driver will pull in the new option string that was added by SANSurfer. 13. Type the following command: For RedHat Distribution: # mkinitrd -f NOTE: This step will overwrite the original ramdisk image file if executed within the /boot directory. Specify a unique ramdisk image name to preserve the original ramdisk image. On IA-32 and AMD-64 ------------------- - Copy the newly built file to /boot. On IA-64 -------- - Copy the newly built file to /boot/efi/efi/redhat For SLES9 Distribution: # /sbin/mk_initrd By default, the RAMDISK images created are: /boot/initrd /boot/initrd.suse NOTE: This step will overwrite the original ramdisk image file. To preserve the original ramdisk image specify a unique ramdisk image name as follows: # /sbin/mk_initrd -k -i 14. Reboot the system to the ram disk image just created. ======================================================================= 8.0 Failover Support --------------------- 8.1 How to enable the Failover support in the Driver ----------------------------------------------------- Failover support can be enabled in the qla2xxx driver by loading the driver using the ql2xfailover module parameter: # insmod qla2xxx.ko ql2xfailover=1 ; insmod qla2400.ko Non-failover support can also be enabled in the qla2xxx driver by loading the driver using the ql2xfailover module parameter: # insmod qla2xxx.ko ql2xfailover=0 ; insmod qla2400.ko These values can be changed in the /etc/modprobe.conf.[local] file as part of the options string created by SANSurfer. For example, you could add the following into your options string in the /etc/modprobe.conf.[local] file, ql2xfailover=1 as shown below. "options qla2xxx ql2xfailover=1 ConfigRequired=1 ql2xretrycount=60 ql2xsuspendcount=70 8.2 Configuration Changes Made via SANSurfer -------------------------------------------- 1. LUN Masking For the new LUN masking configuration to take effect, the driver must be reloaded. The following is an example of the sequence of actions to take: - Load the driver: modprobe - Load the qlremote agent. - Start the GUI and connect it to the destination system. - Make LUN masking changes. - Disconnect the host from GUI and stop qlremote agent. - Unload the driver: modprobe -r - Reload the driver: modprobe - Load qlremote agent again. - Start the GUI and connect it to the destination system. Now you should see the updated LUN masking configuration. Please note that when using modprobe to load the driver, the length of the option line specified in /etc/modprobe.conf.local file has a limit of 2K characters. Any longer option line will cause a string overflow error from modprobe. 8.3 Persistent Binding ---------------------- The Persistent Binding information consists of some adapter parameter entries along with some target entries. However, the Linux entries have been shorten to save space on the command line. Currently, there is no limit on the size of the command line when using modprobe. But, if you embedded the driver in the kernel you are using lilo that has a string size limitation. Persistent Binding can be specified in two ways. Manually or using SANSurfer. We recommend using SANSurfer for ease of use. The following is the procedure to manually add persistent binding commands: The driver displays the current configuration when the displayConfig command line option is specified. The persistent binding configuration is found in /var/log/messages file. It prints the configuration information in the format required by the driver. The best way to extract configuration messages is to use grep and direct the output to a file. You need to remove the Linux timestamp at the beginning of each message and combine them together on single line. For example #insmod qla2400.ko displayConfig=1 #grep "scsi-qla" /var/log/messages > /tmp/info.cfg The format of the persistent binding commands is as follows: Device descriptions scsi-qla<#>-adapter-port=; The designated by qla<#>, where the <#> is the adapter instance number. The parameter specifies the FC port name to be used for the adapter. where is the FC port name value in hexa- decimal format. If this entry is not specified in the conf file, the default value is the adapter's port name as saved in the NVRAM. Example: scsi-qla00-adapter-port=210000e08b01158d\; host adapter instance 0 has a portname of 210000e08b01158d scsi-qla<#1>-tgt-<#2>-di-<#3>-node=; This parameter associates the specified with the SCSI target ID value specified by <#2> and a device id value specified by <#3>. where type is the FC node name of the device, and <#2> is the SCSI target ID to be assigned to the device and <#3> is the device unique id. Where <#1> Specifies the adapter instance number <#2> Specifies the SCSI ID of Target <#3> Specifies the path/device id scsi-qla<#1>-tgt-<#2>-di-<#3>-port=; This parameter associates the specified with the SCSI target ID value specified by <#2> and a device id value specified by <#3>. where type is the FC port Where <#1> Specifies the adapter instance number <#2> Specifies the SCSI ID of Target <#3> Specifies the path/device id (always 0 for non-failover) name of the device, and <#2> is the SCSI target ID to be assigned to the device and <#3> is the device unique id. scsi-qla<#1>-tgt-<#2>-di-<#3>-disabled=<256 bit mask>; This parameter associates the specified <256 bit mask> with the SCSI target ID value specified by <#2> and a device id value specified by <#3>. Where <#1> Specifies the adapter instance number <#2> Specifies the SCSI ID of Target <#3> Specifies the path/device id <256 bit mask> msb lsb 000000000000000000000000000000000000000000000000000000000000000F the mask above will make the first four luns, 3, 2, 1, and 0 of a given Target disabled on that target/path. This mask specification is heavily type checked to be a sequence of 64 hex digits. 8.4 Configuration Data ---------------------- 8.4.1 Limitations with /etc/modprobe.conf.local ----------------------------------------------- Due to size constraints inherent in the user-space applications which load kernel modules, the total amount of configuration data that could be passed via modprobe.conf.local by the modprobe application was around 4096 bytes of information (with minor tuning of the modutil package). Of course, as densities of SANs increase, larger configuration spaces are needed to accommodate the information. In general, the following formula can be used to compute an approximate size in bytes of the configuration data needed to store information pertaining to 'M' HBAs and 'N' targets/device paths: 75 + 42*M + 381*N Plugging in values for common configurations returns some sample results: 2 same type HBAs - (75 + 2*42 == 159) ---------------- 1 target - 75+2*42+381*1 = 540 bytes 2 targets - 75+2*42+381*2 = 921 bytes 3 targets - 75+2*42+381*3 = 1302 bytes 4 targets - 75+2*42+381*4 = 1683 bytes 5 targets - 75+2*42+381*5 = 2064 bytes ... 3 same type HBAs (75 + 3*42 == 201) ---------------- 1 target - 75+2*42+381*1 = 582 bytes 2 targets - 75+2*42+381*2 = 963 bytes 3 targets - 75+2*42+381*3 = 1344 bytes 4 targets - 75+2*42+381*4 = 1725 bytes 5 targets - 75+2*42+381*5 = 2086 bytes ... Please note, a target in this case does not always indicate a distinct piece of storage -- it could represent 'n' paths to the same storage, as is the case with failover in a true-cloud configuration. As an example, the configuration data size needed for two storage devices with four paths (via two HBAs) to each storage (4*2 paths) would need approximately 3200 bytes (75 + 42*2 + 381*8) of configuration space. 8.4.2 QLA_OPTS as an alternative -------------------------------- Modutil (namely modprobe) loads 'option' data present in the modprobe.conf.local file by first loading the module, parsing the 'options' directive of the newly loaded module for parameters (parameters are simple key=value directives, i.e. ql2xfailover=1), for each key, scan the memory area where the module was loaded for the location of the 'key' parameter, and finally, writing the key's 'value' directly into the pre-defined memory space. This basic mechanism is similar in nature to the mechanism employed by the QLA_OPTS application, but does not suffer from the relatively small size constraints within the modutil package. There are two important difference between the modutil and QLA_OPTS mechanism: 1) Configuration data is read from a configuration file in /etc/ with a name based on the ISP type: Configuration File Module name ------------------ ----------- /etc/qla2xxx.conf qla2400 2) Option values are written directly (branded) to the module's '.o' binary file. Approximately 300K of configuration space has been pre-allocated within the driver. 8.4.3 Compatibility with SANSurfer ----------------------------------- QLA_OPTS will work seamlessly with updated SANSurfer applications. Originally, when a SANSurfer application would save a configuration the corresponding data would be written to the 'options' section of the modprobe.conf.local file in an form similar to the following: ql2xopts=scsi-qla0-adapter-port=210000e08b000000\;scsi-qla0-tgt-1-di- 0-node=20000020371682e7\;scsi-qla0-tgt-1-di-0-port=21000020371682e7\ ;scsi-qla0-tgt-1-di-0-pid=0000e2\;scsi-qla0-tgt-1-di-0-control=00\; Now, the information is written to the appropriate qla2xxx.conf file in /etc and then branded to the binary file of the currently executing module (qla2300.ko): /lib/modules//kernel/drivers/scsi Where CURRENT_KERNEL_VERSION is the result of the command 'uname -r'. This operation is performed automatically and requires no user intervention. Of course, if the driver was loaded from an initrd image, as before with the modprobe.conf.local interface, the user would be required to rebuild the initrd image after updating a configuration. 8.4.4 Persisting configuration data while upgrading drivers ----------------------------------------------------------- To maintain configuration data during a driver update (e.g. 7.00.61b11 -> 8.01.06), the 'install' directive during the make process will read the proper qla2xxx.conf file and write the previous configuration into the newly built driver binaries. When updating device drivers from earlier versions you will be required to have the latest version of SANSurfer installed and then re-save your failover configuration so that the qla2xxx.conf file can be written out as needed. This is required due to the change in method that the persistent configuration data is read by the device driver. 8.5 Hot Add new LUNs -------------------- This version supports hot adding new luns, etc. Please see text below on how to perform the lun hot add procedure. The latest driver-v8.01.06 has the mechanism which allows the user to force the driver to do re-scan of the devices to allow a new device to be added. This triggers the driver to initiate lun discovery process. To do this from the command line: #echo "scsi-qlascan" > /proc/scsi// (qlogic driver will re-scan) Where can be either one : qla2xxx/qla2400 is the instance number of the HBA. Once that has been done , user then can force the scsi mid layer to do its own scan and build the device table entry for the new device: # echo "scsi add-single-device 0 1 2 3" >/proc/scsi/scsi (scsi mid layer will re-scan) with "0 1 2 3" replaced by your "Host Channel Id Lun". The scanning has to be done in the above mentioned order. First the driver (qla2xxx/qla2400 driver etc) and then the Linux scsi mid layer. ======================================================================= 9.0 Limitations --------------- * In the xSeries server model x260, x366 and x460 configurations, the DS4000 FC 4Gbps single or dual port Fibre Channel Host Bus Adapter (HBA) might cause the servers to intermittently reboot under heavy Fiber Channel I/O. The server RSA II logs might indicate several PERR (PCI Error) machine checks occurred followed by NMI (non-maskable interrupt), which will cause the system to reboot. The problem occurs when the DS4000 FC 4Gbps single or dual port Fibre Channel HBA operates at PCI-X mode 2 speed (266 MHz). The workaround is to operate the adapter in PCI-X 2.0 mode 1 (133 MHz). A new version of server BIOS is required for this work-around. The version of this BIOS package will be 1.10 or later. The xSeries server BIOS can be downloaded at the xSeries server support web site - http://www.pc.ibm.com/support. After the new server BIOS is installed, for each PCI-X Slot, the BIOS workaround will determine which PCI-X 2.0 mode to run using the following options which are set in the Setup Configuration Utility: 1. Setup Option 1 (default) - if adapter is not mode 2 capable, then run in mode designated on the adapter. - if adapter is mode 2 capable and if adapter has a Fiber Channel class code, then run in mode 1 - if adapter is mode 2 capable and if adapter does not have a Fibre Channel class code, then run in mode 2 2. Setup Option 2 - User Selectable PCI-X 2.0 mode 1 3. Setup Option 3 - User Selectable PCI-X 2.0 mode 2 if the adapter is mode 2 capable. The BIOS workaround has minor to no performance impact to the operation of the 4Gbps Fiber Channel HBA's while operating in PCI-X 2.0 mode 1 (133MHz). * Failover adapters must be of the same type. An IBM FAStT FC2-133 Host Adapter and an IBM DS4000 FC 4Gbps Host Bus Adapter cannot be a failover pair. * The qlremote agent must not be loaded when disk I/O's are running. * Lun masking is not functional in this release. * In the Storage Manager Client Storage Partitioning must be enabled with the host type of 'linux' selected and the UTM must be removed from the 'Linux' storage partition. Otherwise the correct drive information is not presented to the Linux operating system. * Every time a change is made to your configuration you will need to generate a new boot image. This includes changes to LUN ownership and adding LUNs to the storage subsystem. * With sequential LUN numbering, if one LUN is missing due to a controller failure the system associates the wrong SCSI device entry with the LUNs after the failed LUN. The result could be that the wrong LUN could be mounted to an incorrect mount point. You should always be aware of this because it can also be caused by adding a LUN out of sequence. You should ensure that the LUNs are number in sequence starting from 0 in storage partitioning. If there is a gap (for example LUN 0, LUN1, LUN3) Linux will quit probing at the gap where LUN 2 should be and all of the following LUNs will be missing. * When updating Firmware or NVSRAM with high disk activity a controller may become unresponsive during the update. IBM recommends that disk I/O be stopped during these code updates. * When adding logical drives to the storage subsystem with the SMclient you will need to unload the device driver and then modprobe the driver to scan the new logical drives. You will then need to reconfigure using SANSurfer and the qlremote agent so these disks will be available for the OS to utilize. After the driver is reconfigured with SANSurfer you will need to repeat section 7.6 in this readme to rebuild your boot image with the new option string created by SANSurfer. * If a configuration change has been made using SANSurfer and you are not able to APPLY or SAVE the configuration you will need to check the options string in your modprobe.conf.local file. On occasion this string may get corrupted and need to be deleted before your new configuration will be saved properly. See section 7.6 above. ======================================================================= 9.1 Proc Filesystem Support --------------------------- The /proc filesystem for the driver can be found in the /proc/scsi/qla2xxx/ directory. This directory contains a file for each IBM FAStT Host Adapter in the system. Each file will present information about the adapter and transfer statistics for each discovered LUN. ======================================================================= 10.0 Driver file Contents ------------------------- qla2xxx-src-v8.01.06.tgz ql2100_fw.c ql6322.c qla_dbg.h qla_inioct.c qla_rscn.c ql2200.c ql6322_fw.c qla_def.h qla_init.c qla_settings.h Kconfig ql2200_fw.c qla2xxx_conf.c qla_devtbl.h qla_inline.h qla_sup.c Makefile ql2300.c qla_32ioctl.c qla_fo.c qla_iocb.c qla_version.h exioct.h ql2300_fw.c qla_32ioctl.h qla_fo.h qla_isr.c qla_xioct.c exioctln.h ql2322.c qla_cfg.c qla_foln.c qla_listops.h qlfo.h extras ql2322_fw.c qla_cfg.h qla_foln.h qla_mbx.c qlfolimits.h inioct.h ql6312.c qla_cfgln.c qla_gbl.h qla_opts.h qlfoln.h ql2100.c ql6312_fw.c qla_dbg.c qla_gs.c qla_os.c revision.notes ======================================================================= 11.0 WEB Sites and Support Phone Number ---------------------------------------- 11.1 IBM System Storage™ Disk Storage Systems Technical Support web site: http://www.ibm.com/servers/storage/support/disk/ 11.2 IBM System Storage™ Marketing Web Site: http://www.ibm.com/servers/storage/disk 11.3 You can receive hardware service through IBM Services or through your IBM reseller, if your reseller is authorized by IBM to provide warranty service. See http://www.ibm.com/planetwide/ for support telephone numbers, or in the U.S. and Canada, call 1-800-IBM-SERV (1- 800-426- 7378). ======================================================================= 12.0 Trademarks and Notices -------------------------- The following terms are trademarks of the IBM Corporation in the United States or other countries or both: IBM System Storage the e-business logo xSeries, pSeries HelpCenter UNIX is a registered of The Open Group in the United States and other countries. Microsoft, Windows, and Windows NT are of Microsoft Corporation in the United States, other countries, or both. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. QLogic and SANsurfer are trademarks of QLogic Corporation. Other company, product, or service names may be trademarks or service marks of others. ======================================================================= 13.0 Disclaimer -------------- 13.1 THIS DOCUMENT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. IBM DISCLAIMS ALL WARRANTIES, WHETHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE AND MERCHANTABILITY WITH RESPECT TO THE INFORMATION IN THIS DOCUMENT. BY FURNISHING THIS DOCUMENT, IBM GRANTS NO LICENSES TO ANY PATENTS OR COPYRIGHTS. 13.2 Note to U.S. Government Users -- Documentation related to restricted rights -- Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corporation. System Storage the e-business logo xSeries, pSeries HelpCenter UNIX is a registered of The Open Group in the United States and other countries. Microsoft, Windows, and Windows NT are of Microsoft Corporation in the United States, other countries, or both. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. QLogic and SANsurfer are trademarks of QLogic Corporation. Other company, product, or service names may be trademarks or service marks of others. ======================================================================= 13.0 Disclaimer -------------- 13.1 THIS DOCUMENT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. IBM DISCLAIMS ALL WARRANTIES, WHETHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE AND MERCHANTABILITY WITH RESPECT TO THE INFORMATION IN THIS DOCUMENT. BY FURNISHING THIS DOCUMENT, IBM GRANTS NO LICENSES TO ANY PATENTS OR COPYRIGHTS. 13.2 Note to U.S. Government Users -- Documentation related to restricted rights -- Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corporation.