This README file contains the latest information about installing the IBM eServer BladeCenter Fibre Channel Expansion Card Device Driver version 8.01.04-fo for Linux on a HS20 or HS40 BladeServer platform Refer to the IBM eServer BladeCenter Fibre Channel Expansion Card Support CD and Installation and User's guide. Refer to the IBM Support Website listed in "Web Sites and Support" section in this ReadMe document for the latest information and updated README file. IMPORTANT: This version of the Linux Fibre Channel HBA device driver should NOT be used if the IBM Linux RDAC for FastT Storage or the Subsystem Device Driver for ESS Storage is used as the multipath failover /failback driver. The Fibre Channel HBA non-failover device driver should be used instead. Products supported: IBM eServer BladeCenter Fibre Channel Expansion Card (IBM Option P/N 13N2203) IBM eServer BladeCenter Fibre Channel Expansion Card (IBM Option P/N 48P7061) IBM eServer BladeCenter SFF Fibre Channel Expansion Card (IBM Option P/N 26K4841) Blades supported: IBM eServer BladeCenter HS20 IBM eServer BladeCenter HS40 Last updated: 05/19/2006 ======================================================================= Contents -------- 1.0 OS Support 2.0 Supported Features 3.0 Release History 4.0 Host Adapter configuration 4.1 Update the Fibre Channel Expansion Card BIOS 4.2 Configure the NOVRAM Setting for the Fibre Channel Expansion Card 5.0 Creating the Driver Diskette 5.1 Driver Disk for Adding Driver to Existing OS 6.0 Building a Driver from the Sources Code 6.1 Install the kernel source and header packages 6.2 Install the (Qlogic) driver source code 6.3 Build a Uni-Processor (UP) version of the device driver 6.4 Building Symmetric Multi-Processor (SMP) Version of the Driver 7.0 Loading and Configuring the driver 7.1 Enable more than 1 SCSI device per adapter 7.2 Install FAStT_MSJ 7.3 Modify the module load time options 7.4 Loading the Driver manually 7.5 Loading the Driver using a ramdisk image 7.6 Rebuilding the ramdisk image after configuration changes 8.0 Failover Support 8.1 How to enable the Failover support in the Driver 8.2 Configuration Changes Made via FAStT MSJ 8.3 Persistent Binding 9.1 Proc Filesystem Support 10.0 WEB Sites and Support Phone Number 11.0 Trademarks and Notices 12.0 Disclaimer ======================================================================= 1.0 OS Support -------------- The 8.01.04-fo device driver was tested with the following Linux versions H/W Platform Distribution ------------ ------------ AMD/EM64T-64bit RH EL 4.0 (U5) AMD/EM64T-64bit RH EL 4.0 (U5) AMD/EM64T-32bit RH EL 4.0 (U6) AMD/EM64T-32bit RH EL 4.0 (U6) AMD-64bit SLES 9 AMD-64bit SLES 9 Intel-x86 RH EL 4.0 (U5) Intel-x86 RH EL 4.0 (U5) Intel-x86 RH EL 4.0 (U6) Intel-x86 RH EL 4.0 (U6) Intel-x86 SLES 9 (SP3) Intel-x86 SLES 9 (SP4) Earlier or later versions of the vendor kernels or generic kernels have not been tested with this device driver version and may not be supported with this release. The Qlogic device driver in the Linux vendor packages have not been tested and are not supported by IBM on the HS20 and HS40 BladeServers This driver package contains the Qlogic driver source code only. The device driver must be compiled from the source code for your specific Linux installation. This readme file makes an assumption that the user has already configured their storage subsystem properly for a Linux server with storage partitioning enabled and configured with the host type set to LINUX. The IBM eServer BladeCenter Fibre Channel Expansion Card uses the qla2300.o device driver. The qla2200.o driver cannot be used on the Fibre Channel Expansion Card. ======================================================================= 2.0 Supported Features ---------------------- 2.1 ISP2300 Features -------------------- * FCAL - direct attach loop * Point-to-point * Fabric support * Initiator mode only * Fault recovery on down loops * Persistent binding * Extended LUN support up to 255 LUNs * FC tape support * Multi-path failover support * Hot Add LUNs ======================================================================= 3.0 Release History ------------------- Please refer to the Release.txt and the Revision Notes file for a complete list of changes. The 8.01.04 driver list of changes are: ======================================================================= 4.0 Host Adapter configuration ------------------------------ 4.1 Update the Fibre Channel Expansion Card Host Bus Adapter BIOS -------------------------------------------------- The adapter BIOS can be updated by booting the server to the BIOS update diskette, available from the IBM Support website. Refer to the install instructions provided with the BIOS update. 4.2 Configure the NOVRAM settings for the Fibre Channel Expansion Card ------------------------------------------------------------------------- All settings, except for the following, should maintain the IBM defaults. - Host Adapter settings Loop reset delay - 8. - Advanced Adapter Settings LUNs per target - 0 Enable Target Reset - Yes Port down retry count - 12 1. As the host boots, press when prompted. 2. After the Fast!Util program loads, the display will depend on whether there are multiple Fibre Channel Expansion Cards installed. If there are multiple Fibre Channel Expansion Cards, a list of addresses occupied by those Host Adapters will appear. Using the arrow keys, select the desired adapter and press ENTER. The Fast!Util Options menu will then appear. For further information refer to the IBM eServer BladeCenter Fibre Channel Expansion Card publications. ======================================================================= 5.0 Download the Fibre Channel Expansion Card driver source code for Linux and IBM FAStT_MSJ for Linux from the IBM Support website. ------------------------------------------------------------------------ Download the Fibre Channel Expansion Card driver source code for Linux from the IBM Support website. If prompted "What would you like to do with this file?" choose "Save this file to disk". Insert a blank formatted diskette and download to the diskette directly. This file will be in the .tgz file compression format. Download the latest version of IBM FAStT_MSJ for Linux from the IBM Support website. If prompted "What would you like to do with this file?" choose "Save this file to disk". This file will need to be saved to your hard disk drive. This file will be in the .tgz file compression format. ======================================================================= 6.0 Building a Driver from the Source Code ------------------------------------------ From the source code you can build the qla2300.o device driver for your UP or SMP systems, and load the driver manually or automatically using a RAMDISK image during system boot time. 6.1 Install the kernel source, header, and ncurses packages ----------------------------------------------------------- 1. Install the kernel-headers and kernel-source RPM files for the supported kernel. These files are available from your Linux vendor or your OS installation CD set. Also, be sure that the ncurses packages have been installed. Kernel installation instructions for different linux versions may vary. Please refer to your linux vendor documentation for procedures to update the kernel source code for your linux version. NOTE: The newest Red Hat kernel headers are incorporated into the kernel source package. If you are using one of these kernels you may not have a kernel-header package to install. Note: For SuSE Linux (SLES8), there is no separate kernel-header RPM file to be installed. # rpm -iv kernel-headers*.rpm # rpm -iv kernel-source*.rpm # rpm -iv ncurses*.rpm # rpm -iv ncurses-devel*.rpm Verify that kernel-headers, kernel-source, and ncurses RPMS are installed. # rpm -qa | grep kernel # rpm -qa | grep ncurses 6.2 Install the qlogic driver source code for the Fibre Channel Expansion Card ------------------------------------------------------------------------------ 1. Using the adapter driver diskette you created in Section 5, copy the qla2x00-v7.01.01-fo-dist.tgz file to /qla2x00-v8.01.04-fo. Follow these steps: # mcopy a:*.tgz . # tar -xzf *.tgz This will create a folder named "qla2x00-v8.01.04-fo" and extract out the following files to the "qla2x00-v8.01.04-fo" folder: drvrinstall Script file to copy driver source files included in the driver source tgz file. qla2x00src-.tgz Compressed binary distribution file for driver sources. This file is the same type of driver source tgz file as the ones used for distributing earlier versions of the QLA2X00 drivers. libinstall Script file to install/setup HBA API library. libremove Script file to remove HBA API library. qlapi--rel.tgz Compressed binary distribution file for API library. 2. Execute the following commands: # cd qla2x00-v8.01.04-fo # ./drvrinstall (this will extract the driver source files to the current directory) 6.3 Build a Uni-Processor (UP) version of the device driver ----------------------------------------------------------- 1. Prepare source headers for a Uni-processor module build by opening a terminal window and changing to the source directory. # cd /usr/src/linux-2.4 (Redhat) # cd /usr/src/linux (SuSE) 2. You may need to edit the linux kernel makefile extraversion entry to match the kernel version that you are running. The newer Red Hat kernels have this entry set to 'custom'. If this entry is not correct you may not be able to modprobe the driver properly. Example 1: EXTRAVERSION = -e.38custom (Change to: EXTRAVERSION = -e.38, or EXTRAVERSION = -e.38smp) Example 2: EXTRAVERSION = -$(CONFIG_RELEASE)-$(CONFIG_CFGNAME) (*In example 2, no changes are necessary) 3. Verify that the correct kernel information is present in the .config file by running make menuconfig at the command prompt. Note: If a .config file is not present in the directory, copy the appropriate example config from the /usr/src/linux-2.4/configs directory to /usr/src/linux-2.4/.config before you issue the "make menuconfig" command. The "make menuconfig" command may not need to be run if you are not changing or rebuilding the kernel configuration. Note: For SLES8 you may need to run 'make cloneconfig' to pull in your current kernel configuration prior to running 'make menuconfig' # make menuconfig Verify that your system configuration is correct for your server type. NOTE: Refer to the server installation documentation to verify that the proper processor family is selected for your server. - select Exit to exit the Main Menu. The system prompts: "Do you wish to save your new kernel configuration?". Select "Yes". The system saves a new config file called ".config" in the current directory. 4. Rebuild the dependencies for the kernel. Type the following command at the command prompt: # make dep NOTE: For Summit kernel, the make oldconfig command must be run instead of make menuconfig and the make modules command must be run after make dep. 5. Change directories back to the /qla2x00-v7.07.04-fo directory that contains the device driver source code. Make the following modifications in the qla_settings.h file as necessary: 1. Failover support can be enabled in the QLA2XXX driver by setting the macro MPIO_SUPPORT to 1 in qla_settings.h file. #define MPIO_SUPPORT 1 Note: The failover distribution package has the above macro enabled by default. 2. Change the MAX_FAILBACKTIME macro from 5 to 240 in the qla_settings.h file. This change was required for host servers that are connected to a FAStT storage server configuration with SATA drives. This change was implemented starting with version 7.00.61 driver. #define MAX_FAILBACKTIME 240 /* (60) Max suspend time before failing back */ Build the driver qla2300.o from the driver source code by typing: For RedHat, to make a UP version of the ISP2300 drivers: # make all install For SuSE linux versions you will add OSVER=linux to the command line to build the device drivers: # make all OSVER=linux install Using the 'install' option with 'make' will copy your new drivers to /lib/modules//kernel/drivers/scsi which will overwrite the default drivers 6. For RedHat, modify the modules.conf to load the correct device driver modules. Add the following lines to the /etc/modules.conf file depending on the type of adapters you have in your system. alias scsi_hostadapter0 qla2300_conf alias scsi_hostadapter1 qla2300 For SuSE SLES8 Distribution: You will need to modify the /etc/sysconfig/kernel file to specify which modules will be added during initrd creation. NOTE: Please ensure the conf module is listed before the actual driver module. For example: INITRD_MODULES="aic7xxx qla2300_conf qla2300" 7. Load the driver manually by typing: # modprobe qla2300 8. Verify that the driver is loaded correctly. # lsmod (This command will display the loaded modules. qla2300_conf and qla2300 should be in the loaded modules listing.) # cd /proc/scsi/qla2x00 # cat 1 (Displays driver information for Adapter 1) 9. Proceed to section 7.0 6.4 Building Symmetric Multi-Processor (SMP) Version of the Driver ------------------------------------------------------------------ 1. Prepare source headers for a Symmetric Multi-Processor (SMP) module build by opening a terminal window and changing to the source directory # cd /usr/src/linux-2.6 (Redhat) # cd /usr/src/linux (SuSE) 2. You may need to edit the linux kernel makefile extraversion entry to match the kernel version that you are running. The newer Red Hat kernels have this entry set to 'custom'. If this entry is not correct you may not be able to modprobe the driver properly. Example 1: EXTRAVERSION = -e.38custom (Change to: EXTRAVERSION = -e.38, or EXTRAVERSION = -e.38smp) Example 2: EXTRAVERSION = -$(CONFIG_RELEASE)-$(CONFIG_CFGNAME) (*In example 2, no changes are necessary) 3. Verify that the correct kernel information is present in the .config file by running make menuconfig at the command prompt. Note: If a .config file is not present in the directory, copy the appropriate example config from the /usr/src/linux-2.6/configs directory to /usr/src/linux-2.6/.config before you issue the "make menuconfig" command. The "make menuconfig" command may not need to be run if you are not changing or rebuilding the kernel configuration. Note: For SLES you may need to run 'make cloneconfig' to pull in your current kernel configuration prior to running 'make menuconfig' # make menuconfig Verify that your system configuration is correct for your server type. NOTE: Refer to the your server installation documentation to verify that the proper processor family is selected for your server. - select Exit to exit the Main Menu. The system prompts: "Do you wish to save your new kernel configuration?". Select "Yes". The system saves a new config file called ".config" in the current directory. 4. Rebuild the dependencies for the kernel. Type the following command at the command prompt: # make dep NOTE: For Summit kernel, the make oldconfig command must be run instead of make menuconfig and the make modules command must be run after make dep. 5. Change directories back to the /qla2x00-v8.01.04-fo directory that contains the device driver source code. Make the following modifications in the qla_settings.h file as necessary 1. Failover support can be enabled in the QLA2XXX driver by setting the macro MPIO_SUPPORT to 1 in qla_settings.h file. #define MPIO_SUPPORT 1 Note: The failover distribution package has the above macro enabled by default. 2. Change the MAX_FAILBACKTIME macro from 5 to 240 in the qla_settings.h file. This change was required for host servers that are connected to a FAStT storage server configuration with SATA drives. This change was implemented starting with version 7.00.61 driver. #define MAX_FAILBACKTIME 240 /* (60) Max suspend time before failing back */ For RedHat, to make an SMP version of the ISP2300 drivers: # make all SMP=1 install For SuSE linux versions you will add OSVER=linux to the command line to build the device drivers: # make all SMP=1 OSVER=linux install Using the 'install' option with 'make' will copy your new drivers to /lib/modules//kernel/drivers/scsi which will overwrite the default drivers 6. For RedHat, modify the modules.conf to load the correct device driver modules. Add the following lines to the /etc/modules.conf file depending on the type of adapters you have in your system. alias scsi_hostadapter0 qla2300_conf alias scsi_hostadapter1 qla2300 For SuSE SLES8 Distribution: You will need to modify the /etc/sysconfig/kernel file to specify while modules will be added during initrd creation. NOTE: Please ensure the conf module is listed before the actual driver module. For example: INITRD_MODULES="aic7xxx qla2300_conf qla2300" 7. Load the driver manually by typing: # modprobe qla2300 8. Verify that the driver is loaded correctly. # lsmod (This command will display the loaded modules. qla2300_conf and qla2300 should be in the loaded modules listing.) # cd /proc/scsi/qla2x00 # cat 1 (Displays driver information for Adapter 1) 9. Proceed to section 7.0 ======================================================================= 7.0 Load and configure the driver --------------------------------- 7.1 Enable more than 1 scsi device per adapter ---------------------------------------------- Support for multiple LUNs can be configured in one of three ways. Currently, the maximum number of LUNs that can be scanned for each device is 128. (1) The kernel must be configured to have multiple LUN support enabled in order for non-zero LUNs to be configured and accessible. Use "make menuconfig" to build a kernel which has the option under SCSI Support enabled to probe all LUNs on SCSI devices. NOTE: If you have multiple adapters, set max_scsi_luns to the largest number of LUNs supported by any one of these adapters. (2) If the SCSI Mid-Layer is compiled in the kernel, the boot loader can be configured to scan for multiple LUNs each time the system boots. For GRUB, perform the following steps: For RedHat Distribution : a) Append the max_scsi_luns parameters to each of the kernel images listed in the /etc/grub.conf file. For example: kernel /vmlinux-2.4.7-10 ro root=/dev/hda2 max_scsi_luns=128 b) Reboot the system. For SuSE Distribution : a) Append the max_scsi_luns parameters to each of the kernel images listed in the /boot/grub/menu.lst file. For example: kernel (hd0,1)/boot/vmlinuz root=/dev/hda2 max_scsi_luns=128 b) Reboot the system. NOTE: Perform these steps for older versions of linux that use LILO: a) Add the following line to each of the kernel images listed in the /etc/lilo.conf file: append="max_scsi_luns=128" b) Run "lilo" and reboot the system. (3) If the SCSI Mid-Layer is compiled as a module, add the following line to the /etc/modules.conf file to scan for multiple LUNs at each boot: options scsi_mod max_scsi_luns=128 Refer to your Linux distribution documentation to verify the steps to rebuild the boot image and ramdisk for your distribution. Section 7.5 of this readme does cover some of the steps required to rebuild your initrd image. 7.2 Install FAStT_MSJ --------------------- If IBM FAStT_MSJ for Linux is not currently installed you will need to Install it now. IBM FAStT_MSJ and the qlremote agent are needed to configure this device driver and the host adapters for Multi-path failover. See the IBM FAStT_MSJ publications for instructions on the installation procedure for IBM FAStT_MSJ and the qlremote agent for Linux. IBM FAStT_MSJ is required for proper multi-path and failover configuration. IBM FastT_MSJ is available from the IBM Support website. 7.3 Modify the module load time options --------------------------------------- IMPORTANT: Incorrect changes to these settings may cause driver to malfunction. This device driver has multiple load time options that will need to be added to change driver values without having to recompile the device driver. When connecting to a FAStT Storage Server, you will need to add ql2xsuspendcount=70 and ql2xretrycount=60 to your modules.conf options string as shown later in this section. The two entries are to be added to the file after finishing the Load Balancing procedure. ql2xmaxqdepth - Maximum queue depth to report for target devices. ql2xretrycount - Maximum number of mid-layer retries allowed for a command. The default value in non-failover mode is 20, for failover mode the default is 30. ql2xsuspendcount - Number of 6-second suspend iterations to perform while a target returns a status. The default is 10 iterations. which is equal to 60 seconds. This value is required to be modified to 40 for the FAStT Storage Servers. displayConfig - If set to '1' then display the configuration used in /etc/modules.conf. max_srbs - Maximum number of simultaneous commands allowed for an HBA. The default value is set to 4096 in the driver source. These values can be changed in the modules.conf file as part of the options string created by IBM FAStT_MSJ. For example, you could add the following into your options string in the /etc/modules.conf file. ql2xretrycount=60 and ql2xsuspendcount=70 as shown below. "options qla2300 ConfigRequired=1 ql2xretrycount=60 ql2xsuspendcount=70 ql2xopts=scsi-qla00- adapter-port=..." 7.4 Loading the Driver manually ------------------------------- - To load the driver directly from the local build directory, type the following: # insmod qla2300.o NOTE: insmod will not load the device driver with the option string created with IBM FAStT_MSJ. - To load the driver using modprobe type the following to load the Driver: # modprobe qla2300 7.5 Loading the Driver using a ramdisk image -------------------------------------------- 1. Linux may detect new adapters during the system boot, if the adapters are configured during the boot process. The following entries should be added to your modules.conf file. For RedHat, verify the modules.conf to load the correct device driver modules. Add the following lines to the /etc/modules.conf file depending on the type of adapters you have in your system. alias scsi_hostadapter0 qla2300_conf alias scsi_hostadapter1 qla2300 For SuSE SLES8 Distribution: You will need to modify the /etc/sysconfig/kernel file to specify while modules will be added during initrd creation. NOTE: Please ensure the conf module is listed before the actual driver module. For example: INITRD_MODULES="aic7xxx qla2300_conf qla2300" 2. You will then need to run depmod from the command prompt to update your /lib/modules/KERNEL_VERSION/modules.dep file with the information added in your /etc/modules.conf file. # depmod -a 3. You will now need to load the device driver. To load the driver manually, type the following command: # modprobe qla2300 4. Type the following command: For RedHat Distribution: # mkinitrd -f NOTE: This step will overwrite the original ramdisk image file if executed within the /boot directory. Specify a unique ramdisk image name to preserve the original ramdisk image. - Copy the newly built file to /boot. For SuSE Distribution: # /sbin/mk_initrd By default, the RAMDISK images created are: /boot/initrd /boot/initrd.suse NOTE: This step will overwrite the original ramdisk image file. To preserve the original ramdisk image specify a unique ramdisk image name as follows: # /sbin/mk_initrd -k -i 5. Configure the boot loader with the new RAMDISK image. For GRUB: Add "initrd=/boot/" in /etc/grub.conf under one of the kernel entries to use the RAMDISK image. For older versions of linux that use LILO: Add "initrd=/boot/" in /etc/lilo.conf under one of the kernel entries to use the RAMDISK image. Run "lilo" and reboot system. Select the kernel with the new RAMDISK image to come up. 7. Reboot the system, the qla2300 driver will be loaded by the ramdisk image at boot time. 8. You will now need to configure your device driver for Multi- path I/O using FAStT_MSJ and the qlremote agent. Refer to the FAStT_MSJ publication and readme.txt for these instructions. 9. After the system is configured for multi-path I/O steps 3 - 7 above need to be repeated so that during system boot the proper driver configuration is loaded. 7.6 Rebuilding the ramdisk image after configuration changes ------------------------------------------------------------ To rebuild your ramdisk image after adding a new LUN or other configuration changes these steps may be followed: 1. Unload the qla2300 driver module by opening a terminal window and typing: # modprobe -r qla2300 2. Delete the options string in /etc/modules.conf that is added by the FAStT MSJ load-balancing process. This is typically the last line in the /etc/modules.conf file: "options qla2300 ConfigRequired=1 ql2xopts=scsi-qla00- adapter- port=..." 3. In the terminal windows type: depmod -a 4. Reload the qla2300 device drivers typing: # modprobe qla2300 5. Run the SMclient to redistribute the logical volumes. 6. Open a terminal windows and run: qlremote 7. Open another terminal window and run: /usr/FAStT_MSJ 8. Connect to local host, configure LUNs to match the preferred paths shown in IBM FAStT Storage manager. Running Storage Subsystem->Profile in the Subsystem Management window will display the world-wide port name for each controller. Ensure that the preferred path in the FAStT_MSJ LUN configuration window matches the controller assignment in IBM FAStT Storage manager. 9. Apply the configuration and exit MSJ. 10. Switch to the terminal window that has qlremote running and to stop qlremote. 11. In the terminal windows type: depmod -a 12. Unload the qla2300 driver and then reload the driver using modprobe so the driver will pull in the new option string that was added by FAStT_MSJ. 13. Type the following command: For RedHat Distribution: # mkinitrd -f NOTE: This step will overwrite the original ramdisk image file if executed within the /boot directory. Specify a unique ramdisk image name to preserve the original ramdisk image. - Copy the newly built file to /boot. For SuSE Distribution: # /sbin/mk_initrd By default, the RAMDISK images created are: /boot/initrd /boot/initrd.suse NOTE: This step will overwrite the original ramdisk image file. To preserve the original ramdisk image specify a unique ramdisk image name as follows: # /sbin/mk_initrd -k -i 14. If you are using lilo, in a terminal window type: /sbin/lilo Otherwise you will not be able to boot your new ramdisk image. 15. Reboot the system to the ram disk image just created. ======================================================================= 8.0 Failover Support --------------------- 8.1 How to enable the Failover support in the Driver ----------------------------------------------------- Failover support can be enabled in the QLA2XXX driver by enabling the macro MPIO_SUPPORT in qla_settings.h file. #define MPIO_SUPPORT 1 Note: The failover distribution package has the above macro enabled by default. 8.2 Configuration Changes Made via FAStT MSJ -------------------------------------------- 1. LUN Masking For the new LUN masking configuration to take effect, the driver must be reloaded. The following is an example of the sequence of actions to take: - Load the driver: modprobe - Load the qlremote agent. - Start the GUI and connect it to the destination system. - Make LUN masking changes. - Disconnect the host from GUI and stop qlremote agent. - Unload the driver: modprobe -r - Reload the driver: modprobe - Load qlremote agent again. - Start the GUI and connect it to the destination system. Now you should see the updated LUN masking configuration. Please note that when using modprobe to load the driver, the length of the option line specified in /etc/modules.conf file has a limit of 2K characters. Any longer option line will cause a string overflow error from modprobe. 8.3 Persistent Binding ---------------------- The Persistent Binding information consists of some adapter parameter entries along with some target entries. However, the Linux entries have been shorten to save space on the command line. Currently, there is no limit on the size of the command line when using modprobe. But, if you embedded the driver in the kernel you are using lilo that has a string size limitation. Persistent Binding can be specified in two ways. Manually or using FAStT MSJ. We recommend using FAStT MSJ for ease of use. The following is the procedure to manually add persistent binding commands: The driver displays the current configuration when the displayConfig command line option is specified. The persistent binding configuration is found in /var/log/messages file. It prints the configuration information in the format required by the driver. The best way to extract configuration messages is to use grep and direct the output to a file. You need to remove the Linux timestamp at the beginning of each message and combine them together on single line. For example #insmod qla2300 displayConfig=1 #grep "scsi-qla" /var/log/messages > /tmp/info.cfg The format of the persistent binding commands is as follows: Device descriptions scsi-qla<#>-adapter-port=; The designated by qla<#>, where the <#> is the adapter instance number. The parameter specifies the FC port name to be used for the adapter. where is the FC port name value in hexa- decimal format. If this entry is not specified in the conf file, the default value is the adapter's port name as saved in the NVRAM. Example: scsi-qla00-adapter-port=210000e08b01158d\; host adapter instance 0 has a portname of 210000e08b01158d scsi-qla<#1>-tgt-<#2>-di-<#3>-node=; This parameter associates the specified with the SCSI target ID value specified by <#2> and a device id value specified by <#3>. where type is the FC node name of the device, and <#2> is the SCSI target ID to be assigned to the device and <#3> is the device unique id. Where <#1> Specifies the adapter instance number <#2> Specifies the SCSI ID of Target <#3> Specifies the path/device id scsi-qla<#1>-tgt-<#2>-di-<#3>-port=; This parameter associates the specified with the SCSI target ID value specified by <#2> and a device id value specified by <#3>. where type is the FC port Where <#1> Specifies the adapter instance number <#2> Specifies the SCSI ID of Target <#3> Specifies the path/device id (always 0 for non-failover) name of the device, and <#2> is the SCSI target ID to be assigned to the device and <#3> is the device unique id. scsi-qla<#1>-tgt-<#2>-di-<#3>-disabled=<256 bit mask>; This parameter associates the specified <256 bit mask> with the SCSI target ID value specified by <#2> and a device id value specified by <#3>. Where <#1> Specifies the adapter instance number <#2> Specifies the SCSI ID of Target <#3> Specifies the path/device id <256 bit mask> msb lsb 000000000000000000000000000000000000000000000000000000000000000F the mask above will make the first four luns, 3, 2, 1, and 0 of a given Target disabled on that target/path. This mask specification is heavily type checked to be a sequence of 64 hex digits. 8.4 Configuration Data ---------------------- 8.4.1 Limitations with /etc/modules.conf ---------------------------------------- Due to size constraints inherent in the user-space applications which load kernel modules, the total amount of configuration data that could be passed via modules.conf by the modprobe application was around 4096 bytes of information (with minor tuning of the modutil package). Of course, as densities of SANs increase, larger configuration spaces are needed to accommodate the information. In general, the following formula can be used to compute an approximate size in bytes of the configuration data needed to store information pertaining to 'M' HBAs and 'N' targets/device paths: 75 + 42*M + 381*N Plugging in values for common configurations returns some sample results: 2 same type HBAs - (75 + 2*42 == 159) ---------------- 1 target - 75+2*42+381*1 = 540 bytes 2 targets - 75+2*42+381*2 = 921 bytes 3 targets - 75+2*42+381*3 = 1302 bytes 4 targets - 75+2*42+381*4 = 1683 bytes 5 targets - 75+2*42+381*5 = 2064 bytes ... 3 same type HBAs (75 + 3*42 == 201) ---------------- 1 target - 75+2*42+381*1 = 582 bytes 2 targets - 75+2*42+381*2 = 963 bytes 3 targets - 75+2*42+381*3 = 1344 bytes 4 targets - 75+2*42+381*4 = 1725 bytes 5 targets - 75+2*42+381*5 = 2086 bytes ... Please note, a target in this case does not always indicate a distinct piece of storage -- it could represent 'n' paths to the same storage, as is the case with failover in a true-cloud configuration. As an example, the configuration data size needed for two storage devices with four paths (via two HBAs) to each storage (4*2 paths) would need approximately 3200 bytes (75 + 42*2 + 381*8) of configuration space. 8.4.2 QLA_OPTS as an alternative -------------------------------- Modutil (namely modprobe) loads 'option' data present in the modules.conf file by first loading the module, parsing the 'options' directive of the newly loaded module for parameters (parameters are simple key=value directives, i.e. ql2xfailover=1), for each key, scan the memory area where the module was loaded for the location of the 'key' parameter, and finally, writing the key's 'value' directly into the pre-defined memory space. This basic mechanism is similar in nature to the mechanism employed by the QLA_OPTS application, but does not suffer from the relatively small size constraints within the modutil package. There are two important difference between the modutil and QLA_OPTS mechanism: 1) Configuration data is read from a configuration file in /etc/ with a name based on the ISP type: Configuration File Module name ------------------ ----------- /etc/qla2300.conf qla2300 2) Option values are written directly (branded) to the module's '.o' binary file. Approximately 300K of configuration space has been pre-allocated within the driver. 8.4.3 Compatibility with FAStT_MSJ (FAStT Management Suite Java) ---------------------------------------------------------------- QLA_OPTS will work seamlessly with updated FAStT_MSJ applications. Originally, when a FAStT_MSJ application would save a configuration the corresponding data would be written to the 'options' section of the modules.conf file in an form similar to the following: ql2xopts=scsi-qla0-adapter-port=210000e08b000000\;scsi-qla0-tgt-1-di- 0-node=20000020371682e7\;scsi-qla0-tgt-1-di-0-port=21000020371682e7\ ;scsi-qla0-tgt-1-di-0-pid=0000e2\;scsi-qla0-tgt-1-di-0-control=00\; Now, the information is written to the appropriate qla2300.conf file in /etc and then branded to the binary file of the currently executing module (qla2300.o): /lib/modules//kernel/drivers/scsi Where CURRENT_KERNEL_VERSION is the the result of the command 'uname -r'. This operation is performed automatically and requires no user intervention. Of course, if the driver was loaded from an initrd image, as before with the modules.conf interface, the user would be required to rebuild the initrd image after updating a configuration. 8.4.4 Persisting configuration data while upgrading drivers ----------------------------------------------------------- To maintain configuration data during a driver update (e.g. 7.00.61b11 -> 7.04.04-fo), the 'install' directive during the make process will read the proper qla2300.conf file and write the previous configuration into the newly built driver binaries. When updating device drivers from earlier versions you will be required to have the latest version of FAStT_MSJ installed and then re-save your failover configuration so that the qla2300.conf file can be written out as needed. This is required due to the change in method that the persistent configuration data is read by the device driver. 8.5 Hot Add new LUNs -------------------- This version supports hot adding new luns, etc. Please see text below on how to perform the lun hot add procedure. The latest driver-v7.04.04-fo has the mechanism which allows the user to force the driver to do re-scan of the devices to allow a new device to be added. This triggers the driver to initiate lun discovery process. To do this from the command line: #echo "scsi-qlascan" > /proc/scsi// (qlogic driver will re-scan) Where can be either one : qla2300 is the instance number of the HBA. Once that has been done , user then can force the scsi mid layer to do its own scan and build the device table entry for the new device: # echo "scsi add-single-device 0 1 2 3" >/proc/scsi/scsi (scsi mid layer will re-scan) with "0 1 2 3" replaced by your "Host Channel Id Lun". The scanning has to be done in the above mentioned order. First the driver (qla2300 driver etc) and then the Linux scsi mid layer. ======================================================================= 9.0 Limitations --------------- * The qlremote agent must not be loaded when disk I/O's are running. * Lun masking is not functional in this release. * Red Hat Advanced Server 2.1 has a disk device size limitation of 1023.999 GB and a maximum of 128 devices. * SuSE Linux Enterprise Server 8.0, powered by United Linux 1.0 has a disk device size limitation of 2035.997 GB and a maximum of 256 devices. * For FastT Storage Devices, the Storage Manager Client Storage Partitioning must be enabled with the host type of 'linux' selected and the UTM must be removed from the 'Linux' storage partition. Otherwise the correct drive information is not presented to the Linux operating system. * Every time a change is made to your configuration you will need to generate a new boot image and launch lilo. This includes changes to LUN ownership and adding LUNs to the storage subsystem. * With sequential LUN numbering, if one LUN is missing due to a controller failure the system associates the wrong SCSI device entry with the LUNs after the failed LUN. The result could be that the wrong LUN could be mounted to an incorrect mount point. You should always be aware of this because it can also be caused by adding a LUN out of sequence. You should ensure that the LUNs are number in sequence starting from 0 in storage partitioning. If there is a gap (for example LUN 0, LUN1, LUN3) Linux will quit probing at the gap where LUN 2 should be and all of the following LUNs will be missing. * When updating Firmware or NVSRAM with high disk activity a controller may become unresponsive during the update. IBM recommends that disk I/O be stopped during these code updates. * When adding logical drives to the storage subsystem with the SMclient you will need to unload the device driver and then modprobe the driver to scan the new logical drives. You will then need to reconfigure using FAStT_MSJ and the qlremote agent so these disks will be available for the OS to utilize. After the driver is reconfigured with FAStT_MSJ you will need to repeat section 6.6 in this readme to rebuild your boot image with the new option string created by FAStT_MSJ. * If a configuration change has been made using FAStT_MSJ and you are not able to APPLY or SAVE the configuration you will need to check the options string in your modules.conf file. On occasion this string may get corrupted and need to be deleted before your new configuration will be saved properly. See section 6.6 above. ======================================================================= 9.1 Proc Filesystem Support --------------------------- The /proc filesystem for the driver can be found in the /proc/scsi/qla2300/ directory. This directory contains a file for each Fibre Channel Expansion Card in the system. Each file will present information about the adapter and transfer statistics for each discovered LUN. ======================================================================= 10.0 WEB Sites and Support Phone Number --------------------------------------- IBM Support Web Site: http://www.ibm.com/pc/support IBM eServer xSeries BladeCenter Marketing Web Site: http://www.ibm.com/servers/eserver/bladecenter If you have any questions about this update, or problem applying the update go to the following Help Center World Telephone Numbers URL: http://www.ibm.com/planetwide ======================================================================= 11.0 Trademarks and Notices --------------------------- The following terms are trademarks of the IBM Corporation in the United States or other countries or both: IBM eServer BladeCenter UNIX is a registered trademark of The Open Group in the United States and other countries. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Linux is a registered trademark of Linus Torvalds. Other company, product, and service names may be trademarks or service marks of others. ======================================================================= 12.0 Disclaimer --------------- THIS DOCUMENT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. IBM DISCLAIMS ALL WARRANTIES, WHETHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE AND MERCHANTABILITY WITH RESPECT TO THE INFORMATION IN THIS DOCUMENT. BY FURNISHING THIS DOCUMENT, IBM GRANTS NO LICENSES TO ANY PATENTS OR COPYRIGHTS. Note to U.S. Government Users -- Documentation related to restricted rights -- Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corporation. IBM Corporation in the United States or other countries or both: IBM eServer BladeCenter UNIX is a registered trademark of The Open Group in the United States and other countries. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Linux is a registered trademark of Linus Torvalds. Other company, product, and service names may be trademarks or service marks of others. ======================================================================= 12.0 Disclaimer --------------- THIS DOCUMENT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. IBM DISCLAIMS ALL WARRANTIES, WHETHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE AND MERCHANTABILITY WITH RESPECT TO THE INFORMATION IN THIS DOCUMENT. BY FURNISHING THIS DOCUMENT, IBM GRANTS NO LICENSES TO ANY PATENTS OR COPYRIGHTS. Note to U.S. Government Users -- Documentation related to restricted rights -- Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corporation. ntract with IBM Corporation.