IBM PCI Hot-Plug Controller Driver Readme v1.4a CONTENTS --------- 1.0 Overview 2.0 Change History 3.0 Installation and Setup Instructions 4.0 Performing Command-Line Hotplug PCI Operations 5.0 WEB Sites and Support Phone Number 6.0 Trademarks and Notices 7.0 Disclaimer 1.0 Overview ------------ 1.1 General Information ----------------------- The IBM Hot Plug PCI (Peripheral Component Interconnect) driver is fully compliant with the PCI Hot-Plug Specification, version 2.2. Each IBM Hot Plug PCI slot has an interlock switch and two LEDs. One LED will be illuminated when power is applied to the slot, and the other LED is turned on when the hot plug slot requires attention. Slots without the interlock switch and LEDs are NOT hot plug slots. All hot plug operations must be controlled either through a command line interface supplied by the Linux kernel or using the Graphical User Interface discussed in section 5, below. IN NO CASE SHOULD PHYSICAL HOT PLUG OPERATIONS BE DONE WITHOUT FIRST REMOVING POWER FROM THE SLOT. IF A SLOT DOES NOT HAVE A LATCH, DO NOT REMOVE THE ADAPTER. DOING SO CAN CAUSE SERIOUS DAMAGE TO THE SYSTEM AND ADAPTER. The order of events in any hot-plug operation are: - Load the kernel PCI hot plug core driver. - Mount the PCI hot plug file system. - Load the IBM PCI hot plug controller driver. - To hot-add an adapter, open the latch, insert the adapter, add any necessary cabling, close the latch. - At the command prompt, echo the requested values into the hot plug file system to power on the desired PCI slot. - When slot power is turned ON, configure the adapter and load any required adapter device drivers. - To hot-remove an adapter: power off the slot, open the latch, remove the adapter and any cabling, close latch. NOTE: The power to a slot cannot be turned ON until the latch on the slot is closed. If no adapter is present in the hot plug slot, then the position of the latch (opened or closed) is not important. 1.2 Limitations --------------- 1.2.1 This limitation does not apply to Hot Plug driver versions higher than 0.6 or to RPM version 1.2. However, the limitation does exist if the Remote Supervisor Adapter hardware and device driver are not installed in the system. Cannot load the IBM PCI hot plug controller driver multiple times within same boot session. If the hot plug driver is loaded and adapters are hot-added or hot-removed, then unloading and reloading the hot plug driver may cause erroneous behavior of the server. 1.2.2 You need to be logged on as a root user in order to be able to operate on the slots from the command-line interface. 1.2.3 On the xSeries 440, if a latch to a hot plug slot was open during the server boot process, then hot-adding an adapter card in that slot will return the following error after about 1 minute: "setting bus speed failed. was not able to set the bus." To fix the problem, open the latch, wait 2 seconds, close the latch, and send the command again. 1.2.4 This driver Will not support onboard PCI-PCI bridges. Additionally, only adapter cards with a single bus behind its onboard PCI bridge are supported. 1.2.5 This driver does not support hot-adding or hot-removing a nested bridge device or a bridge by itself or any graphical devices. 1.2.6 This driver will not function properly if there is no Advanced Programmable Interrupt Controller (APIC) in the system. 1.2.7 Opening the latch while the adapter card is active may cause the system to reboot or hang. Under no circumstances should the latch be opened while power is applied to an active PCI slot. 1.2.8 Multi-function PCI devices are supported as long as the multi- function capability is not provided with or through a PCI-to-PCI bridge. 1.2.9 Devices that are not PCI v2.2 compliant or devices that do not implement the PCI presence detection pins are not supported. 1.2.10 Non-PCI devices are not hot pluggable. 1.3 Dependencies: All files that are required by the hot plug controller driver are provided in the standard distribution system and included RPM: ibmphp-'VERSION_NUMBER'-[OS Version].rpm NOTE: If you want to be able to load/unload the IBM PCI Hot-Plug Controller Driver, make sure you install IBM Remote Supervisor Adapter and its driver before you unload/reload the hot plug driver. 2.0 Change History ------------------ Version 0.6-1.3 September, 2003 Fixed the incorrect bus indexes when supporting more than 4 buses per hot plug controller. Version 0.6-1.2A November, 2002 Added support for one Hot-Plug controller to be able to support more than 4 buses Version 0.6-1.2A October, 2002 - 0.6-1.2A indicates this is from RPM version 1.2A, since the driver in the kernel will look different than in the RPM. - For SuSE 8.0, the pci_hotplug module had to change and the Man pages were modified to reflect these changes. Version 0.6 July, 2002 - Added PCI support for Hot-Plug controller - Changed pci_hotplug core to report more information on SuSE kernels. - Fixed a polling problem - Added Man pages - Added functionality to store information into NVRAM - Changed the slot names to be [chassis#/rxe#]slot# Version 0.5 June, 2002 - Added support for x-Series 235 - Release of binary RPM for Red Hat 7.3 Version 0.2 Changes March/April, 2002 - Added bus speed limitations - Fixed a polling problem - Added support for RXE100 - Added support for SuSE 8.0 Version 0.1 Changes January 14, 2002 - Made formatting changes to comply with the Linux community. - Updated the readme file to remove incorrect information as well as to add additional instructions and clarifications. 3.0 Installation and Setup Instructions ----------------------------------------- NOTE: Installing the IBM Hot Plug Driver requires that kernel source be installed on your xSeries server. You can get the source code by finding the correct RPM kernel-source-'VERSION-NUMBER'.rpm. Also make sure the source code matches your kernel version. 3.1 Kernel configuration instructions ------------------------------------- 3.1.1 SuSE 8.0/SuSE SLES 8.0 ---------------------------- If SuSE 8.0 Professional or SuSE SLES 8.0 is installed on IBM xSeries server models x440, x445, x235, x365, x360 or x255, then the kernel option 64GB-SMP (Symmetric MultiProcessor) should be selected before installing IBM Hot Plug Driver (ibmphp) RPM. Determine the kernel version and install the matching kernel source files, if they are not already installed. Use Yast to install the kernel source files that match the kernel version obtained above. Configure the kernel sources by executing the following commands: cd /lib/modules/`uname -r`/build make clean cp -f /boot/vmlinuz.version.h include/linux/version.h cp -f /boot/vmlinuz.config .config make cloneconfig make dep 3.1.2 For Red Hat 2.1 Advanced Server ------------------------------------- On xSeries server models x440, x445 and x365, make sure you are running the summit kernel. Make sure that kernel-headers and kernel-source rpm packages are present and that they match your kernel version number. In other words, make sure kernel-headers-VERSION_NUMBER and kernel-source-VERSION-NUMBER have the same VERSION-NUMBER as your currently running kernel. If you are running on the summit kernel, type the following for the kernel version. uname -r You will also need install and configure the source code for the summit kernel: 1. Install the kernel-.src.rpm from RedHat Advanced Server 2.1 CD #4 Ex, rpm -ivh kernel-2.4.9-e.3.src.rpm 2. cd /lib/modules/`uname -r`/build 3. patch -p1 < /usr/src/redhat/SOURCES/linux-2.4.9-summit.patch NOTE: If you have done step #3 previously, you do not need not do it again. 4. make mrproper 5. cp configs/kernel-2.4.9-i686-summit.config .config 6. Go to the directory where ibmphp RPM is stored. Type the following to extract a file: rpm2cpio ibmphp-src-redhat-'VERSION_NUMBER'-2.1AS.rpm | cpio -ivd ./usr/src/redhat/SOURCES/Makefile.script Make sure your bash version is > 2.0. You can verify that by typing bash --version at the prompt. Type the following to launch the script: ./usr/src/redhat/SOURCES/Makefile.script NOTE: This script modifies the Makefile in the /lib/modules/`uname -r`/build directory to match your running kernel. The command above (rpm2cpio) created a file tree in the directory where your RPM is contained. To remove it, type command: rm -rf ./usr Make sure you are in the same directory you were in when you issued rpm2cpio and you are not in the / directory. 7. cd /lib/modules/`uname -r`/build 8. make menuconfig 9. Choose and save configuration file. 10. Execute the following command: make dep 3.1.3 For RedHat 7.3 -------------------- Install RedHat 7.3 and latest kernel errata. Make sure you install kernel-source and that it matches your kernel version number. Once you have the appropriate kernel sources installed, you will need to configure them. Execute the following commands: 1. cd /lib/modules/`uname -r`/build 2. make mrproper 3. ls configs 4. Copy the configuration file with the following command. NOTE: Combination of `uname -r` and `uname -m` will help you decide which configuration file you should copy. cp configs/ .config 5. Go to the directory where ibmphp RPM is stored and extract the file: rpm2cpio ibmphp-src-redhat-'VERSION_NUMBER'-7.3.rpm | cpio -ivd ./usr/src/redhat/SOURCES/Makefile.script 6. Make sure your bash version is > 2.0 with the following command: bash --version 7. Type the following to launch the script: ./usr/src/redhat/SOURCES/Makefile.script NOTE: This script modifies the Makefile in the /lib/modules/`uname -r`/build directory to match your running kernel. The command above (rpm2cpio) created a file tree in the directory where your RPM is contained. To remove it, type command: rm -rf ./usr Make sure you are in the same directory you were in when you issued rpm2cpio and you are not in the / directory. 8. cd /lib/modules/`uname -r`/build 9. make oldconfig 10. Execute the following command: make dep 3.2 Driver Install/Load Instructions ------------------------------------ NOTE: If this is an upgrade to an existing IBM PCI hot plug driver, then the previously installed hot plug driver packages must first be unloaded and removed. Unload the old hot plug driver by issuing the following command: /sbin/ibmphpdown Then remove the old hot plug driver with the following command: rpm -e ibmphp To install the IBM Hot Plug driver, go to the directory where ibmphp rpm package is stored and enter the following command from a command prompt: rpm -ivh ibmphp-'VERSION_NUMBER'-[8.0/sles8.0/2.1AS/7.3].rpm This command will compile the driver source, put the binary object files in the /lib/modules/... directory and load the driver. The RPM will configure the necessary files to load the driver at boot up. A comment is also added to /etc/fstab for mounting (if needed). The default hot plug directory is /hotplug and finally, the RPM installs hot plug driver man pages that you can query by typing: man ibmphp 3.2.1 Load the Hot Plug Driver Manually --------------------------------------- Type the following command to load the IBM PCI hot plug driver, /sbin/ibmphpup 3.2.2 Driver Errors ------------------- To see errors due to improper driver operation, check /var/log/messages or type "dmesg" at the command prompt. If there is an unexpected failure, it would be worthwhile to save the driver debug information on a diskette for debugging purposes. The following steps describe this operation. - Power down the machine, reboot, and reload the IBM PCI hot plug driver with the following debug option. /sbin/ibmphpup --debug - Place a diskette in the floppy drive execute the following commands to save the error logs for debug purposes. In RedHat: mount /mnt/floppy cp /var/log/messages /mnt/floppy/messages.txt dmesg >& /mnt/floppy/dmesg.txt cp /var/log/debug /mnt/floppy/debug.txt umount /mnt/floppy In SuSE: mount /media/floppy cp /var/log/messages /media/floppy/messages.txt dmesg >& /media/floppy/dmesg.txt cp /var/log/debug /media/floppy/debug.txt umount /media/floppy 3.3 Driver Unload/UnInstall Instructions ---------------------------------------- - To unload the hot plug driver, enter the following command: /sbin/ibmphpdown - To remove the hot plug driver, enter the following command: rpm -e ibmphp 3.4 Driver Debug Option ----------------------- You can add the ability to use the separate debug option by editing the /etc/syslog.conf file by adding the following line at the end of file: "*.debug /var/log/debug" For this change to take effect, type the following commands: touch /var/log/debug In RedHat: service syslog restart In SuSE: rcsyslog restart NOTE: Section 3.4 is only required prior to the initial use of the driver. This section can be skipped after the first installation and compilation. 4.0 Performing Command-Line Hot Plug PCI Operations -------------------------------------------------- This section is in the Man pages that ship with the Linux operating systems this driver supports. If you want to read the Man pages describing the command-line interface for hot plug PCI operations, enter the following command: man ibmphp 4.1 Required steps in Hot-Plug Operations ----------------------------------------- Make sure you are logged on as root before issuing any hot plug commands. NOTE: Opening an adapter latch will turn off the power to the slot. If a device driver was loaded for that slot, the system will most likely hang, requiring the system to be rebooted. 4.1.1 Removing an adapter ------------------------- 1. Verify that the pci_hotplug and ibmphp drivers are loaded. This can be done by issuing the "lsmod" command. NOTE: On RedHat systems you will not see pci_hotplug listed as a result of lsmod command. However, if ibmphp loaded with no errors, and /hotplug filesystem is mounted (see step #2 below) is a good indication pci_hotplug driver is loaded. 2. Verify that the /hotplug file system is mounted. This can be done by issuing the "mount" command. 3. Issue the power off command to the desired PCI hot plug capable slot. "echo 0 > /hotplug//power" Ex. "echo 0 > /hotplug/2/power" to turn power off to slot 2. NOTE: with RPM version 1.2A and later, slot number is now chassis_#_slot_#_bus_# if the slot is located in the server, and rxe_#_slot_#_bus_# if the slot is physically located in the PCI expansion box (RXE100). If you are using the Graphical User Interface, you will see bus # only in the 'Detailed slot information' tab. The bus # is the bus id number on which the slot resides. 4. Verify that the power LED for the slot is OFF. If the LED is still ON and the attention LED is OFF, go back to the command line and verify the proper command was issued. NOTE: NEVER REMOVE AN ADAPTER FROM A SLOT WITH THE SLOT POWER STILL ON. THIS COULD RESULT IN A SYSTEM HANG AND/OR SERIOUS DAMAGE TO THE ADAPTER CARD AND/OR SYSTEM UNIT. 5. Open the latch and remove the adapter from the slot. 6. Close latch after adapter has been removed. 4.1.2 Adding a new adapter to an empty slot ------------------------------------------- 1. Verify that the pci_hotplug and ibmphp drivers are loaded. This can be done by issuing the "lsmod" command. NOTE: On RedHat, the pci_hotplug module is statically linked, which means lsmod will not show it. However, if /hotplug file system is mounted, and ibmphp driver loaded with no errors, this is a good indicator that pci_hotlug is loaded. 2. Verify that the /hotplug file system is mounted. This can be done by issuing the "mount" command. 3. Open the latch, place the new adapter in the desired slot and close the latch. 4. Attach any necessary cables that the newly inserted adapter may require. 5. Issue the following power on command to the desired PCI hot plug capable slot where the new adapter will be inserted. "echo 1 > /hotplug//power" Example: "echo 1 > /hotplug/chassis_1_slot_2_bus10/power" will turn power on to slot 2. 6. Verify that the power LED for the selected slot is now on. NOTE: If any error occurs, the IBM PCI hotplug controller driver will turn the attention LED on for the selected slot. To check the corresponding error message, please read /var/log/messages or /var/log/debug (if the --debug or -d ibmphp option was provided). 4.1.3 Changing the state of the attention LEDs ---------------------------------------------- NOTE: In addition to internal indications that the operation is currently being performed or an error occurred, ibmphp driver lets user change the state of attention LEDs. - To turn on attention LED for a slot, type at the command prompt echo 1 > /hotplug//attention - To turn off attention LED for a slot, at the command prompt type echo 0 > /hotplug//attention - To make the attention LED blink for a slot, at the command prompt type: echo 2 > /hotplug//attention 4.1.4 Other files in the slot directory ---------------------------------------- You can find the status of a slot by going to the directory corresponding to that slot and examining the contents of one or more of the files in that directory. cd /hotplug// cat for each particular file. Here is a brief description of what you will find in each of the files: - The latch file contains 1 if the latch of the slot is open, or 0 if the latch is closed. - The attention file contains 0 if the attention LED is off, 1 if the attention is on or 2 if the attention LED is blinking. - The power file contains 0 if the slot power is off or a 1 if the slot power is on. - The adapter file contains 1 if there is an adapter in the slot, or 0 if there is no adapter present. - In SuSE filesystems, there are other files present as well: - cur_bus_speed displays the current bus speed, which can be one of the following: 33 MHz PCI 66 MHz PCI 66 MHz PCI-X 100 MHz PCIX 133MHz PCIX Unknown Bus Speed is displayed if the bus speed is 66 MHz and the bus mode is unknown - max_bus_speed displays the highest speed the bus is capable of running, which will be one of the following: 33 MHz PCI 66 MHz PCI 66 MHz PCI-X 100 MHz PCIX 133 MHz PCIX 4.2 Warnings about incorrect usage of Hotplug PCI ------------------------------------------------- - Do not remove an adapter from a non-hotplug PCI slot. - Grounding equipment, including wrist straps, should be used when working inside any system unit to protect against electrostatic discharge (ESD) that could damage system components and adapter cards. - Make sure that any adapter card that is hot added to the system is fully seated before turning on power to the slot. - Opening an adapter latch will turn off the power to the slot. If a device driver was loaded for that slot, the system could hang, requiring the system to be rebooted. - Do NOT remove the plastic cover over the system components. Removal of the plastic cover exposes system components that may cause a short circuit if they come in contact with adapter brackets during hot plug PCI operations. 5.0 WEB Sites and Support Phone Number --------------------------------------- 5.1 IBM Support Web Site: http://www.pc.ibm.com/support 5.2 IBM Marketing eServer xSeries Web Site: http://www.pc.ibm.com/us/eserver/xseries 5.3 If you have any questions about this update, or problems applying the update go to the following Help Center World Telephone Numbers URL: 800-IBM-SERV (800-426-7378) Hardware Support #: 800-426-4378 Software Support #: 800-237-5511 6.0 Trademarks and Notices --------------------------- 6.1 The following terms are trademarks of the IBM Corporation in the United States or other countries or both: IBM xSeries eServer 6.2 The following terms are trademarks of the Red Hat Corporation in the United States or other countries or both: Red Hat 6.3 The following terms are trademarks of SuSE, Inc. in the United States and SuSE AG in Germany or other countries: SuSE 6.4 The following terms are trademarks of Linus Torvalds in the United States or other countries or both: Linux 7.0 Disclaimer --------------- 7.1 THIS DOCUMENT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. IBM DISCLAIMS ALL WARRANTIES, WHETHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE AND MERCHANTABILITY WITH RESPECT TO THE INFORMATION IN THIS DOCUMENT. BY FURNISHING THIS DOCUMENT, IBM GRANTS NO LICENSES TO ANY PATENTS OR COPYRIGHTS. 7.2 Note to Government Users Documentation related to restricted rights -- Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corporation.