Infiniband Installation of Mellanox Adapters

An OFED (OpenFabrics? Enterprise Distribution) as a collection of drivers and tools to get Infiniband-based clusters running. You can use the OFED provided by Mellanox (although only supported on RHEL and SLES) or the open source OpenFabrics? OFED.

The next steps will provide guidelines on how to install the OpenFabrics? OFED on an openSUSE 11.4 installation.

1. Download the ​OpenFabrics OFED from the OpenFabrics? website

2. Move and extract the tarball on the target machines.

3. Apply this patch when using OFED 1.5.4 and openSUSE 11.4. This updates the distribution and dependencies.

4. Execute the installation script install.pl which builds and installs tools and drivers. Make sure to choose all otherwise libumad will not be built.

5. If the ofa_kernel cannot be built, try to change the option qib=y to qib=n in the ofed.conf configuration file and run the installation tool ./install.pl -c ofed.conf. In this case, the interactive mode is disabled.

6. Reboot

7. Run opensm

Running Tests

1. Check for general information using the hca_self_test.ofed program:

   ---- Performing Adapter Device Self Test ----
   Number of CAs Detected ................. 1
   PCI Device Check ....................... PASS
   Kernel Arch ............................ x86_64
   Host Driver Version .................... OFED-1.5.4: 2.6.37.6-0.9-desktop
   Host Driver RPM Check .................. PASS
   Firmware on CA #0 HCA .................. v2.9.1000
   Firmware Check on CA #0 (HCA) .......... NA
       REASON: NO required fw version
   Host Driver Initialization ............. PASS
   Number of CA Ports Active .............. 0
   Kernel Syslog Check .................... PASS
   Node GUID on CA #0 (HCA) ............... NA
   ------------------ DONE --------------------- 
   

2. Check for adapter status using ibstatus:

   Infiniband device 'mlx4_0' port 1 status:
       default gid:     fe80:0000:0000:0000:0002:c903:0050:2d35
       base lid:    0x1
       sm lid:      0x1
       state:       4: ACTIVE
       phys state:  5: LinkUp
       rate:        40 Gb/sec (4X QDR)
       link_layer:  InfiniBand
   

Alternative Mellanox OFED

For your information, the following steps should be sufficient to install the Mellanox OFED. However this has not been tested as only RHEL and SLES are supported at the moment.

1. Download the Mellanox OpenFabrics? Enterprise Distribution for Linux ISO from the Mellanox website.

2. Mount the distribution

   mount -o ro,loop MLNX_OFED_LINUX-...-x86_64.iso MLNX/
   

3. Change into the mounted system and start the installation using

   ./mlnxofedinstall
   

OpenSuSe 12.1

• Add Kernel:Head repository

  zypper ar http://download.opensuse.org/repositories/Kernel:/stable/standard/Kernel:stable.repo
  

• Update all kernel packages to 3.2 using this repository
• You may need to disable VT-d to use CUDA. Add "iommu=soft" to kernel options in grub
• The following modules should be loaded: mlx4_core, mlx4_ib, mlx4_en, ib_umad, rdma_ucm, ib_ipoib