= Infiniband Installation of Mellanox Adapters =

An OFED (OpenFabrics Enterprise Distribution) as a collection of drivers and
tools to get Infiniband-based clusters running. You can use the OFED provided by
Mellanox (although only supported on RHEL and SLES) or the open source OpenFabrics
OFED.

The next steps will provide guidelines on how to install the OpenFabrics OFED on
an ''openSUSE 11.4'' installation.

1. Download the [https://www.openfabrics.org/ofed-for-linux-ofed-for-windows/linux-sources.html OpenFabrics OFED] from the OpenFabrics website

2. Move and extract the tarball on the target machines.

3. Apply [raw-attachment:OFED-1.5.4.install.pl.patch this] patch when using OFED 1.5.4 and openSUSE 11.4. This updates the distribution and dependencies.

4. Execute the installation script `install.pl` which builds and installs tools and drivers. Make sure to choose ''all'' otherwise `libumad` will not be built.

5. If the ofa_kernel cannot be built, try to change the option `qib=y` to `qib=n` in the `ofed.conf` configuration file and run the installation tool `./install.pl -c ofed.conf`. In this case, the interactive mode is disabled.

6. Reboot

7. Run `opensm`

== Running Tests ==

1. Check for general information using the `hca_self_test.ofed` program:
{{{
---- Performing Adapter Device Self Test ----
Number of CAs Detected ................. 1
PCI Device Check ....................... PASS
Kernel Arch ............................ x86_64
Host Driver Version .................... OFED-1.5.4: 2.6.37.6-0.9-desktop
Host Driver RPM Check .................. PASS
Firmware on CA #0 HCA .................. v2.9.1000
Firmware Check on CA #0 (HCA) .......... NA
    REASON: NO required fw version
Host Driver Initialization ............. PASS
Number of CA Ports Active .............. 0
Kernel Syslog Check .................... PASS
Node GUID on CA #0 (HCA) ............... NA
------------------ DONE --------------------- 
}}}

2. Check for adapter status using `ibstatus`:
{{{
Infiniband device 'mlx4_0' port 1 status:
    default gid:     fe80:0000:0000:0000:0002:c903:0050:2d35
    base lid:    0x1
    sm lid:      0x1
    state:       4: ACTIVE
    phys state:  5: LinkUp
    rate:        40 Gb/sec (4X QDR)
    link_layer:  InfiniBand
}}}

== Alternative Mellanox OFED ==

For your information, the following steps should be sufficient to install the
Mellanox OFED. However this has not been tested as only RHEL and SLES are
supported at the moment.

1. Download the ''Mellanox OpenFabrics Enterprise Distribution for Linux'' ISO from the Mellanox website.

2. Mount the distribution
{{{
mount -o ro,loop MLNX_OFED_LINUX-...-x86_64.iso MLNX/
}}}

3. Change into the mounted system and start the installation using
{{{
./mlnxofedinstall
}}}

== !OpenSuSe 12.1 ==
 * Add Kernel:Head repository
{{{
zypper ar http://download.opensuse.org/repositories/Kernel:/stable/standard/Kernel:stable.repo
}}}
 * Update all kernel packages to 3.2 using this repository
 * You may need to disable VT-d to use CUDA. Add "iommu=soft" to kernel options in grub
 * The following modules should be loaded: mlx4_core, mlx4_ib, mlx4_en, ib_umad, rdma_ucm, ib_ipoib