Troubleshooting OFED* for Use with the Intel® MPSS

This is a list of what are hopefully helpful hints for troubleshooting problems with OFED* and InfiniBand* when used with the Intel® Xeon Phi™ coprocessor.

1) Finding Documentation

The primary sources for information on OFED for the Intel Xeon Phi coprocessor are:

• Intel® Manycore Platform Software Stack (Intel® MPSS) User's Guide provided with each MPSS software release
• Intel® Xeon Phi™ Coprocessor System Software Developers Guide

In addition, once Intel MPSS with OFED has been installed, you will find two text files in the directory /usr/share/doc/ofed-driver-2.6.32-279.el6.x86_64-{mpss_version} 

• lustre-phi.txt, describing how to set up a Lustre* file system client on the Intel Xeon Phi coprocessor
• srp-phi.txt, describing how to set up a SCSI RDMA Protocol (SRP) initiator on the Intel Xeon Phi coprocessor

2) Choosing a Version of OFED

This table is a general summary of the different versions of OFED which may be used. When a version of OFED provides no support for the coprocessor, as is currently the case with the OFED included with most Linux* distributions, you will still be able to use OFED to connect to the host. This means you will be able to use OFED with offload applications (where communication to other nodes is always through the host portion of the offload application) but you will not be able to use it for any direct communication with the coprocessor.

This table glosses over some details. For specific information on different versions, be sure to check the Intel® Manycore Platform Software Stack (Intel® MPSS) User's Guide for the version of the Intel MPSS you are installing. In the case of discrepancies between this table and the User's Guide, follow the User's Guide.

3) Finding the Source Code for the MPSS OFED Support

An rpm build file, ofed-driver-{mpss_version}.src.rpm, containing the source code for the OFED kernel modules in the MPSS, is included in the src directory inside the MPSS tar file, mpss-{mpss_version}-{linux_version}.tar. This code is packaged separately to simplify rebuilding just the kernel modules for the host. These kernel modules are the only part of OFED in the MPSS which must be rebuilt if you are using an unsupported Linux distribution.

A separate tar file containing all the OFED code in the MPSS is included in the mpss-src-{mpss_version}.tar file. This includes the code used on both the host and the coprocessor. It does not include the base OFED for the host, which is provided by one of the OFED distributions shown in the table above. It is not necessary to rebuild this code unless you intend to make changes.

4) Getting a Clean Install

There are a number of different problems which can occur if the OFED files from the MPSS are installed without first removing previously installed, alternate distributions of OFED - for example, if you are replacing the OFED from your Linux distribution with Open Fabrics OFED plus MPSS support. In some cases, a package will occur in one distribution but not another, causing dependency errors. For example, the following error message indicates that the previous version of the dapl package was going to be replaced by a version from the MPSS but, because there is no dapl-utils package in the MPSS files, replacing the dapl package would have left the installed version of dapl-utils with an unresolved dependency.

error: Failed dependencies:
        dapl = 2.0.34-1 is needed by (installed) dapl-utils-2.0.34-1.x86_64

To get a clean install, you should, at a minimum, execute the following commands before installing the MPSS:

sudo rpm -e kernel-ib-devel kernel-ib
sudo rpm -e {dapl,dapl-{devel,devel-static,utils}} 

If you use yum to install the MPSS, it will also be necessary to remove these infinipath packages beforehand:

sudo rpm -e --nodeps --allmatches infinipath-libs infinipath-devel 

Another problem you might see is a message similar to the following:

/{build_location}/ofa_kernel-1.5.4.1/net/sunrpc/xprtrdma/transport.c:218: error: 'struct rpc_xprt' has no member named 'shutdown'

There were a number of struct members removed from Linux 3.10 and rpc_xprt.shutdown was one of them. If you are using a 3.10 or later kernel, you should use an MPSS release that has been tested against RHEL 7.0 or later (e.g. use MPSS 3.3 or later.)

5) Starting OFED - Harmless Warnings

When the OFED services are started, you may see an error message saying:

fmr_pool: Device scif0 does not support FMRs
Error creating fmr pool

Actually this is true; scif0 does not support fast memory registration (FMR) from Mellanox. The message is expected and is not a problem.

6) Starting OFED - When One or More Kernel Modules Will Not Load

Before installing a new version of the Intel MPSS with OFED support, it is always a good idea to follow the directions in the Intel MPSS User's Guide for shutting down OFED. If you fail to do this, you may find that one or more kernel modules will not load after the install and you get the following error message when you execute 'modprobe {module name}' :

FATAL: Error inserting {module name} (/lib/modules/{host kernel version}/updates/drivers/infiniband/{driver type}/{module name}.ko): Unknown symbol in module, or unknown parameter (see dmesg)

The most likely cause of this message is that a preinstall version of the openibd service is still running and is blocking the new modules from loading. You can sometimes get around this error by executing

sudo service openibd restart

or rebooting the host to ensure that you pick up the new kernel modules.

If restarting the openibd service does not resolve the problem, then review the OFED modules installed in the system. Are there any modules containing the strings "ofed" or "ib" which did not get replaced when you installed the Intel MPSS with OFED support? Or did any of the old modules get reinstalled over the new? Sometimes the safest thing is to reinstall OFED - this time stopping all OFED services before starting. (See item 5 Getting a clean install).

7) Determining the Status of OFED/InfiniBand

When OFED for the MPSS is running, much of the status information can be found in /sys/class/infiniband. If you are using IFS OFED+, you will see only one subdirectory - that for the physical adapter itself. For all other versions of OFED, you will see two subdirectories - one for the physical adapter, for example mlx4_0, and one for the ibscif virtual interface, scif0. If you would rather not dig around in obscure directories, you can get much of this information from the command, ibv_devinfo. An example of output from the command is:

hca_id: scif0
        transport:                      iWARP (1)
        fw_ver:                         0.0.1
        node_guid:                      4c79:baff:fe14:0033
        sys_image_guid:                 4c79:baff:fe14:0033
        vendor_id:                      0x8086
        vendor_part_id:                 0
        hw_ver:                         0x1
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               1000
                        port_lmc:               0x00
                        link_layer:             IB

hca_id: mlx4_0
        transport:                      InfiniBand (0)
        fw_ver:                         2.30.8000
        node_guid:                      f452:1403:007c:bd30
        sys_image_guid:                 f452:1403:007c:bd33
        vendor_id:                      0x02c9
        vendor_part_id:                 4099
        hw_ver:                         0x0
        board_id:                       MT_1100120019
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_INIT (2)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             IB

A few things you will want to notice - until the fabric manager, opensm, has assigned a Local IDentifier (LID) to a port, the state of the interface will be listed as PORT_INIT. In the example shown, there is no opensm service running, so the port on the adapter has not yet been given a LID and is in the PORT_INIT state. When an opensm service is started and has assigned a LID, this state will change to PORT_ACTIVE. The ibscif interface (scif0), however, has a preassigned LID and is in the PORT_ACTIVE state even though there is no opensm service. You will also want to notice that the transport for the ibscif interface is given as iWARP. This indicated that the mpxyd service is running and CCL is being used. When the mpxyd service is not running, the value will instead be SCIF.

A command with shorter output is ibstatus:

Infiniband device 'mlx4_0' port 1 status:
        default gid:     fe80:0000:0000:0000:f452:1403:007c:bd31
        base lid:        0x0
        sm lid:          0x0
        state:           2: INIT
        phys state:      5: LinkUp
        rate:            56 Gb/sec (4X FDR)
        link_layer:      InfiniBand

Infiniband device 'scif0' port 1 status:
        default gid:     fe80:0000:0000:0000:4e79:baff:fe14:0033
        base lid:        0x3e8
        sm lid:          0x1
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            40 Gb/sec (4X QDR)
        link_layer:      Ethernet

This command also shows the data rate for the port. For the ibscif interface is 40 Gb/sec (4X QDR).

8) Determining If Your Data is Actually Going Over the InfiniBand Connection

For MPI, you can explicitly determine the path your data is traveling over by passing the environment variable "I_MPI_DEBUG 3" to your MPI job. In this case, dapl is being used to send the data over a Mellanox adapter.

mpirun –genv I_MPI_DEBUG 3 –n 16 ./prog_name
[1] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1
[0] DAPL startup(): trying to open first DAPL provider from I_MPI_DAPL_PROVIDER_LIST: ofa-v2-mlx4_0-1
[0] MPI startup(): DAPL provider ofa-v2-mlx4_0-1
…
[0] MPI startup(): dapl data transfer mode

In cases where MPI is not being used, you can use the InfiniBand counters:

ibclearcounters        # clear counters before starting your job.
ibdatacounters         # check receive data on all ports while your job is running and/or after it has completed

Because the counters do not show which process was transferring data, you cannot know positively that it was your process using the InfiniBand connection. If your process is not the only one running on the node, you may also want to check the counters for the Ethernet connection. For example, for eth0, you could execute the following command immediately before and after running your job, then compare the number of bytes transmitted and received.

ip -s link show eth0

If there is no significant Ethernet traffic, then the InfiniBand counters probably reflect your InfiniBand usage.

9) Unloading the mpss Service with OFED

On occasion, when attempting to unload the mpss service, you will get an error message telling you that the mic kernel module is in use. If you are using OFED, make sure that the mpxyd service has been stopped. When this service is running, the mic kernel module is, indeed, in use.