I am running MPSS 3.5 and OFED+ 7.3.1.0.12 and have 2 nodes with Phi cards and Qlogic. I believe that if I set I_MPI_FABRICS to use either tcp or tmi everything works, but I've heard that dapl is faster and I'm having problems getting that to work everywhere. It works when MPI tasks are either only on the hosts or only on cards in a single host. If there are tasks on the host and a card it appears to have problems connecting to the IP address that is added during the ofed-mic service startup (192.0.2.0/24). I noticed that I am unable to ping that address on a card from the host or vice versa. On the other hand was was able to run dtest from the dapl-utils package and once I asked for the correct providers it passed the test and appeared to use those IPs for connection.
Can anybody tell me if I should be able to ping the 192 address between a card and the host?
Can anybody tell me which dapl providers should get used on each end of the connection between a host and a card with Qlogic HBA and Intel's OFED+?
What else should I be doing to debug this?
Thanks,
Mike Robbert