Offload performance differences

based on a sample program posted here by @Rajiv Deodhar : (thanks for the sample program!) I ran into some strange results with my Xeon Phi's here:

HostA: xeon E5 2695 v2 w/ Xeon Phi 7100, Scientific Linux 6.5, MPSS 3.3.2 stack, Intel compiler icc (ICC) 14.0.1 20131008

Physical hardware is a Dell PowerEdge R720

HostB: xeon E5 2620 w/ Xeon Phi 5100, CentOS 6.5, MPSS 3.3.2 stack, Intel compiler icc (ICC) 15.0.0 20140723

Physical hardware is a Supermicro X9DRG-HF

I've compiled the sample code small_0.c using both icc 14 and icc 15 using

icc -openmp -o small_0 small_0.c

when I run the binaries, small_0.icc14 and small_0.icc15 on hostA I get a nice&consistent 6 GB/s send and receive speed

when I run the binaires on hostB I get mixed results:

- "Send" performance is almost always 6+ GB for buffers that are large enough

- most of the times the icc14 binaries give good "Receive" performance as well but sometimes it collapses to the same level as the icc15 binaries

- the icc15 binaries provide good "Send" performance, "Receive" performance is usually awful:

Bandwidth test for pointers. DeviceID: 0. Data alignment: 2097152. Number of iterations: 10.

         Size(Bytes) Send(GiB/sec) Receive(GiB/sec)
                1024     0.13             0.15
                2048     0.28             0.31
                4096     0.54             0.62
                8192     0.79             0.88
               16384     1.24             1.53
               32768     2.13             2.50
               65536     3.11             3.63
              131072     4.18             4.61
              262144     5.02             5.34
              524288     5.63             5.88
             1048576     5.97             6.19
             2097152     6.23             6.36
             4194304     6.29             6.40
             8388608     5.99             1.26
            16777216     6.16             0.68
            33554432     6.13             0.54
            67108864     6.29             0.55
           134217728     6.37             0.51
           268435456     6.41             0.47
           536870912     6.43             0.38
          1073741824     6.44             0.33

Sometimes it helps to set the env var MIC_USE_2MB_BUFFERS=2M but right now that also does not seem to help.

Running it as root or as a regular user also has no effect. HostB is not loaded , I'm the only user on both the host and the Phi's. Switching to the second phi does not make any difference.

'Netperf' performance on hostB seems OK, 4+ Gbit/s one way, 1.8 Gbit/s the other (compared to 4+/1.9 for hostA). 

How can I further debug this? If need be access to hostB can be provided , contact me offline for that.