Transport retry count exceeded on mlx5_1:1/IB

[J-shel]

Hi,

I got the following error message using OpenMPI 4.1.1 and ucx 1.10 or OpenMPI 4.0.3 and ucx1.10.

[n07299:125295:0:125295] ib_mlx5_log.c:145  Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0)
[n07299:125295:0:125295] ib_mlx5_log.c:145  DCI QP 0x13014 wqe[399]: NOP --- [rqpn 0x0 rlid 0]

I' ve searched and seen similar issues on openucx. As a first try, I installed a lower version of ucx, i.e., ucx1.9. However, when I install Openmpi 4.1.1 or Openmpi4.0.3 with ucx1.9, I got the following message:

Making all in mca/common/ucx
make[2]: Entering directory `/packages/openmpi-4.0.3_ucx1_9/openmpi-4.0.3/opal/mca/common/ucx'
  CC       libmca_common_ucx_la-common_ucx.lo
  CCLD     libmca_common_ucx.la
ld: cannot find -lnuma
make[2]: *** [libmca_common_ucx.la] Error 1
make[2]: Leaving directory `/packages/openmpi-4.0.3_ucx1_9/openmpi-4.0.3/opal/mca/common/ucx'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/packages/openmpi-4.0.3_ucx1_9/openmpi-4.0.3/opal'
make: *** [all-recursive] Error 1

I'm sure I have numa lib (libnuma.so.1) installed, which is located in /usr/lib64. And I added this path to LD_LIBRARY_PATH and LIBRARY_PATH. I have no idea why it cannot find it! The commands I used to install openmpi is:

ucx_libs()
{
        builddir=/es01/software/ucx19
        libs=""
        for x in s m t p
        do
                libs="${libs}:${builddir}/lib/libuc${x}.so"
        done
        echo ${libs}
}
export LD_PRELOAD=$(ucx_libs ${builddir})
echo $LD_PRELOAD

tar zxvf openmpi-4.0.3.tar.gz
cd openmpi-4.0.3
./configure --prefix=/user/local/openmpi_4_0_3_intel2018_ucx_1_9 CC=icc CXX=icpc FC=ifort --with-ucx=/es01/software/ucx19
make
make install

Yep, I want to solve issue 1, but I got another issue. Both of them confused me!
Followed is some information about my platform:

$lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 56
On-line CPU(s) list: 0-55
Thread(s) per core: 1
Core(s) per socket: 28
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6258R CPU @ 2.70GHz
Stepping: 7
CPU MHz: 999.975
CPU max MHz: 4000.0000
CPU min MHz: 1000.0000
BogoMIPS: 5400.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 39424K
NUMA node0 CPU(s): 0-27
NUMA node1 CPU(s): 28-55
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni spec_ctrl intel_stibp flush_l1d arch_capabilities

 
$ibstat
CA 'mlx5_0'
CA type: MT4123
Number of ports: 1
Firmware version: 20.29.2002
Hardware version: 0
Node GUID: 0x0c42a10300387d0c
System image GUID: 0x0c42a10300387d0c
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 749
LMC: 0
SM lid: 1988
Capability mask: 0x2651e848
Port GUID: 0x0c42a10300387d0c
Link layer: InfiniBand
CA 'mlx5_1'
CA type: MT4123
Number of ports: 1
Firmware version: 20.29.2002
Hardware version: 0
Node GUID: 0x0c42a10300387d38
System image GUID: 0x0c42a10300387d38
Port 1:
State: Down
Physical state: Polling
Rate: 10
Base lid: 65535
LMC: 0
SM lid: 0
Capability mask: 0x2651e848
Port GUID: 0x0c42a10300387d38
Link layer: InfiniBand

 
$ucx_info -v
# UCT version=1.10.0 revision a212a09
# configured with: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-optimizations --disable-logging --disable-debug --disable-assertions --enable-mt --disable-params-check --without-java --enable-cma --with-cuda --without-gdrcopy --with-verbs --without-cm --with-knem --with-rdmacm --without-rocm --with-xpmem --without-ugni --with-cuda=/usr/local/cuda-10.2

[yosefe]

@J-shel

1. Regarding libnuma: can you pls install numactl-devel rpm?
2. Regarding the transport error: can you pls try the following commands (on all compute nodes) and run the test again:

sudo mst start
sudo mcra  /dev/mst/mt4123_pciconf0 0x4b0.0:1 1

[J-shel]

Hi yosefe,

Thank you for your reply. Our platform is under maintenance in these few days. I'll update later, maybe tomorrow or the day after tomorrow.

[J-shel]

@yosefe Sorry for the delay. Issue #1 was solved following you suggestion. After the maintenance of our platform, however, I didn't meet issue #2 anymore. I'm not sure whether it has something to do with the OpenMPI mpirun parameters since I add "-x LD_LIBRARY_PATH -x UCX_NET_DEVICES=mlx5_0:1 -mca pml ucx -mca btl ^vader,tcp,openib,uct" after mpirun in the new run, while I use the default parameters, i.e. I add no parameters myself, in the old run.