Hang on UD-only MPI runs on some hosts

[alex--m]

I noticed the problem on the Thor Cluster (Mellanox-internal). The problem did not reproduce on other hosts. The following command hangs, while changing ud to rc (or disabling UCX altogether) make is pass:

salloc -N 4 mpirun -bind-to core -map-by node -mca pml ucx -x UCX_TLS=ud,shm,self -x UCX_NET_DEVICES=mlx5_0:1 osu_igather

Also, hangs even w/o HCOLL, with ud_x, and for other async (i*) OSU collective benchmarks. With osu_gatherv - hangs at around 4K (message size).
Is there any additional data you'd like me to provide?

[yosefe]

Is it Connect-X5? What is FW version?

[yosefe]

see also #1513

[alex--m]

Doesn't look related to #1513, since the hang is on the first sends - it doesn't start showing benchmark results (I attached gdb - all procs seem to wait on poll_cq()). Also, it's UD, not RC (RC works).

I'll post the FW & HCA versions.

[alex--m]

It is Connect-X5. Still waiting on the FW version.

[yosefe]

Currently it works for me on that cluster:

[root@thor001 ~]# ibstat mlx5_0
CA 'mlx5_0'
	CA type: MT4121
	Number of ports: 1
	Firmware version: 16.21.2010
	Port 1:
		State: Active
		Rate: 100
		Link layer: InfiniBand
[root@thor001 ~]#  mpirun --allow-run-as-root -bind-to core -map-by node -mca pml ucx -x UCX_TLS=ud,shm,self -x UCX_NET_DEVICES=mlx5_0:1 $HPCX_MPI_TESTS_DIR/osu-micro-benchmarks-5.3.2/osu_igather

# OSU MPI Non-blocking Gather Latency Test v5.3.2
# Overall = Coll. Init + Compute + MPI_Test + MPI_Wait

# Size           Overall(us)       Compute(us)    Pure Comm.(us)        Overlap(%)
...
   <results>
...

@alex--m can we close this?

[alex--m]

If OSU passes i consider it solved.