Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hang in MPI_Finalize with UCX_TLS=rc[_x],sm on the probe-unexp test #1502

Closed
evgeny-leksikov opened this issue May 11, 2017 · 1 comment
Closed

Hang in MPI_Finalize with UCX_TLS=rc[_x],sm on the probe-unexp test #1502

evgeny-leksikov opened this issue May 11, 2017 · 1 comment
Assignees
@evgeny-leksikov
Labels
Bug
Milestone
v1.2 - release

Comments

@evgeny-leksikov
Copy link
Contributor

Command to reproduce:

mpirun -x -np 2496 -mca btl_openib_warn_default_gid_prefix 0 --bind-to core --tag-output --timestamp-output --display-map -mca pml ucx -x UCX_NET_DEVICES=mlx5_0:1 -mca btl_openib_if_include mlx5_0:1 -mca coll_hcoll_enable 0 -x UCX_TLS=rc,sm -mca opal_pmix_base_async_modex 0 -mca mpi_add_procs_cutoff 100000 --map-by node /hpc/scrap/users/mtt/scratch/ucx_ompi/20170510_132009_18827_734706_clx-hercules-113/installs/50e4/tests/mpich_tests/mpich-mellanox.git/test/mpi/pt2pt/probe-unexp

Reproducibility ~50% with rc and rc_x, ud works fine.

Most of processes (~2461) are in MPIx_Fence, others (~35) are in mca_pml_ucx_waitall.

@alinask
@alinask alinask added the Bug label May 11, 2017
@alinask alinask changed the title Hang in MPI_Finalize with UCX_TLS=rc[_x],sm with 2496 ranks Hang in MPI_Finalize with UCX_TLS=rc[_x],sm May 14, 2017
@alinask alinask changed the title Hang in MPI_Finalize with UCX_TLS=rc[_x],sm Hang in MPI_Finalize with UCX_TLS=rc[_x],sm on the probe-unexp test May 14, 2017
This was referenced May 14, 2017
Closed
Closed
@alinask
Copy link
Contributor

alinask commented May 14, 2017

may be related to #1512 and #1513

@yosefe yosefe modified the milestone: v1.2 - release May 15, 2017
evgeny-leksikov added a commit to evgeny-leksikov/ucx that referenced this issue May 21, 2017
@evgeny-leksikov
UCT: fix openucx#1502, openucx#1513
eae599d 
- Fix hang in MPI_Finalize with UCX_TLS=rc[_x],sm
evgeny-leksikov added a commit to evgeny-leksikov/ucx that referenced this issue May 21, 2017
@evgeny-leksikov
UCT: fix openucx#1502, openucx#1513
7bc4db7 
- Fix hang in MPI_Finalize with UCX_TLS=rc[_x],sm
evgeny-leksikov added a commit to evgeny-leksikov/ucx that referenced this issue May 23, 2017
@evgeny-leksikov
UCT: fix openucx#1502, openucx#1513
29d86d1 
- Fix hang in MPI_Finalize with UCX_TLS=rc[_x],sm
@yosefe yosefe closed this as completed in d2a722a May 24, 2017
yosefe added a commit that referenced this issue May 24, 2017
@yosefe
Merge pull request #1532 from evgeny-leksikov/rc_hang_fin
c8f891f 
UCT: fix #1502, #1513
yosefe added a commit that referenced this issue May 25, 2017
@yosefe
Merge pull request #1541 from evgeny-leksikov/v12_port
edca6c0 
UCT: fix #1502, #1513 (backport to v1.2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@evgeny-leksikov evgeny-leksikov

Labels
Bug
Projects
None yet
Milestone
v1.2 - release
Development

No branches or pull requests
3 participants
@yosefe @alinask @evgeny-leksikov