MPI_Finalize slow with CUDA + OpenMPI 2.x (known OpenMPI issue; fixed in 3.1)

[mhoemmen]

Tpetra::CrsMatrix UnitTests2 takes > 560s in a CUDA 8 release build on K80. Seriously, what's going on? Do I need different KOKKOS_ARCH settings? It sure would have been nice to have had some performance tracking so we could have caught this. I don't think this is anything we did; we've only been fixing CUDA issues over time.

36/145 Test  #36: TpetraCore_CrsMatrix_UnitTests2_MPI_4 .......................................................   Passed  561.02 sec

@trilinos/tpetra

[mhoemmen]

I ran this test with verbose mode enabled. It looks like a known OpenMPI issue open-mpi/ompi#3244 :

37: Summary: total = 48, run = 48, passed = 48, failed = 0
37:
37: End Result: TEST PASSED
37: --------------------------------------------------------------------------
37: The call to cuIpcCloseMemHandle failed. This is a warning and the program
37: will continue to run.
37:   cuIpcCloseMemHandle return value:   1
37:   address: 0x20d120000
37: Check the cuda.h file for what the return value means. Perhaps a reboot
37: of the node will clear the problem.
37: --------------------------------------------------------------------------
37: [...:113163] Sleep on 113163
37: [...:113164] Sleep on 113164
37: [...:113166] Sleep on 113166
37: [...:113165] Sleep on 113165
37: [...:113158] 3 more processes have sent help message help-mpi-common-cuda.txt / cuIpcCloseMemHandle failed
37: [...:113158] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

[mhoemmen]

I am using OpenMPI 2.0.1, which according to the above link, manifests the issue. The patch that fixes it was reportedly merged in OpenMPI 3.1.

@bartlettroscoe @nmhamster @micahahoward might want to know about this.

[bartlettroscoe]

@mhoemmen said:

@bartlettroscoe @nmhamster @micahahoward might want to know about this.

We are not seeing runtimes like that in any of the current Trilinos builds (including all of the ATDM Trilinos builds) as shown today, for example, at:

• https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-05-08&filtercombine=and&filtercount=1&showfilters=1&field1=testname&compare1=65&value1=TpetraCore_CrsMatrix_UnitTests2_MPI_4

The max runtime for that test shown there is 31 seconds.

[mhoemmen]

Today: @trilinos/tpetra suggests adding a configure-time test to Tpetra's CMake logic, to detect the OpenMPI version and report a warning (not an error) if it's one of the versions known to have this issue.

[kddevin]

@trilinos/framework The test be at the Trilinos CMake level, rather than at the Tpetra level, right? Codes could call MPI_Finalize() without enabling Tpetra.
Can @trilinos/framework add the CMake logic?

[mhoemmen]

@kddevin wrote:

The test be at the Trilinos CMake level, rather than at the Tpetra level, right? Codes could call MPI_Finalize() without enabling Tpetra.

I agree. The test applies to any package that uses both MPI and CUDA. It can't be in Kokkos, because Kokkos (at least Core) does not depend on MPI. STK depends on Kokkos(Core) and MPI, but not Tpetra. Thus, it makes practical sense for the test to live at the Trilinos CMake level.

[mhoemmen]

@kddevin

@jwillenbring and I chatted about this on the phone today. I think it's a higher priority to fix the Dashboard CUDA builds so they use the right version of OpenMPI. Jim asked whether it would make more sense for the configure process to stop with an error instead of just printing a warning; I thought that would be good but there would always be that one user: https://xkcd.com/1172/

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[github-actions]

This issue was closed due to inactivity for 395 days.