Sync to PMIx v2.1.0

[karasevb]

Signed-off-by: Boris Karasev karasev.b@gmail.com

[jladd-mlnx]

👍 Good to go.

[jladd-mlnx]

👍
@hppritcha @bwbarrett This is ready to go.

[rhc54]

I am receiving consistent reports of "hangs" of PMIx-based programs using PMIx v2.1.0rc2 when direct launched against Slurm 17.11. I would advise not moving forward until that gets resolved as we have no info as to whether the problem is in the Slurm plugin, or in PMIx itself.

Given reassignment of @artpol84 and @karasevb, I'm not sure when the Slurm problem will be investigated. Perhaps someone here can comment?

[jjhursey]

This PR looks like a fine sync of PMIx 2.1 to the v3.1 branch

I saw @rhc54 's concern in the comments - so I don't know if we want to delay merging until that gets resolved or not.

[jladd-mlnx]

@rhc54 we still maintain the SLURM plugin. We have this under test and do not see issues. In fact, we have problems without it. @artpol84 can you check if this hangs with srun launch with SLURM 17.11, please.

[artpol84]

@rhc54 can you share those reports with me and forward those who has issues to me?

[artpol84]

@rhc54 I’ll appreciate if in future you will let us know ASAP if any issues related to Slurm/PMIx

[artpol84]

From our side we are running Slurm with pmix v2.1 and Slurm on daily basis:
https://mtt.open-mpi.org/index.php?do_redir=2554
You can see in the list that we cover all pmix release versions and multiple OMPI versions and there is no dramatical difference in results between the versions. @karasevb please double-check the failures.
We are using external pmix, but if this difference is the issue - then OMPI has problems with internal integration.

@jjhursey I think we should merge it and we will address the issues if any.

[jjhursey]

@artpol84 Maybe @rhc54 is thinking about this PMIx user reported issue:

• https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/pmix/p_NaCza4y_U/E93ieFZDBAAJ
  Did you all resolve that with them offline?

[artpol84]

@jjhursey according the description it doesn't seem like hang, but data corruption.
@karasevb please check.

[rhc54]

@artpol84 I have been letting you know about these problems, but haven't been getting any response. In addition to the mailing list, I have emailed you directly about it.

The reports are coming from both HPe and Intel. The behavior is the same in both cases. Note that HPe is not using OMPI, but rather a simple PMIx client test code. Same for Intel.

In the Intel case, Slurm 17.11.2.1 is configured with PMIx v2.10rc2. A simple "srun --mpi=pmix_v2" of a PMIx test client that calls PMIx_Init/Finalize hangs until timeout occurs. I don't have a lot of diagnostic output at this time, but have requested more.

[artpol84]

We will resolve this

[rhc54]

Looking into the reports, it appears that the fence may be broken. One possibility that might explain the difference between your tests and what is being reported is - are your tests always using the IB "accelerated" path to do communications? If so, I suspect the other code path is having problems.

[artpol84]

We test all of the cases.
I'm taking it with HPE now. Will update on the result.

[jladd-mlnx]

@bwbarrett @hppritcha:
@artpol84 and @karasevb discovered an error in one of the standalone PMIx tests. This was the root cause of the issue reported by HPE. All of our testing is clean. I think we should merge this which will allow for broader testing coverage.

[artpol84]

Here are some links:

• SchedMD bug:
  • https://bugs.schedmd.com/show_bug.cgi?id=4678 (reported to be closed)
• PMIx issue:
  • Incorrect example "client.c" openpmix/openpmix#648 (closed through the fix)

[rhc54]

Official release is now available: https://github.com/pmix/pmix/releases/tag/v2.1.0

Contains a couple required bug fixes beyond rc2, so I'd recommend updating before commit.

[jjhursey]

Yeah this is good to go. 👍 Thanks!

[bwbarrett]

@karasevb, with PMIx 2.1.0 going GA, is it possible to refresh this patch?

[rhc54]

@bwbarrett I already updated it - should be ready to go