master/v5.0.x: no de-duplication of mpirun launch failure messages

[jsquyres]

On HEAD of master and v5.0.x, we seem to have lost the ability to de-duplicate launch failure messages.

For example, in v4.1.x, we only see one copy of the launch failure message:

$ mpirun --host mpi009,mpi010 -np 4 hostname
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 4
slots that were requested by the application:

  hostname

Either request fewer slots for your application, or make more slots
available for use.

A "slot" is the Open MPI term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which Open MPI processes are run:

  1. Hostfile, via "slots=N" clauses (N defaults to number of
     processor cores if not provided)
  2. The --host command line parameter, via a ":N" suffix on the
     hostname (N defaults to 1 if not provided)
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
  4. If none of a hostfile, the --host command line parameter, or an
     RM is present, Open MPI defaults to the number of processor cores

In all the above cases, if you want Open MPI to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.

Alternatively, you can use the --oversubscribe option to ignore the
number of available slots when deciding the number of processes to
launch.
--------------------------------------------------------------------------

But on master / v5.0.x HEAD, the same message is repeated once per host, and overlays itself in mpirun's output, making it very difficult to read/understand (the problem gets worse if there are more nodes involved):

$ mpirun --host mpi009,mpi010 -np 4 hostname
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 4
slots that were requested by the application:
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 4
slots that were requested by the application:

  hostname

Either request fewer slots for your application, or make more slots
available for use.

A "slot" is the PRTE term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which PRTE processes are run:

  1. Hostfile, via "slots=N" clauses (N defaults to number of
     processor cores if not provided)
  2. The --host command line parameter, via a ":N" suffix on the
     hostname (N defaults to 1 if not provided)
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
  4. If none of a hostfile, the --host command line parameter, or an
     RM is present, PRTE defaults to the number of processor cores

In all the above cases, if you want PRTE to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.

Alternatively, you can use the --map-by :OVERSUBSCRIBE option to ignore the
number of available slots when deciding the number of processes to
launch.
--------------------------------------------------------------------------

  hostname

Either request fewer slots for your application, or make more slots
available for use.

A "slot" is the PRTE term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which PRTE processes are run:

  1. Hostfile, via "slots=N" clauses (N defaults to number of
     processor cores if not provided)
  2. The --host command line parameter, via a ":N" suffix on the
     hostname (N defaults to 1 if not provided)
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
  4. If none of a hostfile, the --host command line parameter, or an
     RM is present, PRTE defaults to the number of processor cores

In all the above cases, if you want PRTE to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.

Alternatively, you can use the --map-by :OVERSUBSCRIBE option to ignore the
number of available slots when deciding the number of processes to
launch.
--------------------------------------------------------------------------

[awlauria]

FWIW: I can only reproduce a single duplicate message no matter how many nodes I involve.

The culprit seems to be this commit, added 2 years ago to prrte: openpmix/prrte@d702d8a

it's sending the message to the tool and stderr, as opposed to just one or the other.

[jsquyres]

Nice work tracking it down.

[rhc54]

Just FYI: The fix I provided doesn't address the issue of "show_help" messages from the OMPI or OPAL layers (or PMIx for that matter). Those messages don't flow thru the same code path - I believe they go thru the PMIx_Log API, or they may just appear on stderr. So those messages are not aggregated or otherwise handled in the "show_help" manner.

So if you get any show_help messages from those sources, I'm afraid you'll still be missing the aggregation feature.

[awlauria]

main: #10348 + #10355
v5.0.x: #10366

PR's merged..closing.