Make stream buffing an MCA option again

[jjhursey]

• Way back when the stream buffering option was introduced
  it was an MCA: 4dd9f89a9
  • It changed to a CLI option at some point.
• Make this an MCA option once again, and connect it with the
  PRRTE CLI options.

[jjhursey]

Pairs with: openpmix/prrte#1489

[rhc54]

Isn't that going to break mpirun operations where PMIx is doing the IOF and has no idea that you have done this? I suspect PRRTE/PMIx will feel free to overwrite, which is going to confuse - yes?

[rhc54]

I think you may run into issues here - but it's totally up to you over here. I'm not yet convinced of the piece over in PRRTE, but that's another matter.

[jjhursey]

@rhc54 I'm not sure I understand your concern. The application (whether in OMPI or their code) can call setvbuf to adjust the kernel buffering of an IO stream at any point. How would this break mpirun? This would only impact how often the kernel is flushing the stream (stdout/stderr in this case) from the MPI application process to whoever is watching that stream.

I did test this with a few different scenarios, and it works fine.

[rhc54]

I'm half blind on painkillers, so my brain is mush - please let things percolate down slowly thru the mud. I agree about the app always being able to do this. Just trying to get my head around the overall IO path and what we want to happen, and what they expect to see.

[gpaulsen]

@rhc54 hope you feel better soon.

Is there a test we can add to ensure proper streaming behavior for both mpirun and prterun? I've been trying to read manpages, but didn't fine a way to query that from a file descriptor.

[rhc54]

I think Josh and I are discussing whether mpirun needs to have a particular behavior or not over on the PRRTE side.

[jjhursey]

FYI: We resolved the issue on the PRRTE side (openpmix/prrte#1489) so this PR is ready to go.