Errata: MPI Active Target synchronization requirements

[jeffhammond]

Problem

The MPI-3.0 standard says the following on page 441, lines 44-47.

A fence call usually entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call. However, a call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier.

That is, a fence is not necessarily a barrier synchronization.

For post-start-complete-wait, there is no specified requirement that the post and start calls need to synchronize.

This implies that the outcome of Example 11.2 (page 424) and Figure 11.2 (page 439) is undefined. In example 11.2, if MPI_WIN_FENCE is not doing a process synchronization, the initialization of array B at the target might be concurrent with a GET operation from the origin on the same memory location, thus giving undefined results.

In Figure 11.2, if post and start are not synchronizing, the store at the target before the post operation might be concurrent with the PUT operation from the origin on the same memory location, thus giving undefined results.

Proposal

Describe the ideas of the proposal.

Changes to the Text

Add the following rule after the 6 rules on page 454:

1. An RMA operation issued at the origin after MPI_WIN_START or MPI_WIN_FENCE to a specific target, accesses the public window copy at the target that is available after the matching MPI_WIN_POST or MPI_WIN_FENCE at the target.

Additional explanation

This rule is the counterpart for rule 3, which specifies synchronization at the end of the epoch, necessary for load/store operations to updated data on the window after the epoch.

Changelog Entry:

Section 11.7 on page 453.

New synchronization rule 7 added. The outcome of Examples 11.1 (page 422) and 11.2 (page 424) is now as expected.

Impact on Implementations

Some implementations (e.g., MPICH) will need to add an extra barrier synchronization to enable correct behavior as intended above.

Impact on Users

Correct MPI programs would have gotten incorrect results in the past. They would now function correctly.

References

Old ticket: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/434

[jeffhammond]

https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/434#comment:7

Just to summarize the discussion on the RMA mailing list, which concluded that the ticket was in error:

The sentence "For post-start-complete-wait, there is no specified requirement that the post and start calls need to synchronize." is not right.

pg 442, ln 31-33: "MPI_WIN_START is allowed to block until the corresponding MPI_WIN_POST calls are executed, but is not required to."

When the standard says the first fence "may" not be a barrier or the above where start "may" not block, it means that if the implementation is able to provide the right fence or pscw semantics without a barrier or block, it may. If it cannot, then it should barrier or block or do something.

An example of where the "may" case works is where the implementation defers all RMA operations to the "second" fence or to the complete-wait. In that case, it is free not to barrier in the first fence or wait for the post.

The definition of Win_fence on pg 441 specifies that

"RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process."

Similarly, the definition of Win_start on page 442 specifies that "RMA accesses to each target window will be delayed, if necessary, until the target process executed the matching call to MPI_WIN_POST."

So both examples mentioned in the ticket should work correctly.

The semantic rules on pg 454 come under this header

453:32-33 "The following rules specify the latest time at which an operation must complete at the origin or the target"

and all those 6 items do indeed talk about latest time or when an operation is completed.

So the proposed addition of item 7 in the ticket is out of place there.

The proposed item 7 is already covered in the definitions of Win_fence and Win_start as mentioned above.

[jeffhammond]

https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/434#comment:9

From the June 2015 Forum meeting: Redo the description of the ticket to focus on the clarity issue, examine the whole chapter for similar issues, and consider whether some reorganization of the semantic rules in the chapter would help make things clearer.