Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

prov/cxi: peer infrastructure support #10436

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

prov/cxi: peer infrastructure support #10436

wants to merge 4 commits into from
+528 −168

Commits on Oct 4, 2024

  1. prov/cxi: Add FI_PEER capability bit 

    Add the FI_PEER capability bit to the CXI provider fi_info

Signed-off-by: Amir Shehata <shehataa@ornl.gov>
    @amirshehataornl
    amirshehataornl committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    6ddb318 View commit details
    Browse the repository at this point in the history

  2. prov/cxi: Implement shared Completion Queues 

    On cq_open, check the FI_PEER_IMPORT, if set, set all internal cq operation
to be enosys, with the exception to the read callback.

The read callback is overloaded to operate as a progress callback
function. Invoking the read callback will progress the enpoints linked to
this CQ.

Keep track of the fid_peer_cq structure passed in.

If the FI_PEER_IMPORT flag is set, then set the callbacks in cxip_cq structure
which handle writing to the peer_cq, otherwise set them to the ones which
write to the util_cq.

A provider needs to call a different set of functions to insert
completion events into an imported CQ vs an internal CQ.

These set of callback definition standardize a way to assign a different
function to a CQ object, which can then be called to insert into the CQ.

For example:

	struct prov_cq {
		struct util_cq *util_cq;
		struct fid_peer_cq *peer_cq;
		ofi_peer_cq_cb cq_cb;
	};

When a provider opens a CQ it can:

	if (attr->flags & FI_PEER_IMPORT) {
		prov_cq->cq_cb.cq_comp = prov_peer_cq_comp;
	} else {
		prov_cq->cq_cb.cq_comp = prov_cq_comp;
	}

Collect the peer CQ callbacks in one structure for use in CXI.

Signed-off-by: Amir Shehata <shehataa@ornl.gov>
    @amirshehataornl
    amirshehataornl committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    538e2fb View commit details
    Browse the repository at this point in the history

  3. prov/cxi: Support shared receive queues 

    Restructure the code to allow for posting on the owner provider's shared
receive queues.

Do not do a reverse lookup on the AV table to get the fi_addr_t, instead
register an address matching callback with the owner. The owner can then
call the address matching callback to match an fi_addr_t to the source
address in the message received.

This is more efficient as the peer lookup can be an O(1) operation;
AV[fi_addr_t]. The peer's CXI address can be compared with the CXI address
in the message received.

Signed-off-by: Amir Shehata <shehataa@ornl.gov>
    @amirshehataornl
    amirshehataornl committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    ef4a04a View commit details
    Browse the repository at this point in the history

  4. prov/cxi: handle late peer insertion 

    Handle the case where a message from a peer arrives before the peer is
inserted.

Implement the callflow to support this scenario.

Signed-off-by: Amir Shehata <shehataa@ornl.gov>
    @amirshehataornl
    amirshehataornl committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    e9db7c5 View commit details
    Browse the repository at this point in the history