fi_getname() doesn't work with infiniband verbs

[theWayofthecode]

Hi all,

I am trying to implement a basic one-sided RMA communication mechanism with libfabric and I'm stuck. I would appreciate a pointer about what I'm missing.

Expected result: fi_getname() returns the name of the current endpoint (to be used by another endpoint which will perform remote writes)
Result that I get: invalid string in the char * (garbage value ?)

My code works with the "shm" provider but it doesn't with "verbs;ofi_rxm".

This is a minimal reproducer for my problem (the compilable version of the code is attached):

int main (int argc, char *argv[])
{
    fi_context ctx;

    struct fi_info *hints = fi_allocinfo();
    assert(hints);
    hints->ep_attr->type = FI_EP_RDM;
    hints->addr_format = FI_FORMAT_UNSPEC;
    hints->caps = FI_MSG | FI_RMA;
    hints->mode = FI_CONTEXT;
    hints->domain_attr->mr_mode = FI_MR_LOCAL | FI_MR_ALLOCATED | FI_MR_PROV_KEY | FI_MR_VIRT_ADDR;
    hints->fabric_attr->prov_name = strdup("verbs");

    struct fi_info *info;
    CHECK(fi_getinfo(FI_VERSION(1, 11), argv[1], argv[2], 0, hints, &info));
    assert(info);
    std::cerr << "name=" << info->fabric_attr->name << " prov_name=" << info->fabric_attr->prov_name << std::endl;

    struct fid_fabric *fabric;
    CHECK(fi_fabric(info->fabric_attr, &fabric, &ctx));

    struct fid_domain *domain;
    CHECK(fi_domain(fabric, info, &domain, &ctx));

    struct fid_ep *ep;
    CHECK(fi_endpoint(domain, info, &ep, &ctx));

    // Init CQ
    fi_cq_attr cqAttr = {0};
    cqAttr.format = FI_CQ_FORMAT_CONTEXT;
    cqAttr.wait_obj = FI_WAIT_NONE;
    cqAttr.size = info->tx_attr->size;
    struct fid_cq *cq_tx, *cq_rx;
    CHECK(fi_cq_open(domain, &cqAttr, &cq_tx, &ctx));
    cqAttr.size = info->rx_attr->size;
    CHECK(fi_cq_open(domain, &cqAttr, &cq_rx, &ctx));
    int flags = FI_TRANSMIT;
    flags |= FI_SELECTIVE_COMPLETION;
    CHECK(fi_ep_bind(ep, &cq_tx->fid, flags));
    flags = FI_RECV;
    flags |= FI_SELECTIVE_COMPLETION;
    CHECK(fi_ep_bind(ep, &cq_rx->fid, flags));

    // Init Address vector
    fi_av_attr avAttr;
    memset(&avAttr, 0, sizeof(avAttr));
    avAttr.count = 1;
    struct fid_av *av;
    CHECK(fi_av_open(domain, &avAttr, &av, NULL));
    CHECK(fi_ep_bind(ep, &av->fid, 0));

    CHECK(fi_enable(ep));

    // GET NAME
    size_t len = 256;
    char addr[len];
    memset(addr, '\0', len);
    CHECK(fi_getname(&ep->fid, addr, &len));
    std::cerr << "addr=" << addr << " len=" << len << std::endl;
    std::cerr << std::hex << "0x" << *((unsigned long long *)(addr)) << std::endl;

This is the output with FI_LOG_LEVEL=warn and node=the_ip_address_corresponding_to_ib:

libfabric:46066:verbs:fabric:vrb_get_device_attrs():613<warn> device cxgb4_0: there are no active ports
libfabric:46066:core:core:fi_getinfo_():1019<warn> fi_getinfo: provider verbs returned -61 (No data available)
libfabric:46066:core:core:fi_getinfo_():1019<warn> fi_getinfo: provider verbs returned -61 (No data available)
libfabric:46066:core:core:fi_getinfo_():1019<warn> fi_getinfo: provider ofi_rxd returned -61 (No data available)
libfabric:46066:ofi_mrail:fabric:mrail_get_core_info():289<warn> OFI_MRAIL_ADDR_STRC env variable not set!
libfabric:46066:core:core:fi_getinfo_():1019<warn> fi_getinfo: provider ofi_mrail returned -61 (No data available)

name=IB-0xfe80000000000000 prov_name=verbs;ofi_rxm
addr=� len=16
0x3c00e10ae1a30002

The "addr=�" part is what doesn't make sense to me.

Relevant details of the environment:

• Libfabric version: 1.11.1
• Linux kernel version: 3.10.0-1160.15.2.el7.x86_64
• Modules: ib_core, ko2iblnd, iw_cxgb4, rpcrdma, rdma_ucm, ib_iser, ib_umad, rdma_cm, iw_cm, ib_ipoib, ib_cm, mlx5_ib, ib_uverbs
• Mellanox mlx5_0: CA type: MT4119, Firmware: 16.24.1000

In this environment the fi_pingpong -p verbs -e rdm works.

Let me know if you need additional information.

Thank you in advance,
Aram

Reproducer.zip

[shefty]

fi_getname() returns a binary address, not a string. In the example above, the format of that address can be determined by looking at info->addr_format. I think verbs uses a sockaddr based format.

[theWayofthecode]

Oh, I didn't realize that the address can also be in binary form (might be useful to add a note in the docs). In my case the addr format is FI_ADDR_IB_UD but anyway the conclusion is that the fi_getname() was working fine all along. Thanks for clarifying that.

I figured out why my code was not working. I was using fi_cntr to wait for FI_REMOTE_WRITE events, which was fine with the shm provider but the ib doesn't support the FI_RMA_EVENT cap which is a requirement for the counter to count remote RMA operations. I guess I just have to rely on fi_cq, and I should be fine.