OMPI compilation issues with rdma-core

[fsl-latif]

RDMA-core v - Release 13
https://github.com/linux-rdma/rdma-core

OMPI v - 2.0.2

Compilation Issues
o Missing driver.h header.
The header does exists in rdma-core/libibverbs and we had to manually copy it to /usr/include/infiniband to get things to compile. driver.h is classified as an internal header as part of libibverbs. There are a couple of libraries that includes this file and all providers have it. Not sure why OpenMPI requires it? should OMPI stick to verbs API instead of including something that’s internal to the libraries?

o Openmpi requires libopensm (headers) and libfabric library and these packages are not part of rdma-core package at all.
Is their any config options in openmpi to not include these if not needed.

[jsquyres]

Can you post the details of the compilation failure?

Also, note that libfabric is unrelated to the verbs infrastructure. You can always build Open MPI without any libfabric or verbs support, but I suspect that that is not what you want. Can you provide more information -- e.g., what exactly are you trying to do? See https://www.open-mpi.org/community/help/.

[fsl-latif]

(1) Missing driver.h in .usr/include/infiniband

make[2]: Entering directory `/root/openmpi-2.0.2/opal/mca/common/verbs'
  CC       libmca_common_verbs_la-common_verbs_basics.lo
common_verbs_basics.c:36:31: fatal error: infiniband/driver.h: No such file or directory
 #include <infiniband/driver.h>
                               ^
compilation terminated.
make[2]: *** [libmca_common_verbs_la-common_verbs_basics.lo] Error 1
make[2]: Leaving directory `/root/openmpi-2.0.2/opal/mca/common/verbs'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/openmpi-2.0.2/opal'
make: *** [all-recursive] Error 1

(2) Without libfabric installed

make[2]: Entering directory `/root/openmpi-2.0.2/opal/mca/btl/usnic'
  CC       btl_usnic_compat.lo
In file included from btl_usnic_frag.h:28:0,
                 from btl_usnic_compat.c:23:
btl_usnic_module.h:29:25: fatal error: rdma/fabric.h: No such file or directory
 #include <rdma/fabric.h>
                         ^
compilation terminated.
make[2]: *** [btl_usnic_compat.lo] Error 1
make[2]: Leaving directory `/root/openmpi-2.0.2/opal/mca/btl/usnic'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/openmpi-2.0.2/opal'
make: *** [all-recursive] Error 1

(3) Without opensm installed 
make[2]: Entering directory `/root/openmpi-2.0.2/opal/mca/btl/openib'
  CC       connect/btl_openib_connect_sl.lo
  CCLD     mca_btl_openib.la
/usr/bin/ld: cannot find -losmcomp
collect2: error: ld returned 1 exit status
make[2]: *** [mca_btl_openib.la] Error 1
make[2]: Leaving directory `/root/openmpi-2.0.2/opal/mca/btl/openib'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/openmpi-2.0.2/opal'

Well I do want verbs support, but not libfabric at this point. If OMPI can be compiled without libfabric, please point me to how.

I am trying to run some IMB benchmarks with OMPI on a verbs capable device, using the user-space rdma libs specifically from rdma-core only (this is the new open source consolidated package for rdma user-space libs).
https://github.com/linux-rdma/rdma-core

[coderex2522]

when I install the rdma-core/providers/rxe,it informs me that"fatal error: infiniband/driver.h: No such file or directory".So I check the rdma-core package, and I find older package libibverbs contains the "infiniband/driver.h",but the rdma-core package doesn't contains the infiniband/driver.h. How do I solve it.

[jsquyres]

I'm sorry, I somehow totally missed your previous reply.

You can disable libfabric support via configure --without-libfabric. See the Open MPI README file for lots of common configure switches and descriptions of what they do.

Additionally, later versions of Open MPI have adapted to the newer versions of RDMA core that no longer provide infiniband/driver.h. You might try using Open MPI 3.0.0, for example. Let me know if that works.

I didn't think that we required Open SM to be installed, but try again with Open MPI 3.0.0 and see what happens. Post the details if that doesn't work.

[jsquyres]

Looking at this issue again today, I now realize that the comment yesterday was from a different person than the original submitter. 😄

If both of you can try with a newer version of Open MPI (e.g., 3.0) that should be compatible with the newer versions of RDMA core, that would be great. Thanks.

[jsquyres]

Haven't heard anything back from the reporting users, so I'm going to close this issue. If you need more help, reply here and we can re-open if relevant. Thanks!