-
Notifications
You must be signed in to change notification settings - Fork 423
OpenMPI and OpenSHMEM installation with UCX
Requirements: Autoconf 2.63 and above.
- Get latest version of the UCX code
$ git clone https://github.com/openucx/ucx.git ucx
$ cd ucx
- Run autogen:
$ ./autogen.sh
- This step is only required for OpenPOWER platforms - Power 8 On Ubuntu platform the config.guess file is a bit outdated and does not have support for power. In order to resolve the issue you have to download an updated config.guess. From the root of the project:
$ wget https://github.com/shamisp/ucx/raw/topic/power8-config/config.guess
- Configure:
$ mkdir build
$ cd build
$ ../configure --prefix=/your_install_path
Note: For best performance configuration, use ../contrib/configure-release
.
This will strip all debugging and profiling code.
- Build and install:
$ make
$ make install
- Running unit tests (using google test).
This only work if gtest was installed and detected on your platform, and--enable-gtest
was passed toconfigure
:
$ make -C test/gtest test
- Get latest-and-gratest OpenMPI version:
$ git clone https://github.com/open-mpi/ompi.git
- Autogen:
$ cd ompi
$ ./autogen.pl
- Configure with UCX:
$ mkdir build
$ cd build
../configure --prefix=/your_install_path/ --with-ucx=/path_to_ucx_installation
- Build:
$ make
$ make install
Example of the command line (for InfiniBand RC + shared memory):
$ mpirun -np 2 -mca pml ucx -x UCX_NET_DEVICES=mlx5_0:1 -x UCX_TLS=rc,sm ./app
- By default OpenMPI enables build-in transports (BTLs), which may result in additional software overheads in the OpenMPI progress function. In order to workaround this issue you may try to disable certain BTLs.
$ mpirun -np 2 -mca pml ucx --mca btl ^vader,tcp,openib -x UCX_NET_DEVICES=mlx5_0:1 -x UCX_TLS=rc,sm ./app
- OpenMPI version https://github.com/open-mpi/ompi/commit/066370202dcad8e302f2baf8921e9efd0f1f7dfc leverages more efficient timer mechanism and there fore reduces software overheads in OpenMPI progress
- There is a know issue with ARM HPC Compiler v1.1 - #1382. In order to workaround the issue please add
CPPFLAGS="-D_GLIBCXX_HAVE_OBSOLETE_ISNAN=1"
to configure command line. Example:configure --prefix=$PWD/install CPPFLAGS="-D_GLIBCXX_HAVE_OBSOLETE_ISNAN=1" && make && make install
-
UCX current tarball: https://github.com/openucx/ucx/archive/master.zip
-
The table of MPI and OpenSHMEM distributions that are tested with the HEAD of UCX master
MPI/OpenSHMEM project | |
---|---|
OpenMPI/OSHMEM | 2.1.0 |
MPICH | Latest |
UCX can be compiled with the --with-verbs option:
--with-verbs(=DIR) Build OpenFabrics support, adding DIR/include,
DIR/lib, and DIR/lib64 to the search path for
headers and libraries
This is useful when compiling UCX on a host where the needed lib/include files aren't under /usr .
When compiling ucp_hello_world.c:
Can compile it dynamicaly:
gcc -o ucp_hello_world.dynamic ucp_hello_world.c -L${UCX_LIB} -I${UCX_INC} -lucp -lucs -luct -pedantic -Werror -Wl,-rpath,${UCX_LIB}
Or statically (For example, if UCX was compiled with: --with-verbs=/tmp/my_libs/ ):
gcc -o ucp_hello_world.static ucp_hello_world.c -Wl,--whole-archive ${UCX_LIB}/libucp.a ${UCX_LIB}/libuct.a -I${UCX_INC} ${UCX_LIB}/libucs.a ${UCX_LIB}/libucm.a /tmp/my_libs/lib/libibverbs.a /tmp/my_libs/lib/libibcm.a -Wl,--no-whole-archive -lpthread -ldl -lrt -lc -lbfd -lm -lnuma -lnl