Matt's Titan notes

Matt Baker's notes on getting UCX working on titan

On titan, your scratch space is separated by project name. To make the below instructions work you will need to set a project name variable or edit the code to have your project name instead.

export PROJNAME= #put project name here. Should show up possible names with 'ls $MEMBERWORK'

download sources to $MEMBERWORK/$PROJNAME/ucx-work/ ((TODO: Put source links here))

First, swap the PGI programming environment for the GNU programming environment. PGI may work, but is not tested currently.

module swap PrgEnv-pgi PrgEnv-gnu

Some versions of GCC are broken. Version 4.9.1 is known to work fine, so swap out for that version.

module swap gcc gcc/4.9.1

librte:

./configure --prefix=$MEMBERWORK/$PROJNAME/ucx-work/librt --with-pmi=/opt/cray/pmi/default/ --with-pmi-lib=/opt/cray/pmi/default/lib64/ --with-pmi-include=/opt/cray/pmi/default/include/  
make  
make install

gtest:
Make sure that gtest is in $MEMBERWORK/$PROJNAME/ucx-work/. These instructions assume gtest-1.7.0 so gtest will be built in $MEMBERWORK/$PROJNAME/ucx-work/gtest-1.7.0

./configure
make

UCX build: Note that in the instructions below your gtest and librte maybe someplace else than described here, especially if you use a different version of gtest.

./configure --prefix=$MEMBERWORK/$PROJNAME/ucx-work/ --enable-instrumentation --enable-frame-pointer --enable-stats --enable-memtrack --enable-fault-injection --enable-debug-data --without-verbs --with-rte=$MEMBERWORK/$PROJNAME/ucx-work/librt/ --enable-gtest=$MEMBERWORK/$PROJNAME/ucx-work/gtest-1.7.0/ CC=cc CXX=CC LDFLAGS="-dynamic"
make

running:

qsub -I -A (account) -q debug -l nodes=1,walltime=30:00
(Wait for interactive shell)
cd $MEMBERWORK/$PROJNAME/ucx-work/ucx/
make -C test/gtest test LAUNCHER="aprun -n 1 -cc none" UCS_HANDLE_ERRORS="debug"