Skip to content

Getting Started Guide

Jump to bottom
Kieran Holland edited this page Feb 6, 2023 · 14 revisions

Getting Started Guide

Build Sandia-OpenSHMEM (SOS) Libraries

  • This project utilizes the GNU Autotools build system to generate SOS’s Makefiles, and which will determine where the installation files are built.
    • If you are downloading a release, the configure file which generates the Makefiles will already be available.
    • If you’re cloning the SOS source code you’ll need to generate a configure script using Autotools:
      • The GNU Autotools packages can be installed with the Linux package manager. For example, on Ubuntu this can be done by executing the following command: $ apt install automake autoconf libtool
  • To maintain a more organized build area, it may be useful to create a separate build and install directory. Here are the steps to do so for each of the three libraries you must build:
$ cd <top-level-of-repo>
$ mkdir build install
$ cd build
$ ../configure --prefix=<path-to-install-dir> …<other-options>…
$ make
$ make install
  • The configure options required for building can be found under the subheadings for how to build SOS and each of its dependencies.

Clone Sandia-OpenSHMEM Repository

The Sandia OpenSHMEM repository is located here: https://github.com/Sandia-OpenSHMEM/SOS.

To clone it:

$ git clone https://github.com/Sandia-OpenSHMEM/SOS.git

Clone and Build OFI Libfabric

  • https://github.com/ofiwg/libfabric
  • $ git clone https://github.com/ofiwg/libfabric.git
  • If using Intel® Omni Path, follow the steps in the Intel Omni Path 100 (PSM2) Build Instructions wiki on building OFI libfabric:
  • It may be required to disable providers that will not be utilized in a given configuration (for example to disable the efa provider): --disable-efa
  • One can check their providers by running: $ fi_info in the terminal once the SOS binaries are in the environment PATH.

Download/Build a Compatible Process Launcher

  • Users will need a compatible process launcher in order to run an OpenSHMEM program.
  • The script that SOS uses to run OpenSHMEM programs, oshrun, picks up up the following launchers in this order of preference: yod, mpirun, mpiexec, and srun.
  • SOS may work with other process launchers, but it only tests/guarantees compatibility with the Hydra Process Manager (e.g. mpirun and mpiexec).

Build and install SOS

Configuration Tips

  • To list all available options of configure (including which options are enabled by default), run the following command:
    • $ configure --help
  • An easy way to check what the last configuration ran was is to look at the top of the config.log file (which is dumped after the configure script is run):
    • $ head config.log
  • To rerun the last configuration with the same options as the last time it was executed run:
    • $ ./config.status --recheck

Test Sandia-OpenSHMEM (SOS) Libraries

  • In order to compile and run an OpenSHMEM program one must first set the environment PATH to the bin directories which contain the following:
    • the SOS binaries - which allow one to invoke the compiler and the running script
    • the hydra binaries – which contain the hydra process manager invoked by the SOS run script
    • An example of setting the environment PATH on bash:
      • $ export PATH=<path-to-SOS-install>/bin:<path-to-hydra-install>/bin:$PATH
  • Navigate to the directory where the build was configured and run:
    • $ make check

Compiling an OpenSHMEM Program

  • To compile an OpenSHMEM application, invoke one of the Sandia-OpenSHMEM (SOS) compilers:
    • oshcc – for c files
    • oshc++ or oshCC – for c++ files
    • oshfort – for fortran files
    • Example for a OpenSHMEM program written in C:
      • $ oshcc hello.c -o hello
  • These binaries can be found in the bin directory within the SOS install area. It is convenient to set the environment PATH to this bin directory so that these compilers can be invoked from anywhere without having to pass the full path:
    • An example of setting the environment PATH on bash:
      • $ export PATH=<path-to-SOS-install>/bin:$PATH

Running an OpenSHMEM Program

  • Once the program has been successfully compiled the application can be run using the OpenSHMEM launcher—oshrun oshrun, requires a minimum of two arguments:
    • -n - which tells the script how many threads to launch
    • <path-to-the-binary> - which tells the script the location of the binary it will be executing
  • An example OpenSHMEM program can be launched with the following command:
    • $ oshrun -n 10 ./hello
    • where 10 is the number of processes to launch, and
    • ./hello is the path to the binary which will be executed
  • The oshrun binary can be found in the bin directory within the SOS install area.
    • When invoking this script, it will pick up up the following launchers in this order of preference: yod, mpirun, mpiexec, and srun.
    • This implementation only tests/guarantees compatibility with the Hydra Process Manager (e.g. mpirun and mpiexec).
  • It is convenient to set the environment PATH to the directories where oshrun and the process launcher binaries are installed so that the executables can be invoked from anywhere without having to pass the full path.
    • An example of setting the environment PATH on bash:
      • $ export PATH=<path-to-SOS-install>/bin:<path-to-hydra-install>/bin:$PATH

Example OpenSHMEM Code

Hello World Example

#include <shmem.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char* argv[], char *envp[])
{
    int me, myshmem_n_pes;
    /*
    ** Starts/Initializes SHMEM/OpenSHMEM
    */
    shmem_init();
    /*
    ** Fetch the number or processes
    ** Some implementations use num_pes();
    */
    myshmem_n_pes = shmem_n_pes();
    /*
    ** Assign my process ID to me
    */
    me = shmem_my_pe();

    printf("Hello World from %d of %d\n", me, myshmem_n_pes);

    shmem_finalize();

    return 0;  
}
  • This simple “Hello World” example highlights OpenSHMEM’s Single Program Multiple Data (SPMD) functionalities by having each Processing Element (PE) perform the same task while garnering a unique result. Here, each process will display the message “Hello World from” followed by that PE’s unique identifier.
  • In order to use OpenSHMEM routines a program must bring in the header file “shmem.h”
    • #include <shmem.h>
  • Every OpenSHMEM API call must happen after the OpenSHMEM runtime is initialized and before it is finalized.
    • The OpenSHMEM runtime is initialized with: shmem_init()
    • It is finalized with: shmem_finalized()
  • shmem_n_pes() will return the total number of PEs running in the program.
    • Note: This is determined by an argument passed to the launcher which dictates how many processes to launch this program with.
  • shmem_my_pe() will return the unique identifier for whichever PE has called the function.
  • There is no guarantee as to how long each PE will take to execute the program, therefore, each PE may print their messages in any order.
  • The result of this example should resemble the following output:
$ oshrun -n 10 ./hello
Hello World from 6 of 10
Hello World from 8 of 10
Hello World from 9 of 10
Hello World from 0 of 10
Hello World from 1 of 10
Hello World from 2 of 10
Hello World from 3 of 10
Hello World from 4 of 10
Hello World from 5 of 10
Hello World from 7 of 10

Pi Example

#include <shmem.h> 
#include <stdio.h> 
#include <stdlib.h> 
#include <assert.h> 
#include <time.h> 
#include <math.h> 
 
#define NUM_POINTS 10000 
 
long long inside = 0, total = 0; 
 
int main(int argc, char* argv[], char *envp[]) 
{
    int me, myshmem_n_pes;
    /*
    ** Starts/Initializes SHMEM/OpenSHMEM
    */
    shmem_init();
    /*
    ** Fetch the number or processes
    ** Some implementations use num_pes();
    */
    myshmem_n_pes = shmem_n_pes();
    /*
    ** Assign my process ID to me
    */
    me = shmem_my_pe();

    srand(1+me);

    for(total = 0; total < NUM_POINTS; ++total) {
        double x,y;
        x = rand()/(double)RAND_MAX;
        y = rand()/(double)RAND_MAX;

        if(x*x + y*y < 1) {
            ++inside;
        }
    }

    shmem_barrier_all();

    if(me == 0) {
        for(int i = 1; i < myshmem_n_pes; ++i) {
            long long remoteInside,remoteTotal;
            shmem_longlong_get(&remoteInside,&inside,1,i);
            shmem_longlong_get(&remoteTotal,&total,1,i);
            total += remoteTotal;
            inside += remoteInside;
        }

        double approx_pi = 4.0*inside/(double)total;

        printf("Pi from %llu points on %d PEs: %lf\n", total, myshmem_n_pes, approx_pi);
    }

    shmem_finalize();

    return 0;
}
  • This next sample, pi.c, calculates the value of pi by generating a variable number of random values, determining whether they exist within a unit circle, and then multiplying the ratio of values inside the unit circle to those outside of it by four to get an approximate value of pi.
  • Like the Hello World Sample, all the OpenSHMEM API calls are made between the calls to shmem_init() and shmem_finalize().
  • The total number of PE’s along with each PE’s unique identifier are also collected in the same manner as the earlier sample—shmem_n_pes() and shmem_my_pe(), respectively.
  • The first for loop is used to generate and count the coordinates within the unit circle.
  • Next, shmem_barrier_all() is invoked. This routine will synchronize all of the PEs in the world team by blocking the calling PE until all PEs have called shmem_barrier_all().
  • Then, using shmem_longlong_get(), the PE with id zero will get the values for both inside and total from the remote memory space of the other PEs and accumulate them all into its own inside and total variables. These accumulated values are then used as a part of the formula to approximate pi.
  • The result of this example should resemble the following output:
$ oshrun -n 10 ./pi
Pi from 100000 points on 10 PEs: 3.147560

Pi Reduction Example

  • As an exercise, update the Pi example to replace the second for loop with an OpenSHMEM sum reduction routine (section 9.9.9 in the OpenSHMEM Specification v1.5) to accumulate the inside and total values.
  • Note: The solution to this exercise can be found in pi_reduce.c example in the examples directory.
Clone this wiki locally