NERSC/cori: ch4:ofi:gni hangs #6143

haampie · 2022-02-24T01:38:11Z

haampie
Feb 24, 2022

Using mpich 3.4.3, libfabric 1.14.0, ch4:ofi:gni, I'm getting an issue similar to #4720.

mpichversion:

$ ../view/bin/mpichversion 
MPICH Version:    	3.4.3
MPICH Release date:	Thu Dec 16 11:20:57 CST 2021
MPICH Device:    	ch4:ofi
MPICH configure: 	--prefix=/scratch/snx3000/hstoppel/mpich-testing/store/linux-sles15-haswell/gcc-11.2.0/mpich-3.4.3-gpfaam42ta5n4odivbv3u24ovjiolvxy --disable-silent-rules --enable-shared --with-hwloc-prefix=/scratch/snx3000/hstoppel/mpich-testing/store/linux-sles15-haswell/gcc-11.2.0/hwloc-2.7.0-gu25lfekyjxozkltn6mqrev6imqa5onh --with-pm=no --enable-romio --without-ibverbs --enable-wrapper-rpath=yes --disable-fortran --with-slurm=yes --with-slurm-include=/usr/include --with-slurm-lib=/usr/lib --with-pmi=cray --with-device=ch4:ofi:gni --with-libfabric=/scratch/snx3000/hstoppel/mpich-testing/store/linux-sles15-haswell/gcc-11.2.0/libfabric-1.14.0-dduik3sal2ao6vnhg3qmt2rlxrcjlboi --enable-libxml2
MPICH CC: 	/scratch/snx3000/hstoppel/spack/lib/spack/env/gcc/gcc    -O2
MPICH CXX: 	/scratch/snx3000/hstoppel/spack/lib/spack/env/gcc/g++   -O2
MPICH F77: 	/scratch/snx3000/hstoppel/spack/lib/spack/env/gcc/gfortran -fallow-argument-mismatch 
MPICH FC: 	/scratch/snx3000/hstoppel/spack/lib/spack/env/gcc/gfortran

perf output:

# Total Lost Samples: 0
#
# Samples: 23K of event 'cycles:u'
# Event count (approx.): 18594043750
#
# Children      Self  Command  Shared Object        Symbol                                 
# ........  ........  .......  ...................  .......................................
#
    55.31%     6.81%  bcast    libfabric.so.1.17.0  [.] __gnix_cq_sreadfrom.constprop.0
            |          
            |--53.32%--__gnix_cq_sreadfrom.constprop.0
            |          |          
            |          |--39.48%--_gnix_prog_progress
            |          |          |          
            |          |          |--26.87%--_gnix_nic_progress
            |          |          |          |          
            |          |          |          |--17.33%--__nic_tx_progress
            |          |          |          |          |          
            |          |          |          |          |--6.92%--__nic_get_completed_txd.constprop.0
            |          |          |          |          |          
            |          |          |          |          |--2.37%--GNI_CqGetEvent.part.5
            |          |          |          |          |          
            |          |          |          |           --1.93%--GNI_CqGetEvent
            |          |          |          |          
            |          |          |          |--1.80%--pthread_spin_lock
            |          |          |          |          
            |          |          |          |--1.67%--_gnix_vc_nic_progress
            |          |          |          |          
            |          |          |          |--0.93%--pthread_spin_unlock
            |          |          |          |          
            |          |          |          |--0.93%--pthread_spin_unlock@plt
            |          |          |          |          
            |          |          |           --0.89%--pthread_spin_lock@plt
            |          |          |          
            |          |          |--3.73%--_gnix_cm_nic_progress
            |          |          |          
            |          |          |--3.17%--GNI_CqTestEvent
            |          |          |          
            |          |           --1.18%--__nic_tx_progress
            |          |          
            |          |--3.35%--__pthread_rwlock_unlock
            |          |          
            |          |--2.44%--__pthread_rwlock_rdlock
            |          |          
            |           --1.04%--_gnix_cm_nic_progress
            |          
            |--0.87%--_gnix_queue_peek
            |          
             --0.60%--pthread_spin_unlock@plt

    41.92%     6.34%  bcast    libfabric.so.1.17.0  [.] _gnix_prog_progress
            |          
            |--35.58%--_gnix_prog_progress
            |          |          
            |          |--26.87%--_gnix_nic_progress
            |          |          |          
            |          |          |--17.33%--__nic_tx_progress
            |          |          |          |          
            |          |          |          |--6.92%--__nic_get_completed_txd.constprop.0
            |          |          |          |          
            |          |          |          |--2.37%--GNI_CqGetEvent.part.5
            |          |          |          |          
            |          |          |           --1.93%--GNI_CqGetEvent
            |          |          |          
            |          |          |--1.80%--pthread_spin_lock
            |          |          |          
            |          |          |--1.67%--_gnix_vc_nic_progress
            |          |          |          
            |          |          |--0.93%--pthread_spin_unlock
            |          |          |          
            |          |          |--0.93%--pthread_spin_unlock@plt
            |          |          |          
            |          |           --0.89%--pthread_spin_lock@plt
            |          |          
            |          |--3.73%--_gnix_cm_nic_progress
            |          |          
            |          |--3.17%--GNI_CqTestEvent
            |          |          
            |           --1.18%--__nic_tx_progress
            |          
             --6.34%--__gnix_cq_sreadfrom.constprop.0
                       |          
                       |--4.15%--_gnix_prog_progress
                       |          
                        --1.04%--_gnix_cm_nic_progress

    27.82%     3.47%  bcast    libfabric.so.1.17.0  [.] _gnix_nic_progress
            |          
            |--24.35%--_gnix_nic_progress
            |          |          
            |          |--17.33%--__nic_tx_progress
            |          |          |          
            |          |          |--6.92%--__nic_get_completed_txd.constprop.0
            |          |          |          
            |          |          |--2.37%--GNI_CqGetEvent.part.5
            |          |          |          
            |          |           --1.93%--GNI_CqGetEvent
            |          |          
            |          |--1.80%--pthread_spin_lock
            |          |          
            |          |--1.67%--_gnix_vc_nic_progress
            |          |          
            |          |--0.93%--pthread_spin_unlock
            |          |          
            |          |--0.93%--pthread_spin_unlock@plt
            |          |          
            |           --0.89%--pthread_spin_lock@plt
            |          
             --3.47%--__gnix_cq_sreadfrom.constprop.0
                       _gnix_prog_progress
                       |          
                        --2.99%--_gnix_nic_progress

    27.29%     0.00%  bcast    [unknown]            [.] 0000000000000000
            |
            ---0
               |          
               |--7.02%--GNI_CqGetEvent.part.5
               |          
               |--4.94%--progress_test
               |          
               |--4.56%--GNI_CqGetEvent
               |          
               |--2.55%--MPIDI_POSIX_progress
               |          
               |--2.54%--MPIDI_POSIX_eager_recv_begin
               |          
               |--1.67%--MPIDI_OFI_progress
               |          
               |--1.27%--GNII_DlaProgress@plt
               |          
               |--1.14%--MPIR_Progress_hook_exec_all
               |          
               |--1.10%--MPIDI_SHM_progress
               |          
                --0.50%--GNII_DlaProgress

    19.82%     8.02%  bcast    libfabric.so.1.17.0  [.] __nic_tx_progress
            |          
            |--11.79%--__nic_tx_progress
            |          |          
            |          |--6.92%--__nic_get_completed_txd.constprop.0
            |          |          
            |          |--2.37%--GNI_CqGetEvent.part.5
            |          |          
            |           --1.93%--GNI_CqGetEvent
            |          
             --8.02%--__gnix_cq_sreadfrom.constprop.0
                       _gnix_prog_progress
                       |          
                       |--7.26%--_gnix_nic_progress
                       |          |          
                       |          |--5.95%--__nic_tx_progress
                       |          |          
                       |          |--0.73%--pthread_spin_lock@plt
                       |          |          
                       |           --0.57%--pthread_spin_unlock@plt
                       |          
                        --0.77%--__nic_tx_progress

     9.65%     9.65%  bcast    libugni.so.0.6.0     [.] GNI_CqGetEvent.part.5
            |          
            |--7.02%--0
            |          GNI_CqGetEvent.part.5
            |          
             --2.63%--__gnix_cq_sreadfrom.constprop.0
                       _gnix_prog_progress
                       _gnix_nic_progress
                       __nic_tx_progress
                       |          
                        --2.37%--GNI_CqGetEvent.part.5

     9.28%     8.78%  bcast    libugni.so.0.6.0     [.] GNII_DlaProgress
            |
            ---GNII_DlaProgress

     7.75%     6.70%  bcast    libugni.so.0.6.0     [.] GNI_CqGetEvent
            |          
            |--5.83%--0
            |          |          
            |          |--4.56%--GNI_CqGetEvent
            |          |          
            |           --1.27%--GNII_DlaProgress@plt
            |          
            |--1.06%--GNI_CqGetEvent
            |          
             --0.87%--__gnix_cq_sreadfrom.constprop.0
                       _gnix_prog_progress
                       _gnix_nic_progress
                       __nic_tx_progress
                       GNI_CqGetEvent

     6.92%     6.66%  bcast    libfabric.so.1.17.0  [.] __nic_get_completed_txd.constprop.0
            |          
             --6.66%--__gnix_cq_sreadfrom.constprop.0
                       _gnix_prog_progress
                       _gnix_nic_progress
                       __nic_tx_progress
                       __nic_get_completed_txd.constprop.0

     6.53%     6.12%  bcast    libmpi.so.12.1.12    [.] progress_test
            |          
            |--5.31%--0
            |          |          
            |           --4.94%--progress_test
            |          
             --1.22%--progress_test

     5.37%     5.08%  bcast    libmpi.so.12.1.12    [.] MPIDI_OFI_progress
            |          
            |--3.69%--MPIDI_OFI_progress
            |          
             --1.67%--0
                       MPIDI_OFI_progress

     4.77%     3.70%  bcast    libfabric.so.1.17.0  [.] _gnix_cm_nic_progress
            |          
            |--3.70%--__gnix_cq_sreadfrom.constprop.0
            |          _gnix_prog_progress
            |          _gnix_cm_nic_progress
            |          
             --1.07%--_gnix_cm_nic_progress

     3.55%     2.77%  bcast    libpthread-2.26.so   [.] __pthread_rwlock_unlock
            |          
            |--2.57%--__gnix_cq_sreadfrom.constprop.0
            |          __pthread_rwlock_unlock
            |          
             --0.98%--__pthread_rwlock_unlock

     3.51%     3.20%  bcast    libugni.so.0.6.0     [.] GNI_CqTestEvent
            |          
             --3.20%--__gnix_cq_sreadfrom.constprop.0
                       _gnix_prog_progress
                       |          
                        --2.86%--GNI_CqTestEvent

     3.39%     2.98%  bcast    libmpi.so.12.1.12    [.] MPIDI_POSIX_progress
            |          
            |--2.63%--0
            |          |          
            |          |--2.13%--MPIDI_POSIX_progress
            |          |          
            |           --0.50%--MPIDI_POSIX_eager_recv_begin
            |          
             --0.76%--MPIDI_POSIX_progress

     2.73%     1.20%  bcast    libfabric.so.1.17.0  [.] pthread_spin_unlock@plt
            |          
            |--1.53%--pthread_spin_unlock@plt
            |          
             --0.93%--__gnix_cq_sreadfrom.constprop.0
                       _gnix_prog_progress
                       _gnix_nic_progress
                       pthread_spin_unlock

     2.55%     2.41%  bcast    libfabric.so.1.17.0  [.] _gnix_vc_nic_progress
            |          
             --2.41%--__gnix_cq_sreadfrom.constprop.0
                       _gnix_prog_progress
                       |          
                        --2.10%--_gnix_nic_progress
                                  |          
                                   --1.59%--_gnix_vc_nic_progress

     2.54%     2.04%  bcast    libmpi.so.12.1.12    [.] MPIDI_POSIX_eager_recv_begin
            |          
            |--2.04%--0
            |          MPIDI_POSIX_eager_recv_begin
            |          
             --0.50%--MPIDI_POSIX_eager_recv_begin

     2.54%     2.16%  bcast    libmpi.so.12.1.12    [.] MPIR_Progress_hook_exec_all
            |          
            |--1.77%--MPIR_Progress_hook_exec_all
            |          
             --0.77%--0
                       MPIR_Progress_hook_exec_all

     2.45%     1.30%  bcast    libfabric.so.1.17.0  [.] pthread_spin_lock@plt
            |          
            |--1.19%--__gnix_cq_sreadfrom.constprop.0
            |          _gnix_prog_progress
            |          _gnix_nic_progress
            |          pthread_spin_lock
            |          
             --1.15%--pthread_spin_lock@plt

     2.44%     2.15%  bcast    libpthread-2.26.so   [.] __pthread_rwlock_rdlock
            |          
             --2.15%--__gnix_cq_sreadfrom.constprop.0
                       __pthread_rwlock_rdlock

     2.16%     0.86%  bcast    libpthread-2.26.so   [.] pthread_spin_lock
            |          
            |--1.54%--pthread_spin_lock
            |          
             --0.61%--__gnix_cq_sreadfrom.constprop.0
                       _gnix_prog_progress
                       _gnix_nic_progress
                       pthread_spin_lock

     1.77%     0.50%  bcast    libugni.so.0.6.0     [.] GNII_DlaProgress@plt
            |          
            |--1.27%--GNII_DlaProgress@plt
            |          
             --0.50%--0
                       GNII_DlaProgress

     1.74%     0.88%  bcast    libfabric.so.1.17.0  [.] _gnix_queue_peek
            |          
            |--0.88%--__gnix_cq_sreadfrom.constprop.0
            |          
             --0.87%--_gnix_queue_peek

     1.59%     0.39%  bcast    libpthread-2.26.so   [.] pthread_spin_unlock
            |          
             --1.20%--pthread_spin_unlock

     1.52%     1.52%  bcast    libmpi.so.12.1.12    [.] MPIDI_SHM_progress
            |
            ---0
               |          
                --1.10%--MPIDI_SHM_progress

     1.27%     0.78%  bcast    libfabric.so.1.17.0  [.] pthread_rwlock_unlock@plt
            |          
             --0.78%--__gnix_cq_sreadfrom.constprop.0
                       __pthread_rwlock_unlock

     1.06%     1.06%  bcast    libfabric.so.1.17.0  [.] GNI_CqGetEvent@plt
            |
            ---__gnix_cq_sreadfrom.constprop.0
               _gnix_prog_progress
               _gnix_nic_progress
               __nic_tx_progress
               GNI_CqGetEvent

     0.77%     0.77%  bcast    libmpi.so.12.1.12    [.] MPID_Progress_wait
     0.74%     0.74%  bcast    libmpi.so.12.1.12    [.] MPIDI_OFI_get_buffered
     0.47%     0.29%  bcast    libfabric.so.1.17.0  [.] pthread_rwlock_rdlock@plt
     0.33%     0.33%  bcast    libfabric.so.1.17.0  [.] gnix_cq_read
     0.31%     0.31%  bcast    libfabric.so.1.17.0  [.] GNI_CqTestEvent@plt
     0.01%     0.01%  bcast    libfabric.so.1.17.0  [.] _gnix_dgram_poll
     0.01%     0.01%  bcast    libugni.so.0.6.0     [.] GNI_PostDataProbeById
     0.01%     0.01%  bcast    [unknown]            [k] 0xffffffff81800b40

mpich@3.4.3%gcc@11.2.0~argobots~fortran+hwloc~hydra+libxml2+pci+romio+slurm~two_level_namespace~verbs+wrapperrpath device=ch4 netmod=ofi pmi=cray arch=linux-sles15-haswell
    ^cray-pmi@5.0.17%gcc@11.2.0 arch=linux-sles15-haswell
    ^findutils@4.8.0%gcc@11.2.0 patches=440b954 arch=linux-sles15-haswell
    ^hwloc@2.7.0%gcc@11.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-sles15-haswell
        ^libpciaccess@0.16%gcc@11.2.0 arch=linux-sles15-haswell
            ^libtool@2.4.6%gcc@11.2.0 arch=linux-sles15-haswell
                ^m4@1.4.19%gcc@11.2.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-sles15-haswell
                    ^libsigsegv@2.13%gcc@11.2.0 arch=linux-sles15-haswell
            ^pkgconf@1.8.0%gcc@11.2.0 arch=linux-sles15-haswell
            ^util-macros@1.19.3%gcc@11.2.0 arch=linux-sles15-haswell
        ^libxml2@2.9.12%gcc@11.2.0~python arch=linux-sles15-haswell
            ^libiconv@1.16%gcc@11.2.0 libs=shared,static arch=linux-sles15-haswell
            ^xz@5.2.5%gcc@11.2.0~pic libs=shared,static arch=linux-sles15-haswell
            ^zlib@1.2.11%gcc@11.2.0+optimize+pic+shared arch=linux-sles15-haswell
        ^ncurses@6.2%gcc@11.2.0~symlinks+termlib abi=none arch=linux-sles15-haswell
    ^libfabric@1.14.0%gcc@11.2.0~debug~kdreg fabrics=gni arch=linux-sles15-haswell
    ^slurm@20-11-8-1%gcc@11.2.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-sles15-haswell

#include "stdio.h"
#include "stdlib.h"
#include "mpi.h"

int main() {
  size_t num_bytes = 16;
  size_t num_trials = 100000;
  double total_time = 0.0;
  int world_rank;
  char* data = malloc(num_bytes);
  if (data == NULL) return 1;

  MPI_Init(NULL, NULL);
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

  for (size_t i = 0; i < num_trials; i++) {
    if (world_rank == 0) {
        printf("iteration = %zu\n", i);
        fflush(stdout);
    }
    MPI_Barrier(MPI_COMM_WORLD);
    if (i > 10)
        total_time -= MPI_Wtime();
    MPI_Bcast(data, num_bytes, MPI_CHAR, 0, MPI_COMM_WORLD);
    MPI_Barrier(MPI_COMM_WORLD);
    if (i > 10)
        total_time += MPI_Wtime();
  }

  if (world_rank == 0)
      printf("%f\n", total_time);

  free(data);
  MPI_Finalize();
}

bcast: bcast.c
	env -i PATH=/usr/bin:/bin /opt/gcc/11.2.0/bin/gcc -o $@ -march=haswell -O3 -I../view/include $< -L../view/lib -lmpi -Wl,-rpath,$(CURDIR)/../view/lib

$ make bcast
$ srun -N2 -n2 --pty ./bcast
iteration = 0
...
iteration = 4448
iteration = 4449
# hangs...

haampie · 2022-02-24T01:42:43Z

haampie
Feb 24, 2022
Author

Oh, I got the impression #5085 was targeted at 3.x, but it only landed in 4.0? I can try mpich 4.x soon.

0 replies

haampie · 2022-02-24T02:02:52Z

haampie
Feb 24, 2022
Author

Issue persists on mpich 4.0.1 :( exactly as above.

$ ../view/bin/mpichversion 
MPICH Version:    	4.0.1
MPICH Release date:	Tue Feb 22 16:37:51 CST 2022
MPICH Device:    	ch4:ofi
MPICH configure: 	--prefix=/scratch/snx3000/hstoppel/mpich-testing/store/linux-sles15-haswell/gcc-11.2.0/mpich-4.0.1-cms67ilsebo4zbm6scx65bxxs64iooh7 --disable-silent-rules --enable-shared --with-hwloc-prefix=/scratch/snx3000/hstoppel/mpich-testing/store/linux-sles15-haswell/gcc-11.2.0/hwloc-2.7.0-gu25lfekyjxozkltn6mqrev6imqa5onh --with-pm=no --enable-romio --without-ibverbs --enable-wrapper-rpath=yes --disable-fortran --with-slurm=yes --with-slurm-include=/usr/include --with-slurm-lib=/usr/lib --with-pmi=cray --with-device=ch4:ofi:gni --with-libfabric=/scratch/snx3000/hstoppel/mpich-testing/store/linux-sles15-haswell/gcc-11.2.0/libfabric-1.14.0-dduik3sal2ao6vnhg3qmt2rlxrcjlboi --enable-libxml2
MPICH CC: 	/scratch/snx3000/hstoppel/spack/lib/spack/env/gcc/gcc    -O2
MPICH CXX: 	/scratch/snx3000/hstoppel/spack/lib/spack/env/gcc/g++   -O2
MPICH F77: 	/scratch/snx3000/hstoppel/spack/lib/spack/env/gcc/gfortran -fallow-argument-mismatch 
MPICH FC: 	/scratch/snx3000/hstoppel/spack/lib/spack/env/gcc/gfortran  
MPICH Custom Information:

0 replies

hzhou · 2022-04-08T21:59:30Z

hzhou
Apr 8, 2022
Maintainer

I am trying this on Cori with the current main branch, but I can't even get past the init:

nid00094 ~/pull_requests/mpich-main$ FI_PROVIDER=gni ./cpi
Abort(471977871): Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(45413).....: MPI_Init(argc=0x7fffffff718c, argv=0x7fffffff7180) failed
MPII_Init_thread(208)....:
MPID_Init(448)...........:
MPIDI_OFI_init_local(651):
create_vni_context(1047).: OFI endpoint open failed (ofi_init.c:1047:create_vni_context:Invalid argument)

Turning on FI_LOG_LEVEL=debug shows:

libfabric:38999:gni:core:_gnix_ref_init():254<debug> [38999:1] 0x6e9878 refs 1
libfabric:38999:core:core:fi_fabric():1257<info> Opened fabric: gni
libfabric:38999:gni:domain:gnix_domain_open():579<trace> [38999:1]
libfabric:38999:gni:fabric:gnix_domain_open():591<info> [38999:1] failed to find authorization key, creating new authorization key
libfabric:38999:gni:fabric:__gnix_ccm_init():171<debug> [38999:1] Reading job info file /tmp/ccm_alps_info
libfabric:38999:gni:fabric:__gnix_alps_init():313<warn> [38999:1] lli get response failed, alps_status=5(Success)
libfabric:38999:gni:fabric:gnixu_get_rdma_credentials():437<warn> [38999:1] __gnix_app_init() failed, ret=-5(Success)
libfabric:38999:gni:domain:_gnix_auth_key_enable():347<info> [38999:1] pkey=617a6972 ptag=111 key_partition_size=0 key_offset=0 enabled
libfabric:38999:gni:domain:gnix_domain_open():597<info> [38999:1] authorization key=0x73bbd0 ptag 111 cookie 0x617a6972
libfabric:38999:gni:core:gnix_domain_open():652<debug> [38999:1] 0x6e9878 refs 2
libfabric:38999:gni:core:_gnix_ref_init():254<debug> [38999:1] 0x73b690 refs 1
libfabric:38999:gni:mr:_gnix_auth_key_enable():354<debug> [38999:1] authorization key already enabled, auth_key=0x73bbd0
libfabric:38999:gni:av:gnix_av_open():879<trace> [38999:1]
libfabric:38999:gni:av:gnix_av_open():879<trace> [38999:1]
libfabric:38999:gni:core:_gnix_ref_init():254<debug> [38999:1] 0x73f840 refs 1
libfabric:38999:gni:cq:gnix_cntr_open():433<trace> [38999:1]
libfabric:38999:gni:cq:__verify_cntr_attr():65<trace> [38999:1]
libfabric:38999:gni:core:_gnix_ref_init():254<debug> [38999:1] 0x73acc8 refs 1
libfabric:38999:gni:core:gnix_cntr_open():463<debug> [38999:1] 0x73b690 refs 2
libfabric:38999:gni:eq:gnix_cntr_set_wait():96<trace> [38999:1]
libfabric:38999:gni:cq:gnix_cq_open():656<trace> [38999:1]
libfabric:38999:gni:cq:verify_cq_attr():140<trace> [38999:1]
libfabric:38999:gni:core:_gnix_ref_init():254<debug> [38999:1] 0x6e9930 refs 1
libfabric:38999:gni:core:gnix_cq_open():695<debug> [38999:1] 0x73b690 refs 3
libfabric:38999:gni:cq:gnix_cq_set_wait():193<trace> [38999:1]
libfabric:38999:gni:ep_ctrl:gnix_ep_open():2300<trace> [38999:1]
libfabric:38999:gni:core:_gnix_ref_init():254<debug> [38999:1] 0x73d0d8 refs 1
libfabric:38999:gni:ep_ctrl:gnix_ep_open():2356<debug> [38999:1] ep(0x73cd60) is using addr_format(FI_ADDR_GNI)
libfabric:38999:gni:ep_ctrl:__init_tag_storages():2097<info> [38999:1] initializing tag storage, tag_type=1
libfabric:38999:gni:ep_ctrl:_gnix_xpmem_handle_create():238<trace> [38999:1]
libfabric:38999:gni:core:_gnix_ref_init():254<debug> [38999:1] 0x73c530 refs 1
libfabric:38999:gni:ep_ctrl:_gnix_cm_nic_alloc():609<trace> [38999:1]
libfabric:38999:gni:ep_ctrl:_gnix_resolve_gni_ep_name():121<trace> [38999:1]
libfabric:38999:gni:ep_ctrl:_gnix_cm_nic_alloc():628<info> [38999:1] creating cm_nic for 111/0x617a6972/8744961
libfabric:38999:gni:ep_ctrl:gnix_nic_alloc():954<trace> [38999:1]
libfabric:38999:gni:ep_ctrl:gnix_nic_alloc():1059<warn> [38999:1] GNI_CdmAttach returned GNI_RC_PERMISSION_ERROR
libfabric:38999:gni:fabric:_gnix_dump_gni_res():729<warn> [38999:1] Device Resources:
dev res:       MDD, avail: 4086 res: 409 held: 0 total: 4095
dev res:        CQ, avail: 2039 res: 10 held: 0 total: 2047
dev res:       FMA, avail: 126 res: 4 held: 0 total: 127
dev res:        CE, avail: 4 res: 0 held: 0 total: 4
dev res:       DLA, avail: 16384 res: 1024 held: 0 total: 16384
dev res:       TCR, avail: 64864 res: 0 held: 0 total: 16
dev res:       DVA, avail: 4398046511104 res: 1099511627776 held: 0 total: 4398046511104
dev res:      VMDH, avail: 4 res: 0 held: 0 total: 4
libfabric:38999:gni:fabric:_gnix_dump_gni_res():745<warn> [38999:1] Job Resources:
libfabric:38999:gni:ep_ctrl:_gnix_cm_nic_alloc():649<warn> [38999:1] gnix_nic_alloc returned Permission denied
libfabric:38999:gni:ep_ctrl:_gnix_ep_nic_init():2193<warn> [38999:1] _gnix_cm_nic_alloc returned Permission denied
libfabric:38999:gni:ep_ctrl:gnix_ep_open():2409<warn> [38999:1] _gnix_ep_nic_init returned -13
libfabric:38999:gni:ep_ctrl:_gnix_xpmem_handle_destroy():315<trace> [38999:1]
libfabric:38999:gni:core:_gnix_xpmem_handle_destroy():317<debug> [38999:1] 0x73c530 refs 0
libfabric:38999:gni:ep_ctrl:__xpmem_hndl_destruct():103<trace> [38999:1]
libfabric:38999:gni:ep_ctrl:__destruct_tag_storages():1600<info> [38999:1] destroying tag storage
Abort(1008848783): Fatal error in internal_Init: Other MPI error, error stack:

Quite mysterious to me. I'll try static configuration (--with-device=ch4:ofi:gni) next.

EDIT: hmm, same. There must be some system setting I am not getting it right.

nid00084 ~/pull_requests/mpich-main$ MPIR_CVAR_DEBUG_SUMMARY=1 ./cpi
==== Various sizes and limits ====
sizeof(MPIDI_per_vci_t): 192
Required minimum FI_VERSION: 10005, current version: 1000d
==== Capability set configuration ====
libfabric provider: gni - gni
MPIDI_OFI_ENABLE_AV_TABLE: 1
MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0
MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 1
MPIDI_OFI_ENABLE_MR_ALLOCATED: 1
MPIDI_OFI_ENABLE_MR_PROV_KEY: 1
MPIDI_OFI_ENABLE_TAGGED: 0
MPIDI_OFI_ENABLE_AM: 1
MPIDI_OFI_ENABLE_RMA: 1
MPIDI_OFI_ENABLE_ATOMICS: 1
MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
MPIDI_OFI_ENABLE_HMEM: 0
MPIDI_OFI_NUM_AM_BUFFERS: 8
MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0
MPIDI_OFI_CONTEXT_BITS: 16
MPIDI_OFI_SOURCE_BITS: 24
MPIDI_OFI_TAG_BITS: 20
MPIDI_OFI_VNI_USE_DOMAIN: 1
MAXIMUM SUPPORTED RANKS: 16777216
MAXIMUM TAG: 1048576
==== Provider global thresholds ====
max_buffered_send: 64
max_buffered_write: 64
max_msg_size: 4294967295
max_order_raw: 0
max_order_war: 0
max_order_waw: 0
tx_iov_limit: 8
rx_iov_limit: 8
rma_iov_limit: 1
max_mr_key_size: 8
==== Various sizes and limits ====
MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
MPIDI_OFI_MAX_AM_HDR_SIZE: 255
sizeof(MPIDI_OFI_am_request_header_t): 416
sizeof(MPIDI_OFI_per_vni_t): 51776
MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
Abort(337760143): Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(59)........: MPI_Init(argc=0x7fffffff717c, argv=0x7fffffff7170) failed
MPII_Init_thread(208)....:
MPID_Init(448)...........:
MPIDI_OFI_init_local(651):
create_vni_context(1047).: OFI endpoint open failed (ofi_init.c:1047:create_vni_context:Invalid argument)

0 replies

hzhou · 2022-08-30T16:25:07Z

hzhou
Aug 30, 2022
Maintainer

I accidentally converted this issue to discussion thinking that this issue is specific for cori. But It is really an issue for running MPICH with gni.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NERSC/cori: ch4:ofi:gni hangs #6143

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

NERSC/cori: ch4:ofi:gni hangs #6143

haampie Feb 24, 2022

Replies: 4 comments

haampie Feb 24, 2022 Author

haampie Feb 24, 2022 Author

hzhou Apr 8, 2022 Maintainer

hzhou Aug 30, 2022 Maintainer

haampie
Feb 24, 2022

haampie
Feb 24, 2022
Author

haampie
Feb 24, 2022
Author

hzhou
Apr 8, 2022
Maintainer

hzhou
Aug 30, 2022
Maintainer