Coarray ICAR mini-app generates runtime error in caf_send_by_ref

[rouson]

Avg response time

Defect/Bug Report

• OpenCoarrays Version: 2.1.0 e9f7d19
• Fortran Compiler: gfortran 8.1.0 (patched)
• C compiler used for building lib: gcc 8.1.0
• Installation method: install.sh
• Output of uname -a: Linux sourcery-institute-vm 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:15:38 UTC 2018 i686 i686 i686 GNU/Linux
• MPI library being used: MPICH 3.2
• Machine architecture and number of physical cores: x86 (4 cores)
• Version of CMake: 3.10.2

Observed Behavior

caf_send_by_ref prints a runtime error message

Expected Behavior

Program runs without error with GCC 6.4.0

Steps to Reproduce

git clone https://github.com/gutmann/coarray_icar
cd src/tests
export COMPILER=gnu
mkdir ../build
make
cafrun -n 4 ./test-ideal
$ cafrun -n 4 ./test-ideal
 Number of images =            4
           1 domain%initialize_from_file('input-parameters.txt')
 ximgs=           2 yimgs=           2
 call master_initialize(this)
 call this%variable%initialize(this%get_grid_dimensions(),variable_test_val)
  Layer height       Pressure        Temperature      Water Vapor
      [m]              [hPa]             [K]            [kg/kg]
   9750.00000       271.047180       206.509430       9.17085163E-06
   7750.00000       364.236938       224.725403       7.91717321E-05
   5750.00000       481.825287       243.449936       5.01311326E-04
   3750.00000       628.424316       262.669800       2.46796524E-03
   1750.00000       809.217651       282.372711       9.08217859E-03
 ThompMP: read qr_acr_qg.dat instead of computing
 qr_acr_qg initialized:   8.90000015E-02        
 ThompMP: read qr_acr_qs.dat instead of computing
 qr_acr_qs initialized:   7.19999969E-02        
 ThompMP: read freezeH2O.dat instead of computing
 freezeH2O initialized:  0.333000004            
 qi_aut_qs initialized:   1.40000004E-02        
 
 Beginning simulation...
Fortran runtime error on image 2: libcaf_mpi::caf_send_by_ref(): unable to allocate memory on remote image.

Fortran runtime error on image 4: libcaf_mpi::caf_send_by_ref(): unable to allocate memory on remote image.

Fortran runtime error on image 1: libcaf_mpi::caf_send_by_ref(): unable to allocate memory on remote image.

Fortran runtime error on image 3: libcaf_mpi::caf_send_by_ref(): unable to allocate memory on remote image.

Error: Command:
   `/home/rouson/Desktop/Builds/opt/mpich/3.2/gnu/8.1.0-patched/bin/mpiexec -n 4 --disable-auto-cleanup ./test-ideal`
failed to run.

Reproducing this result takes only seconds with the if the referenced *.dat files are in the present working directory when the run is launched. These can be provided upon request. Alternatively, the program will generate them if they don't exist. Generating them could take roughly 20-30 minutes the first time. They will then be detected and used in subsequent runs. I'm hoping @gutmann or @scrasmussen will report whether GCC 9 fixes this issue. If not, please add a comment in this issue with a short reproducer.

[zbeekman]

@rouson how big are the files? You can probably upload them here (just drag them into the comment windows)

[gutmann]

@rouson <https://github.com/rouson> how big are the files? You can probably upload them here (just drag them into the comment windows)

They are >300 MB in total

[zbeekman]

Well maybe not then, unless there are a bunch of them? I'm not sure what the GH upload limit is, but it might not be an issue, especially if there are a few of them.

[gutmann]

I think you can access them here: ftp://ftp.rap.ucar.edu/pub/gutmann/coarray_icar_inputs/ though I've had issues with this ftp site and outside access. They are also accessible here : ftp://ftp.rap.ucar.edu/incoming/irap/gutmann/coarray_icar_inputs, though that ftp site will get scrubbed in about a week.

[zbeekman]

Thanks for the FTP links. It turns out the files size limit for GH issue comment uploads is 10MB.

[zbeekman]

@gutmann Thanks to @neok-m4700 this should be fixed on master now. If you and/or @rouson can confirm that would be great!