Test failure building against hdf5 1.10.0

[opoplawski]

Please provide as much of the following information as you can, as applicable to the issue being reported. Naturally, not all information is relevant to every issue, but the more information we have to start, the better!

Environment Information

Feel free to skip this if the issue is related to documentation, a feature request, or general discussion.

• What platform are you using? (please provide specific distribution/version in summary)
  • [X ] Linux
  • Windows
  • OSX
  • Other
  • NA
• 32 and/or 64 bit?
  • [ X] 32-bit
  • [ X] 64-bit
• What build system are you using?
  • [ X] autotools (configure)
  • cmake
• Can you provide a sample netCDF file or C code to recreate the issue?
  • [ X] Yes (please attach to this issue, thank you!)
  • No
  • Not at this time

Summary of Issue

Building netcdf 4.4.0 against hdf5 1.10.0 I get the following test failure:

 cat build/h5_test/test-suite.log
==========================================
   netCDF 4.4.0: h5_test/test-suite.log
==========================================
# TOTAL: 26
# PASS:  25
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
.. contents:: :depth: 2
FAIL: tst_h_dimscales
=====================
HDF5-DIAG: Error detected in HDF5 (1.10.0) thread 0:
  #000: ../../src/H5Gdeprec.c line 1092 in H5Gget_objtype_by_idx(): can't get object type
    major: Symbol table
    minor: Inappropriate type
  #001: ../../src/H5Gdeprec.c line 1144 in H5G_obj_get_type_by_idx(): can't locate type
    major: Symbol table
    minor: Object not found
  #002: ../../src/H5Gstab.c line 1219 in H5G__stab_get_type_by_idx(): index out of bound
    major: Invalid arguments to routine
    minor: Inappropriate type
*** Checking HDF5 dimension scales.
*** Creating simple dimension scales file...ok.
*** Checking that simple dimscale file can be read...ok.
*** Creating simple dimension scales file with lots of datasets...ok.
*** Creating a file with an unlimited dimension scale...ok.
*** Checking that unlimited dimscale file can be read...Sorry! Unexpected result, ../../h5_test/tst_h_dimscales.c, line: 387
FAIL tst_h_dimscales (exit status: 2)

Steps to reproduce the behavior

See https://copr.fedorainfracloud.org/coprs/orion/hdf5110/build/172669/

[opoplawski]

I've also reported to hdf5 folks. Not sure who is to blame.

[DennisHeimbigner]

You should be aware that HDF5 1.10 is not supported (yet) for netcdf-c.
It is apparently alpha. Also, its format is (according to the hdf group) incompatible
with version 1.8.

[opoplawski]

Here is the response from the hdf5 folks:

It looks like netCDF needs to update their tests to use H5Oget_info in
place of H5Gget_objtype_by_idx.

The H5Gget_objtype_by_idx function was deprecated in HDF5-1.8 and
was removed in the HDF5-1.10 APIs. Here is the documentation page
for H5Gget_objtype_by_idx:

https://www.hdfgroup.org/HDF5/doc/RM/RM_H5G.html#Group-GetObjTypeByIdx

hdf5 1.10 is officially released now.

[WardF]

Thanks for sharing your response from the HDF Group. Since the HDF5 1.10 release changed the underlying format, we're going to have to give some thought about how to move forward adopting the HDF5 1.10 feature set. For now, we're going to require the 1.8 libraries until we have a chance to evaluate things. I see from the 1.10 release that they are providing a way to retain the 1.8 API (which is good), but I'm wondering if we'll need to specify a new file format type, to differentiate between netCDF4 files created with 1.8 and netCDF4 files created with 1.10. Almost certainly this will be the case when we leverage the new 1.10 features, but what about files created using the 1.10 library via the 1.8 API?

Anyways, more thought is needed; for now, I recommend staying with hdf5 1.8 for netcdf.

[opoplawski]

I'm still getting this with 4.4.1-rc1:

FAIL: tst_h_dimscales
=====================
HDF5-DIAG: Error detected in HDF5 (1.10.0) thread 0:
  #000: ../../src/H5Gdeprec.c line 1092 in H5Gget_objtype_by_idx(): can't get object type
    major: Symbol table
    minor: Inappropriate type
  #001: ../../src/H5Gdeprec.c line 1144 in H5G_obj_get_type_by_idx(): can't locate type
    major: Symbol table
    minor: Object not found
  #002: ../../src/H5Gstab.c line 1219 in H5G__stab_get_type_by_idx(): index out of bound
    major: Invalid arguments to routine
    minor: Inappropriate type
*** Checking HDF5 dimension scales.
*** Creating simple dimension scales file...ok.
*** Checking that simple dimscale file can be read...ok.
*** Creating simple dimension scales file with lots of datasets...ok.
*** Creating a file with an unlimited dimension scale...ok.
*** Checking that unlimited dimscale file can be read...Sorry! Unexpected result, ../../h5_test/tst_h_dimscales.c, line: 387
FAIL tst_h_dimscales (exit status: 2)

[WardF]

Ok, I've recreated this and am sorting it out. I had planned on treating it in a straight-forward manner; check for the existence of the deprecated function calls and only run the tests if they were found. I've now observed something strange, however; the test fails when building with autoconf, as reported. Building with cmake, however, results in the same test passing.

I haven't started tracking down what's going on; I wanted to make a note of it before progressing.

[WardF]

Ok. The comments above regarding replacing the deprecated symbols all hold true. However, they don't seem to be the problem here.

In tst_h_dimscales.c:574, we call H5Get_num_objs() to determine the number of objects; this results in num_obj = 5. We then iterate through the objects using a standard i = 0; i < num_obj; i++ loop.

In a cmake-based build, the loop exits after 5 iterations, as expected.

In an autotools-based build, we run into problems when i == 2. Having set a watchpoint on num_obj, we see the value is changed in the function alien_visitor by the call `H5Iget_name(did,name1,STR_LEN).

This suggests there is a memory problem somewhere. My next avenue of investigation will be to recompile libhdf5-1.10.0 with --enable-using-memchecker and see if more information presents itself. I'll also run it through the (val)grinder and see what pops out. I won't rule out the memory issue is in the test itself at this point, but it doesn't feel like it will be. Assuming it's a memory issue in the first place.

[WardF]

Compiling netcdf-c with autotools as follows eliminates the error observed in tst_h_dimscales:

$ CFLAGS="-g -Wall -Wconversion" LDFLAGS="-Wl,--no-undefined" ./configure --enable-extra-tests

These settings mimic the default cmake compiler flags. The default flags used by configure are simply -g -O2.

Merely using CFLAGS=-O0 is enough to correct the tst_h_dimscales issue, which would seem to confirm this is an optimization issue.

Note that if hdf5 is built in debug mode, we see a bevy of additional failures when running make check in netcdf-c. This is concerning and suggests that 1.10.0 may not be ready for use with netcdf, file format issues aside.

Investigation is ongoing.

[WardF]

Edit: gcc below refers to gcc-4.8, the default on my linux system. I've tested against gcc-4.7, gcc-4.4; see the subsequent comment for details.

Summary of findings thus-far

• The optimization level is the key independent variable, not the configuration system.
• gcc + O2: Failure in tst_h_dimscales when a variable, num_obj, becomes garbage on the third iteration of a loop.
• gcc + O0: tst_h_dimscales, all other tests pass.
• clang + O2: No failures.
• clang + O0: No failures.

HDF5-twiddling observations:

• Building hdf5 1.10.0 in debug mode, we see a number of other netcdf-c tests fail, for example tst_h_enums. The failure here is when free() is called on memory that has been passed back from hdf5 via H5Tget_member_name(). Note that this function, according to the HDF5 API documentation, returns memory which must be subsequently freed by the caller, so our invocation of free() should not be causing it to crash.

Conclusion

At this point, I'm not sure there there's much we can do on our end about this other than change the default optimization level used by autotools/configure. We must eventually update these tests to move away from the deprecated functions (as outlined above), but our use of deprecated functions in these tests do not seem to be causing any problems. Rather, something with how the code is being optimized with O2 is resulting in a garbage loop-control variable, halfway through iterating through the loop, resulting in the bounds of an array being overrun.

[WardF]

Further summary of compiling netcdf-c, linking against libhdf5 1.10.0.

• gcc-4.7, gcc-4.8 + O2: Failure in tst_h_dimscales as previously observed.
• gcc-4.4 + O2: tst_h_dimscales, no error observed.

[WardF]

I've modified configure.ac so that -g -O2 are not the default CFLAGS, until such time as we can determine what's going on. I'll be merging in branch gh244 as soon as the travis-ci checks finish, at which point I'll tag this to revisit and close it out for now.

[WardF]

The fix (for now) has been merged, will revisit in the future, but for now it should work out of the box with autoconf as well as cmake.

[DennisHeimbigner]

Coulkd you make the choice of opt. flags dependent on the HDF5 version?

[opoplawski]

This "fix" is a bit problematic for Fedora as "-g -O2" are the default flags used in Fedora. But I'm fine with sticking with hdf5 1.8 in Fedora for now.

[edhartnett]

Have you reported this issue to the HDF5 team? I think it should be regarded as a bug if the code cannot work with any level of compiler optimization.

[WardF]

I haven't, but I will; I'm not certain if the issue is truly with the HDF library, since optimization with GCC 4.4 presents no problems, but I will pass along the info when i speak with them next week. I should also try 1.10.0-patch1 to see if the issue persists. I agree with the sentiment that the removal of the optimization isn't a great fix, and am open to other ideas. This should at least work in the meantime.

[edhartnett]

If I understand correctly, you have some code that works with some level of optimization, but does not work with a different level of optimization. That's clearly a bug in HDF5. They are doing something clever with memory and they are not quite being clever enough.

[ArchangeGabriel]

We also have O2 by default in Arch, and no more hdf5 1.8 for us. And I confirm this is still and issue with GCC 7.1.1 and HDF5 1.10.1:

============================================
   netCDF 4.4.1.1: h5_test/test-suite.log
============================================

# TOTAL: 26
# PASS:  25
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: tst_h_dimscales
=====================

HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
  #000: H5Gdeprec.c line 1090 in H5Gget_objtype_by_idx(): can't get object type
    major: Symbol table
    minor: Inappropriate type
  #001: H5Gdeprec.c line 1142 in H5G_obj_get_type_by_idx(): can't locate type
    major: Symbol table
    minor: Object not found
  #002: H5Gstab.c line 1217 in H5G__stab_get_type_by_idx(): index out of bound
    major: Invalid arguments to routine
    minor: Inappropriate type

*** Checking HDF5 dimension scales.
*** Creating simple dimension scales file...ok.
*** Checking that simple dimscale file can be read...ok.
*** Creating simple dimension scales file with lots of datasets...ok.
*** Creating a file with an unlimited dimension scale...ok.
*** Checking that unlimited dimscale file can be read...Sorry! Unexpected result, tst_h_dimscales.c, line: 387
FAIL tst_h_dimscales (exit status: 2)

[WardF]

Revisiting this for the imminent release candidate.

[WardF]

UPDATE: I went back and re-read the thread, nevermind below. I'll continue to work on this.

I'm ~~~~still~~~~ failing to recreate this issue, using -gO2 and hdf5 1.10.1. I wonder if it's something to do with how I'm building hdf5. Can you provide any information about how hdf5 1.10.x was built?

[WardF]

./tst_h_dimscales

*** Checking HDF5 dimension scales.
*** Creating simple dimension scales file...ok.
*** Checking that simple dimscale file can be read...ok.
*** Creating simple dimension scales file with lots of datasets...ok.
*** Creating a file with an unlimited dimension scale...ok.
*** Tests successful!

$ otool -L tst_h_dimscales
tst_h_dimscales:
	@rpath/libnetcdf.11.dylib (compatibility version 11.0.0, current version 11.4.0)
	/Users/wfisher/local-serial-1.10.1/lib/libhdf5_hl.100.dylib (compatibility version 101.0.0, current version 101.1.0)
	/Users/wfisher/local-serial-1.10.1/lib/libhdf5.101.dylib (compatibility version 102.0.0, current version 102.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)

[WardF]

Leaving this open for now but this still appears to be a compiler optimization issue, somehow. I still can't recreate it on my end (although I could previously) but will mitigate the deprecated API call at least.

[ArchangeGabriel]

Hum… My configure command:

  ./configure \
    CXX="mpicxx" \
    CC="mpicc" \
    FC="mpif90" \
    F9X="mpif90" \
    RUNPARALLEL="mpirun" \
    OMPI_MCA_disable_memory_allocator=1 \
    --prefix=/usr \
    --disable-static \
    --enable-hl \
    --enable-build-mode=production \
    --with-pic \
    --docdir=/usr/share/doc/hdf5/ \
    --disable-sharedlib-rpath \
    --enable-cxx \
    --enable-fortran \
    --enable-parallel \
    --enable-unsupported \
    --with-zlib

What’s yours? So I can try it here. What compiler?

[WardF]

I was not building with parallel, but can try that. My command was as follows:

$ CFLAGS="-O2" ./configure 

I'm working in a docker container where I have installed everything into /usr/, so my paths are all very simple.

[ArchangeGabriel]

So, I’ve retried with a serial HDF5 version, same issue.
Config flags:

    ./configure \
        --prefix=/usr \
        --disable-static \
        --enable-hl \
        --enable-build-mode=production \
        --with-pic \
        --docdir=/usr/share/doc/hdf5/ \
        --disable-sharedlib-rpath \
        --enable-cxx \
        --enable-fortran \
        --enable-unsupported \
        --with-zlib

Compil flags:

CFLAGS:                 -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong
CPPFLAGS:               -D_FORTIFY_SOURCE=2
LDFLAGS:                -Wl,-O1,--sort-common,--as-needed,-z,relro

I’ll try some other configuration flags and see if that changes anything.

[WardF]

Thanks for the flags @ArchangeGabriel I will try to recreate on a fresh VM with the same flags you've used.

[WardF]

Ok, using your flags I've been able to recreate this @ArchangeGabriel. Diving into GDB now to see what I can see.

[WardF]

@ArchangeGabriel What version of gcc are you using?

[ArchangeGabriel]

7.1.1 as said above. ;)

[WardF]

Ok. Changing optimization from -O2 to -O0 is the difference between tst_h_dimscales passing or failing. Sorry for missing the gcc version! I'm testing with 5.4.0 and seeing the same. When compiling netCDF, can you try passing -O0 (it doesn't matter how hdf5 was compiled at this point) and seeing if the test passes or fails? That should tell us a bit about gcc 7, or at least confirm that it is behaving the same as 5.4.

[ArchangeGabriel]

OK, I’ll try that, but in one hour or so, I’m a bit busy right now. ;)

[WardF]

I am seeing the behavior previously described here although I am still uncertain why exactly it is happening or what I can do about it, at this point. I'll leave this issue open and would love to hear any ideas anybody else has, while I mull this over.

[WardF]

@ArchangeGabriel No hurry, thanks! Just trying to poke any holes that I can, or establish some sort of pattern. Using the same flags and the same compiler/linker, but configuring with cmake instead of configure also eliminates the error. So this is a failure case if (autotools AND optimization). Strange!

[ArchangeGabriel]

I’ve realized I never posted the results, but gcc 7 with -O0 works.

[WardF]

Thanks; at this point I have to figure it is either a memory issue somehow in the old test (I'll run it through valgrind and check the results when I have a chance), or this hdf5-only test has exposed an issue in libhdf5. In either case, perhaps it is best to disable it as part of the distribution, for the time being, if I can confirm the fix is not something we can do on our end.

[ArchangeGabriel]

Is there any downside in building with CMake rather than autotools? That could be a solution for most people.

[WardF]

From a technical standpoint, no, although the majority of users would be adjusting their workflow; configure && make && make check && sudo make install is very much alive and baked into who-knows-how-many scripts out there :).

[ArchangeGabriel]

For the record, I’m switching ArchLinux builds to CMake, but as said in #363 the issue is still present in 4.5.0 with autotools and with CMake I’m havin a new issue: #500.

[edhartnett]

I believe now that this is caused by a memory problem in tst_h_dimscales. See #504, #505 for more info.

A memory overflow problem could easily appear and disappear based on compiler optimization. But using the gcc address sanitizer allows the problem to be shown.