Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak in Windows when building netcdf with hdf5 1.10.8 #2203

Closed
abhibaruah opened this issue Jan 27, 2022 · 1 comment · Fixed by #2204
Closed

Memory leak in Windows when building netcdf with hdf5 1.10.8 #2203

abhibaruah opened this issue Jan 27, 2022 · 1 comment · Fixed by #2204

Comments

@abhibaruah
Copy link

NetCDF version: 4.8.1
HDF5 version: 1.10.8
OS: Windows 10

I am a developer at The MathWorks Inc. For MATLAB, we were building netcdf with hdf5-1.8.12.
However, we are trying to change netcdf's dependency on hdf5 to v1.10.8.

I built netcdf successfully with hdf5 1.10.8. However, while qualifying, I see that there is a memory leak in Windows when netcdf is built with hdf5 1.10.8. The leak is seen only in Windows, and I did not see it in Linux or MacOS.

I was able to resolve the leak by modifying the definition of the function

static int
ansi2utf8(const char* local, char** u8p)

in netcdf\libdispatch\dpathmgr.c

wchar_t* u16
is not being freed, which leads to this leak in Windows.
I was able to resolve this by adding nullfree(u16) before the return statement in ansi2utf8.

The issue can be reproduced by simply opening and closing a netcdf4 file, and running a leak detector in Windows.

# include <stdio.h>
# include <netcdf.h>

#define FILE "demo_example.nc"


int main(int argc, char *argv[])
{
    int ncid, retval;
    retval = nc_open(FILE, NC_NOWRITE, &ncid);
    printf("NCID = %d\n", ncid);
    retval = nc_close(ncid);
    printf("Retval = %d\n", retval);
    return 0;
}
@WardF
Copy link
Member

WardF commented Jan 27, 2022

Thank you, this will be an easy fix to incorporate!

WardF added a commit to WardF/netcdf-c that referenced this issue Jan 28, 2022
WardF added a commit that referenced this issue Jan 28, 2022
Correct a memory leak as reported in #2203
clrpackages pushed a commit to clearlinux-pkgs/netcdf that referenced this issue Feb 9, 2023
Aaron Kaplan (1):
      Fix incorrect PR link

Alexander Barth (2):
      Include getopt.h if HAVE_GETOPT_H is defined
      use isnan instead of _fpclass on non-MSVC compilers

Ben Boeckel (4):
      size_t: use for type consistency where necessary
      ncx: match loop variable types with checks
      libdispatch: avoid warnings about string size computations
      cmake: extract the HDF5 version when finding HDF5 manually

Charlie Zender (12):
      First draft of Granular BitGroom feature for netcdf-c
      Eliminate GBG-specific initialization, pad syntax with whitespace, try to fix syntax bugs
      add missing variables
      Change NC_QUANTIZE_ATT_NAME to NC_QUANTIZE_BITGROOM_ATT_NAME
      Change test to verify that using quantize mode one greater than NC_GRANULARBG (instead of NC_QUANTIZE_BITGROOM) fails.
      First draft of BitRound implementation
      Fix bone-headed stuff. tst_quantize and tst_filter remain borken.
      Set bit_xpl_nbr_sgn_dbl=52 not 53. No longer add additional bit to prc_bnr_xpl_rqr for BitGroom NC_DOUBLE. Bugfix maintains original BG behavior, fixes corner case.
      Change name to _QuantizeGranularBitRoundNumberOfSignificantDigits. Change identifier from QUANTIZEBR to QUANTIZEGBR to make room for new BitRound token = BR.
      Add BitRound to ncgen
      Add missing BitRound block to zsync.c
      Fix mis-copied comparator

Dave Allured (9):
      tst_misc.sh: Fix hang in make check
      Format compatibility when re-opening files
      Add compatibility function prototype
      Improve comments about format compatibility
      Minor comment fix for return codes
      CI: Update HDF5 testing 1.8.21 --> 1.8.22
      hdf5set_format_compatibility.c: Function docs.
      Release notes: fix link and spelling
      Release notes for #2176, format compatibility

Dennis Heimbigner (179):
      Adding missing file ncdap_test/manyurls.h
      Make Issue Unidata/netcdf-c#2077 work when build is repeated.
      Make Issue Unidata/netcdf-c#2077 work when build is repeated.
      Update release notes
      Make the fortran cache API always be defined.
      Remove no-longer-needed debug directory
      Update RELEASENOTES.md
      ENOTNC4 -> ENOTBUILT
      Add filter support to NCZarr
      Update Release Notes
      Fix handling of enum constants nested in compound types.
      Significantly Improve Amazon S3 Cloud Storage Support
      Update release notes
      Fix FreeBSD fileno problem in the ncgen parsers
      remove actions debug
      Fix new lgtm alerts
      One additional lgtm fix
      (1) improve INI parser (2) Fix make discheck
      Patch errors
      Update release notes
      Fix memory leak
      Fix bug in the default HDF5 byte-range reader
      Update Release Notes
      ckp
      Fix additional S3 support issues
      Update release notes
      Allow optional use of libxml2
      Update release notes
      Typo in previous PR
      Fix a number of OS specific bugs
      Update release notes
      Fix use of S3 with a non-aws appliances
      Update release notes
      Get signature of NC_s3sdk*ize correct
      Verify failure
      not verified
      The remotetest server is down for a while
      The remotetest server is down for a while
      Restore default on for enable-dap-remote-testsc
      Cleanup the CMake inter-test dependencies
      Update RELEASE Notes
      Replace ezxml with tinyxml2
      Update RELEASENOTES.md
      Remove debug
      remove ezxml code as no longer required
      Support MSYS2/Mingw platform
      Update RELEASENOTES.md
      Fix new LGTM alerts
      Fix various problem around VLEN's
      update release notes
      Update actions
      1. Fix an additional flaw in fill_value handling where non-atomic default values were not properly being handled.
      1. Fix an additional flaw in fill_value handling where non-atomic default values were not properly being handled.
      Make sure mode flags are properly defined in netcdf.h
      Update release notes
      Add complete bitgroom support to NCZarr
      Update release notes
      Rename GranularBitGroom to GranularBitRound
      Add support for setting HDF5 alignment property when creating a file
      Update release notes
      Remove conflicts; does not work with OSX
      Improve UTF8 Support On Windows
      Update Release Notes
      Update Release Notes (2)
      Explicitly disallow variable length type compression
      Update Release Notes
      Update get/set alignment documentation
      Incorporate Dave Allured's documentation
      Enhance/Fix filter support
      Update release notes
      Fix new LGTM errors
      remove use of strlcat
      more strlcat remove
      retry
      typo
      1. Fix conflicts with current master.
      Update selected documentation
      Update release notes
      Clean up some ncdap tests.
      Turn on the xarray convention for NCZarr format
      Update release notes
      Fix float precision issue in ncdap_tests
      precision fix
      OSX only
      Add Release Notes warning
      Change so explicit xarray => pure zarr
      Remove debugging
      minor fix
      Fix typo "Nan" -> "NaN"
      restrict the json convention to dictionaries only
      update file permission
      ckp
      testing
      final
      Allow the read/write of JSON-valued Zarr attributes.
      ckp
      missing file'
      update file permission
      cmake
      ckp
      installtest
      switch
      ckp
      ckp
      ckp
      restore
      switch
      ckp
      stdf
      endif
      missing
      update file permission
      test sz
      debug
      debug2
      debug3
      debug4
      debug5
      debug7
      nobzip2
      debug8
      debug10
      examp1
      debug20
      debug21
      setx
      debug30
      exam5
      debug1
      debug2
      debug5
      libdir
      final?
      testing
      dap1
      testenv
      ckp
      Two fixes for the price of one!
      Forgot to fix CMake
      restore
      final
      push
      Support installation of filters into user-specified location
      Remove debug
      Update release notes
      bug2
      update docs
      bug3
      bug4
      bug5
      bug6
      bug10
      bug8
      debug1
      debug2
      debug10
      debug12
      szip1
      Fix szip handling
      recap
      Turn off debug
      final
      final
      ckp
      cleanup
      Fix distcheck problems
      Update release notes
      update
      final1
      Improve filter installation process to avoid use of an extra shell script
      Update release notes
      swith
      I made a major update to this PR with the following changes:
      typo
      Update documentation
      Update docs again
      rebuild
      Update to default --with-plugin-dir to yes
      Fix internal bz2 problems

Edward Hartnett (126):
      added 1.12.1 to hdf5 versions built by GitHub actions
      added commas
      added info about logging to build summary, fixed info about byterange in build summary
      added info about logging to build summary, fixed info about byterange in build summary
      added info about logging to build summary, fixed info about byterange in build summary
      getting ready for next try at quantization code
      further preparation for try 2 at quantizing
      further preparation for try 2 at quantizing
      adding quantize functions to all the dispatch tables
      now qunatizing with inq function in dispatch table
      fixed version numbers
      merged configure.ac and CMakeLists.txt with changes from master branch
      merged nc4hdf.c with changes from master branch
      fixed comment
      further development
      now reading quantize attribute to get settings
      moving function
      perparing to apply bitgroom algorithm
      more testing of qunatize setting
      more quantize testing
      more quantize testing
      more quantize testing
      more quantize testing
      more tests for quantization
      more quantize testing
      more quantize testing
      changed name of attribute to _quantize_bitgroom_number_of_significant_digits
      improved doxygen documenation
      improved doxygen documenation
      change name of att to _QuantizeBitgroomNumberOfSignificantDigits
      bitgroom working for floats
      more quantize testing
      cleanup quantize code
      whitespace cleanup
      moving quantize to its own function
      attempting to fix ncdap test on appvayor
      quantize now working for NC_DOUBLE
      testing of quantize with scalars
      added more documentation, also started on test code for type conversion
      type conversion with quantize between float and double
      undid suggested change to ncdap_test/CMakeLists.txt
      more testing with type conversion
      testing with fill values
      testing with fill values
      testing with fill values
      more tests
      turned off failing quantize test
      code clean up
      refactored quantize code
      more tests for quantize
      added parallel I/O quantize test
      added parallel I/O quantize test
      improving benchmark program
      changed makefile to allow tst_gfs_data_1 to pick up libz from LD_LIBRARY_PATH first
      changed makefile to make benchmark bm_file work properly with zlib-ng
      changed name of tst_gfs_data_1.c to tst_compress_par.c
      tinker with data algorithm for tst_compress_par.c
      now nsd of 0 is NC_EINVAL for nc_def_var_quantize()
      fixed benchmark program run
      adding quantize test
      better comments
      now setting deflate_level to zero does not turn on deflate
      added sequential benchmark program for compression
      more test development
      more test development
      starting to make logging output files for each process for parallel IO builds
      removing unneeded file error4.c
      better handling of logging for parallel I/O builds
      turned off logging in test
      turned off logging in test
      turned logging back on in one test
      turning on parallel I/O tests in CI
      changed Makefile.am flags in examples/C
      turning off parallel tests?
      turning on parallel tests
      fixed makefile
      adding pnetcdf
      untabified
      turning off tests that depend on ncpathcvt when --disable-utilities is used
      fixing some issuses of pnetcdf tests in nc_test
      fixing some issuses of pnetcdf tests in nc_test
      now running MPI-enabled tests with mpiexec for parallel I/O builds
      fixed problems running mpi-enabled tests with --enable-pnetcdf in nc_test
      fixing some nc_test builds when --enable-pnetcdf is not used
      fixing some nc_test builds when --enable-pnetcdf is not used
      fixing some nc_test builds when --enable-pnetcdf is not used
      fixed missing NC_HAS_QUANTIZE
      added ZSTD to netcdf_meta.h and libnetcdf.settings
      adding test to demonstrate parallel I/O numrecs problem
      attempts to get working
      working on parallel numrec problem
      ensuring that MPI_File_close() is only called on open MPI file handle
      fixed parallel functions for netcdf-fortran build
      fixed parallel functions for netcdf-fortran build
      fixed warnings
      added BENCHMARKS to the summary
      fixing parallel I/O bug with dim length for unlimited dimensions
      fixing parallel I/O bug with dim length for unlimited dimensions
      fixing parallel I/O bug with dim length for unlimited dimensions
      fixing parallel I/O bug with dim length for unlimited dimensions
      declared a variable outside for loop
      change to trigger CI
      added  sequential test for enddef issue
      added some entries to nc-config
      no longer run tst_parallel_zlib with 1 processor in testing
      undoing unintended change
      undoing unintended change
      added tst_parallel6 to CMake parallel build
      added nc4cache.c to list of doxygen input files
      fixing doxygen warnings
      more doxygen warning fixes
      doxygen for filter functions
      fixed missing szip constants in netcdf.h
      removed all-error-codes.md
      added netcdf_filter.h to doxygen build
      adding quantize documentation
      fixed some doxygen warnings
      fixed some doxygen warnings
      fixed some doxygen warnings
      fixed more doxygen warnings
      fixing issues with dimlens of unlitmited dims in complex situations
      more detail on which algorithm to choose
      fixed documentation warnings in filters.md
      change to trigger CI
      turn on set -x for failing test script
      turned set -x off again

Egbert Eich (5):
      swap[4|8]b: Satisfy strict aliasing rules
      Fix type punning in val_NC_check_voff() by using memcpy instead of assignment
      Fix type punning in xxdrntohdouble() by using memcpy instead of assignment
      NCD4_dumpbytes: use correct swapline for object size
      d4util.h: make swapinlineXX more robust against type punning

Ethan Davis (1):
      NUG appendix files and CDL doc file moved to netcdf/NUG. Remove here.

Even Rouault (1):
      Fix a stack-read-overflow in ncindexlookup()

Greg Sjaardema (26):
      WIP: attribute creation order on/off
      Address code review issues
      Fix indentation...
      Finish argument name refactoring...
      Detect attribute creation order tracking setting
      Make sure HDF5_C_LIBRARY and HDF5_HL_LIBRARY set
      Refactor Z library detection
      Put threads find_package back where it was.
      fix last commit.
      Remove duplicate line
      Remove another duplicate find_path
      Better detection of libz library
      Use consistent form for find_package
      Clean up some whitespace junk
      Avoid infinite loop for finding large prime values
      Remove duplicate line
      Make dimscale attachement to variables optional
      Better implementation of the no dimscale attach option
      Eliminate tabs added in last commit
      Revert all whitespace changes
      One more try at new implementation with no whitespace changes
      Initialize variable
      Initialize potentially uninitialized variable
      Fix syntax error
      fix typo
      Specify correct plugin suffix

Jennifer Oxelson (7):
      intellij
      Docs migration
      docs migration, part 2
      docs cleanup
      docs link updates
      Update release_header.html
      Update CMakeInstallation.cmake

Mark Harfouche (2):
      Fix a few memory leaks and erroneous logic in nczarr_test/testfilter_misc
      Fixup nc_test4/test_filter_misc.c too

Michael Hirsch (1):
      CMake: CMP0075 requires CMake >= 3.12.

Milton Woods (27):
      Support ANSI *printf specifiers on mingw-w64
      valgrind not supported by mingw
      Avoid setting RPATH in mingw
      hostname lacks arguments on mingw
      Use relative pathnames in __FILE__ macros on mingw
      Use canonical path to derive name of dataset
      Avoid redefinition of strcasecmp in mingw-w64
      Define strcasecmp for Windows in ncconfigure.h
      Avoid duplicate definition of strcasecmp in config.h
      Use strcasecmp definitions from config.h
      Remove config.h from ncjson.c and nclist.c
      Use Windows paths in tests of mingw executables
      Replace pwd by $PWDCMD in test_common.in
      Replace pwd by $PWDCMD in d4test_common.sh
      Replace pwd by $PWDCMD in test_nczarr.sh
      Disable bash path conversions in testpathcvt.sh on mingw
      Replace pwd by $PWDCMD after test_common.sh is sourced
      Revert "Use Windows paths in tests of mingw executables"
      Revert "Replace pwd by $PWDCMD in test_common.in"
      Revert "Replace pwd by $PWDCMD in d4test_common.sh"
      Revert "Replace pwd by $PWDCMD in test_nczarr.sh"
      Revert "Replace pwd by $PWDCMD after test_common.sh is sourced"
      Use Windows format for pwd in test_common on mingw
      Fix NCJTHROW definition after merge from main
      Open datasets in binary mode on mingw
      Use binary mode for tst_inmemory in mingw
      Fix dll exports for ncxml

Paul Harris (1):
      Fix libraries for CURL and HDF5

Philippe Antoine (1):
      fix memory leak in genc_netcdf

Rostislav Kouznetsov (2):
      Add doc on errors introduced by lossy compression
      Fix typo

Sean McBride (1):
      Fixed Clang -Wstrict-prototypes warnings

Sergey Kosukhin (1):
      fix configure check for the -fno-strict-aliasing flag

Tobias Kölling (3):
      added test for opening a trunceded file using in-memory open
      allow missing udata when closing file with abort=1
      hand over udata to h5->mem a little earlier

Ward Fisher (130):
      Replace source for Unidata logo.
      Update windows binary download link url.
      Update footer.html and Doxyfile to use local Unidata image.
      Remove obsolete doxygen tags.
      Added uniLogo.png to make dist.
      Added a one-off stanza for autoconf-based tests on Github Actions.
      Propagate github actions tests into cmake-based stanza.
      Correct spacing issue in GA yaml file.
      Modify GA workflow a bit.
      Reverse GA workflow. Run the one-off tests first and then run the full test matrix. This should shorten the test/failure cycle.
      Thanks to newer ctest functionality, if ctest tests fail, re-run just the failed ones with more verbose output.
      Attempt at test orchestration to avoid a race condition when running tests in parallel.
      A bit more twiddling of the github actions workflow.
      Syntax debugging.
      Test dependency on one-offs updated.
      Correct a typo in the github actions yaml file.
      Correct a typo.
      Revamping the HDF5_VERSION logic.
      Plugged a hole where HDF5_VERSION wouldn't be checked against minimum HDF5 required version.
      Added logic for checking the version when the HDF5 libraries and include directories are set explicitly.
      Caching HDF5_VERSION string for diagnostic purposes.
      Correct an issue with appveyor builds, specific to systems where a bash shell is not available.
      Correct logic error.
      Begin wiring in option to toggle automatic search for libxml2 in support of https://github.com/Unidata/netcdf-c/pulls/2135
      Updated release notes.
      Updated .gitignore to ignore .vscode directory.
      Adding control block, --enable and disable option for libxml2 to configure.ac
      Added missing include for read() function.
      Addressed an issue in ncbytes where long unsigned ~= size_t
      Preprocessing for generated ezxml files.
      Fencepost unistd.h inclusion.
      Renamed the workflow file for running ubuntu-based tests.
      Adding github actions.
      Renamed some of the job steps.
      Collapse some redundant definitions in the Ubuntu-based workflow.
      Syntax
      Syntax
      It was worth a shot.
      Renaming jobs to differentiate between Ubuntu and Linux jobs.
      Temporarily disable ubuntu tests on PR.
      Temporarily disable ubuntu tests on PR.
      More testing.
      Sorting out the reported syntax error.
      Remove ubuntu-specific thing.
      Syntax error discovered (I hope)
      Removed call to autoreconf
      Remove more calls to apt.
      Installing autoconf tools
      More scripting.
      Move autoconf installation stanza
      Cache automake install.
      Additional expansion of naming.
      More work on the OSX github actions.
      Remove HDF4
      Add a missing space.
      Refactor triggers on GA runs.
      Removed OSX tests from run-on-push, modify so that they run on pull request.
      Avoid name collision in macOS tests.
      Collapse and update the HDF5 versions being tested on Github Actions.
      Add skeleton mingw/msys2 Github CI tests.
      Added cmake to the mingw-based test.
      Remove stale travis-ci configuration file.
      Working on MinGW/MSYS2 support. Attempting workflow with system-installed libhdf5
      Remove stray text.
      Further developments unfolding.
      Temporarily disable cache
      Clean up some Unix-related cruft.
      Really simplify things.
      Additional compilation work.
      Next iteration of compilation test.
      Remove stray text.
      Missing symbol in MSYS2/MinGW compilation.
      Dangling undefined references to nulldup. These should be unified in a header, but that is a separate issue to open/resolve. For now, resolving these as part of getting mingw/msys2-based github actions working.
      Cleaned up another undefined reference.
      More cleanup.
      Collapse build and run test steps into a single step, no need to build all the tests before seeing if any fail.
      More tweaks.
      Remove stray character.
      Moving on to cmake-based github actions.
      Adding hdf5 info for cmake-based tests.
      Force CMake to use gcc
      Oh right, Unix Makefiles
      HDF5 configuration diagnostic output
      Syntax error.
      Syntax error.
      Missing pipe character.
      Removing cmake-based tests for now.
      Correct trigger event for mingw tests to pull_request from push'
      Missed a merge conflict marker, fixed that.
      Modify cmakelists.txt to generate oc2 auth html file whenever a doxygen-enabled build is run.
      Correct a memory leak as reported in Unidata/netcdf-c#2203
      Correct the accidental removal of NOUNDEFINED, as reported in Unidata/netcdf-c#2196
      Cleaning up make dist, it's generating tarballs that don't work with cmake.
      Adding a missing file to make dist.
      Adding a missing reference file.
      Remove stale badges, add badge for Repology in support of https://github.com/Unidata/netcdf-c//pull/1350
      Change order in which auth.html is generated and copied.
      Tweak to software.html static page.
      Removed empty, useless script.
      Added new credits file.
      Reconfiguring the Ubuntu Github Actions test.
      Added first pass at a parallel github actions one-off
      Correct dependency chain for Github Actions.
      Correct compiler issue with parallel one-off test.
      Remove on-push trigger for OSX-based workflows.
      Made the parallel build action label mention pnetcdf.
      Update URL for windows downloads.
      Added error message.
      correct links for gliderscope and netcdfninja in software.html
      Removed precision fiddling with OSX to correct issues that resulted in remote testing. Also integrated the one-line change from Unidata/netcdf-c#2347, which is proving problematic (for some reason).
      Add missing file to EXTRA_DIST
      Turned off plugin directory during CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/bash '/home/wfisher/Desktop/gitprojects/netcdf-c/missing' aclocal-1.16 -I m4 in support of Unidata/netcdf-c#2348 (comment)
      Revised netCDF version and SO version
      Bumped dev version in CMakeLists.txt.  SO Version remains unchanged.
      Updated doxygen version, added target release date to RELEASE_NOTES
      Updated release date to reflect actual release.
      Updated release numbers for 4.9.0
      Update configure.ac to incorporate maintainer mode
      Testing manual workflow_dispatch for github actions.
      Updated .gitignore for generated files we need as part of the release process.
      Added autoconf-generated files for a stand-alone release.
      Added generated files for the stand-alone release.
      Correct error in make dist
      Updated generated makefile.
      Make dist is generating a distribution with missing files, that only cmake seems to complain about.
      Corrected Makefile.am error.
      Updated generated Makefile.in
      Incorporated GitHub-generated release notes into the manually managed release notes, and updated the list of included dependencies for Windows installer downloads.
      Make plugin dir off by default for this release.
      Regenerated configure script.

wkliao (7):
      use NC_EDIMSCALE for errors related to dimscale
      fix a compile error
      use AC_MSG_WARN/AC_MSG_NOTICE instead of echo
      vpath build: use top_srcdir
      avoid type define MPI_Comm and MPI_Info
      Compile dparallel.c only when parallel I/O is enabled
      allow to test tst_cdf5format when PnetCDF is enabled

xantares (1):
      cmake: list zip libs before szip
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants