Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Access the memory contents of a file created by NC_DISKLESS #708

Closed
DennisHeimbigner opened this issue Dec 3, 2017 · 3 comments
Closed

Comments

@DennisHeimbigner
Copy link
Collaborator

DennisHeimbigner commented Dec 3, 2017

re: e-support MQO-415619
Provide a method whereby the memory contents of a file created by NC_DISKLESS
can be extracted the time of nc_close(). We could do this for netcdf-3 pretty easily, but
not for netcdf-4 (currently).
For netcdf-4, the only solution I can see is to create our own version of the HDF5 core
driver to add extra extraction functionality.

@DennisHeimbigner
Copy link
Collaborator Author

Proposal:
I am in the process of adding support for retrieving
the final file image for an in-memory file. This would
extend the existing nc_open_mem functionality.

This involves several changes.

  1. nc_open_mem() would now allow the NC_WRITE mode flag
    so a chunk of memory can be passed in and be modified
  2. nc_create() would now allow the NC_INMEMORY flag to be set
    to cause the created file to be kept in memory.
  3. the ability would be provided to obtain the in-memory contents
    of a file at the point it is closed using a new function
    nc_close_mem(int ncid, size_t* sizep, void** memory)
    Note the following.
  4. If nc_open_mem is called with NC_WRITE, then a copy of the incoming
    memory would be made internally (dictated by HDF5 functionality).
    If instead, the file is read-only, no copy would be made.
  5. The memory returned by nc_close_mem would be a COPY of the current
    contents if the original was writeable (again HDF5 forced).

I also have a couple of thoughts.

  1. Instead of nc_close_mem, I could separate out the memory extraction
    from the close operation. Hence one would call something like
    nc_get_mem() and then nc_close() instead of nc_close only.
    I prefer the nc_close_mem solution, but can anyone provide a
    use case where the separate functions is better?
  2. I could always make it so that the incoming and extracted memory
    chunks were copies. This would mean that read-only opening
    a memory chunk would pay the copy cost when it did not have to.
    But the always copy soln might be simpler for the user to understand
    and use.
    Note that this is all preliminary and is subject to change if,
    for example, I cannot get some piece of functionality to work.

DennisHeimbigner added a commit that referenced this issue Feb 25, 2018
and #708

Expand the NC_INMEMORY capabilities to support writing and accessing
the final modified memory.

Three new functions have been added:
nc_open_memio, nc_create_mem, and nc_close_memio.

The following new capabilities were added.
1. nc_open_memio() allows the NC_WRITE mode flag
   so a chunk of memory can be passed in and be modified
2. nc_create_mem() allows the NC_INMEMORY flag to be set
   to cause the created file to be kept in memory.
3. nc_close_mem() allows the final in-memory contents to be
   retrieved at the time the file is closed.
4. A special flag, NC_MEMIO_LOCK, is provided to ensure that
   the provided memory will not be freed or reallocated.

Note the following.
1. If nc_open_memio() is called with NC_WRITE, and NC_MEMIO_LOCK is not set,
   then the netcdf-c library will take control of the incoming memory.
   This means that the original memory block should not be freed
   but the block returned by nc_close_mem() must be freed.
2. If nc_open_memio() is called with NC_WRITE, and NC_MEMIO_LOCK is set,
   then modifications to the original memory may fail if the space available
   is insufficient.

Documentation is provided in the file docs/inmemory.md.
A test case is provided: nc_test/tst_inmemory.c driven by
nc_test/run_inmemory.sh

WARNING: changes were made to the dispatch table for
the close entry. From int (*close)(int) to int (*close)(int,void*).
DennisHeimbigner added a commit that referenced this issue Feb 26, 2018
and #708

Expand the NC_INMEMORY capabilities to support writing and accessing
the final modified memory.

Three new functions have been added:
nc_open_memio, nc_create_mem, and nc_close_memio.

The following new capabilities were added.
1. nc_open_memio() allows the NC_WRITE mode flag
   so a chunk of memory can be passed in and be modified
2. nc_create_mem() allows the NC_INMEMORY flag to be set
   to cause the created file to be kept in memory.
3. nc_close_mem() allows the final in-memory contents to be
   retrieved at the time the file is closed.
4. A special flag, NC_MEMIO_LOCK, is provided to ensure that
   the provided memory will not be freed or reallocated.

Note the following.
1. If nc_open_memio() is called with NC_WRITE, and NC_MEMIO_LOCK is not set,
   then the netcdf-c library will take control of the incoming memory.
   This means that the original memory block should not be freed
   but the block returned by nc_close_mem() must be freed.
2. If nc_open_memio() is called with NC_WRITE, and NC_MEMIO_LOCK is set,
   then modifications to the original memory may fail if the space available
   is insufficient.

Documentation is provided in the file docs/inmemory.md.
A test case is provided: nc_test/tst_inmemory.c driven by
nc_test/run_inmemory.sh

WARNING: changes were made to the dispatch table for
the close entry. From int (*close)(int) to int (*close)(int,void*).
@DennisHeimbigner
Copy link
Collaborator Author

Closed by pr #879

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant