Provide a way to do bit grooming before compression

[edwardhartnett]

In the NOAA GFS data files they are doing something neat to get better compression. They are zeroing out a bunch of bits they don't care about in the float.

For this to work, all we need to know is the number of bits. So we could have:

nc_def_var_smoothing(ncid, varid, nbits);

to turn this on.

For some code from Jeff Whitaker see #1543. (They call this "quantizing" and maybe that's a better name. I'm not sure if "smoothing" is correct.)

This is part of #1545

[czender]

I support adding lossy compression to netCDF. However, bit smoothing is sub-optimal because it always rounds down. Bit Grooming alternately rounds up (to 1s) and down (to 0s) which reduces the statistical biases of rounding by orders of magnitude. See the comparisons in:

Zender, C. S. (2016), Bit Grooming: Statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+), Geosci. Model Dev., 9, 3199-3211, doi:10.5194/gmd-9-3199-2016.

[edwardhartnett]

@czender excellent point! And great reference.

I propose to add this to netcdf-c after the agu. I am thinking of the following functions:

nc_def_var_quantize(ncid, varid, nbits)
nc_inq_var_quantize(ncid, varid, &nbits_in)

These would only apply to NC_DOUBLE and NC_FLOAT, calling with any other type will return an error.

nc_def_var_quantize() will add an attribute _netcdf_nbits=12 for example.

Some things to note:

• The savings in file size can be enormous.
• NOAA GFS is already doing this; other model teams also are or will be. This will be popular.
• This is fully backward compatible (as not all our new compression features will be).

[czender]

You are welcome to use the NCO code lines 576--860 (not all are necessary) of https://github.com/nco/nco/tree/master/src/nco/nco_ppc.c

[edwardhartnett]

@czender I was already planning to steal your code. ;-)

(With full attribution of course. We're classy that way on the netCDF project. We steal your code, but we do say "thank you"). ;-)

[WardF]

Tagging in @lesserwhirls and @DennisHeimbigner to this discussion, so that we can consider the potential impact on NetCDF java as well. Adding the FORTRAN and C++ interfaces shouldn’t be a problem.

[edwardhartnett]

This will have no impact on Java, because the files can be read in the normal way.

If netCDF-Java would also like to add this feature, that is of course possible. I would not hold back the rest of the netCDF community while waiting for Java.

I will add these calls to the Fortran APIs as well; just forgot to mention it, but of course we want both the F77 and F90 APIs to fully reflect the C API.

[edwardhartnett]

Also would welcome input from @jswhit and @gsjaardema on this issue...

[lesserwhirls]

I assume we’re talking about something like this?

https://www.unidata.ucar.edu/blogs/developer/en/entry/compression_by_bit_shaving

We don’t have a way of doing this (writing) via the netCDF-Java API currently, but would be easy to add (have code in our tests). Could also tie in to whatever the C library ends up doing since we use the C library for writing. Reading won’t be an issue.

One question though - what would a variable look like? How would nbits be encoded? Would it be exposed as an attribute (so a need to update the NUG)?

No technical concerns from a netCDF-Java point of view, but thank you @WardF for even asking the question. Surprises are never fun :-)

[edwardhartnett]

@lesserwhirls this is part of a larger discussion about netcdf-c and compression. See #1545

I was going to ask if netcdf-java is now using the C library. You say for writing only? Why is that? Why not read as well as write?

[WardF]

@lesserwhirls no problem, I don't want to (potentially) add to your roadmap/workload without at least waving in your direction first ;).

No particular hurry on this; when lossy compression has come up in meetings or at conferences, the scientists have expressed their... hesitancy to truncate any of their data, even when it objectively contains no information. The lack of people asking for it makes it a relatively low priority internally, but adding support through a PR contribution is effort I can justify ;). Anyways, back to prepping for AGU. Thanks all!

[lesserwhirls]

@lesserwhirls this is part of a larger discussion about netcdf-c and compression. See #1545

I was going to ask if netcdf-java is now using the C library. You say for writing only? Why is that? Why not read as well as write?

Note: this was all done before my time, but here is my take.

Writing because, I think, the HDF documentation was too unclear regarding how writing is to be done (for example, chunking, VLEN, I think?) to support all of the features needed to write netCDF-4. We'd need to rope in John Caron for the details.

In terms of reading, we do support reading through the C library, but it's not threads safe (we have to thread lock on the java side to ensure we're ok there). That's a pretty big bummer. We've enabled reading through C on the server side (TDS), and performance of the server was...well, we don't do that, and never recommend for users it outside of super rare debugging cases (once). John wrote native HDF5 read support, so he utilized that to support reading netCDF-4. We do run into some issues in making sure the Java library and the C library see things in a similar way (Enums, for example), but that's helpful for everyone as it exposes areas in need of clarification in the netCDF data model documentation (not always, but sometimes).

Also (my opinion), it's really hard to recommend an on disk data format for archive purposes that relies on a single implementation for reading. That's one of the reasons why I loved the netCDF-3 on disk formats before my time at Undiata.

[edwardhartnett]

@lesserwhirls thanks for the clarifications. Thread-safety is something Denis has been working on I believe.

The classic formats are all still alive and well, for those who don't want to depend on HDF5. That said, HDF5 is an official NASA format that is guaranteed support based on the large amount of very expensive Earth science data NASA owns in HDF5. ;-)

[lesserwhirls]

I don't doubt the support one bit, and my comments should not be taken as a ding (just in case it came across that way), but knowing that a separate read implementation can be done is certainly a plus :-)

[DennisHeimbigner]

I think this functionality could handled with the current library
since quant can be packaged as jusy another filter.

[edwardhartnett]

I agree with Elena, although a general purpose filter handling ability is great, it is too complex for most users. As with HDF5, I believe the most common compression methods should be specified in the API, including this one.

[czender]

As implemented in NCO, Bit Grooming exposes to the user the number of significant digits, then uses an internal table to convert this to minimum number of bits to quantize while preserving the requested number of significant digits. Bits are finer-grained and thus more flexible, yet less familiar to scientists than significant digits. The underlying bit-masking implementation is the same. NCO chose to expose digits rather than bits because there is something satisfying about knowing how many significant digits the quantization is guaranteed to preserve. netCDF is lower-level than NCO so perhaps bits are the better quantity to expose throught the API. In any case, I hope netCDF makes an intentional decision that considers this aspect of usability before deciding on the API. This is just another example of how user-friendliness and flexibility often lead to different outcomes.

[lesserwhirls]

As implemented in NCO, Bit Grooming exposes to the user the number of significant digits, then uses an internal table to convert this to minimum number of bits to quantize while preserving the requested number of significant digits. Bits are finer-grained and thus more flexible, yet less familiar to scientists than significant digits. The underlying bit-masking implementation is the same. NCO chose to expose digits rather than bits because there is something satisfying about knowing how many significant digits the quantization is guaranteed to preserve. netCDF is lower-level than NCO so perhaps bits are the better quantity to expose throught the API. In any case, I hope netCDF makes an intentional decision that considers this aspect of usability before deciding on the API. This is just another example of how user-friendliness and flexibility often lead to different outcomes.

100% agree. I think setting the significant digits is the right call, although if the API could support both, that'd be even better.

@czender - does NCO add any metadata to variables that use Bit Grooming to indicate that the process has been done?

[czender]

Yes, see http://nco.sf.net/nco.html#number_of_significant_digits and/or the Bit Grooming paper referenced above.

[gsjaardema]

@edwardhartnett Sorry for not responding earlier, but I think this would be a great capability.

[edwardhartnett]

Setting significant digits is an interesting take on this. I had not considered it but am certainly not opposed. I agree that for users, significant digits is much more understandable than significant bits. I will study Charlie's paper and then get back with a proposed implementation, after the AGU next week.

[lesserwhirls]

Yes, see http://nco.sf.net/nco.html#number_of_significant_digits and/or the Bit Grooming paper referenced above.

Excellent - thank you @czender! number_of_significant_digits looks good to me. Maybe also an attribute to for retained bits as well (perhaps as a less visible "special attribute")? At any rate, would want to make sure this gets added to the NUG.

[edwardhartnett]

I don't think we want to make this a special attribute (i.e. one that is hidden from the user by the netCDF API).

There's a problem adding to the list of special attributes: earlier versions of netCDF will not know to hide that attribute. So there will be a different number of attributes in the file, depending on library version used to read it.

I believe we can simply add a new attribute, visible to users, but start with an underscore. Since users only get this attribute when they explicitly call nc_dev_var_groooming(), everyone will know what to expect and all versions of the library will be able to read and understand the file (even versions that don't have the nc_def_var_grooming() function!)

Absolutely this would be documented in the NUG. The CF Conventions gang might also be interested in the definition of this attribute.

[edhartnett]

At the AGU I have been chatting with people about this and there is a great deal of interest.

Stephan Siemen, Head of Development for ECMWF, says that compression, including lossy compression, is of great interest for ECMWF data sets, both model output and data from the Copernicus Sentinel satellites.

So we have strong interest from NOAA for their main operational forecasting system (GFS), from NCAR's biggest climate modeling group (CESM), and from the ECMWF.

[czender]

That's great to hear. I'll be at the EarthCube booth (#510) from 1-2 today (Tuesday) if you would like to chat in person.

[WardF]

I'll drop by as well, I'd be interested in this conversation. Hope the conference is going well for everybody. @czender I tried to stop by your poster earlier but, predictably, it was pretty well mobbed.

[edwardhartnett]

It was good to discuss this with @czender and @WardF at the AGU! ;-)

I have studied Charlie's paper - it is excellent and does a great job answering all the detailed questions and comparing the different approaches. I agree with Charlie that bit grooming can be a very valuable lossy compression technique, and the fact that NOAA is already using a similar algorithm (and absolutely depends on it to get the required compression) is a strong indication that this is a good feature.

I have a paper in which I am outlining all the proposals for new compression in the C/Fortran APIs. Anyone interested in reviewing or collaboration should shoot me an email.

[edwardhartnett]

@czender quick question: you support specifying both NSD (number of significant digits) or DSD (decimal significant digits).

Why both? It would seem that NSD would be sufficient. Am I missing something?

[czender]

NSD is better suited for science. DSD is more familiar to some laypeople who think of significance as how many digits before or after the decimal point. The section of the paper that discusses 800 pound gorillas attempts to convey the possible DSD use-cases. I personally would always use NSD and never use DSD, but I wrote the software for generic purposes that others may find useful.

[edwardhartnett]

Bit-grooming is the only one I know of.

All the other filters are compression/decompression, so needed for both reading and writing.

[DennisHeimbigner]

At this point we need an implementation of this
as an HDF5 custom filter. That way I can incorporate it into both libhdf5
and libnczarr.

[edwardhartnett]

@DennisHeimbigner we have a working bit-groom filter here: https://github.com/ccr/ccr/blob/master/hdf5_plugins/BITGROOM/src/H5Zbitgroom.c

I can do the netcdf/HDF5 implementation.

We need to consult closely with @czender - he has some upgrades in mind for the filter, we should take on-board.

If this is implemented in netcdf-c, then we would remove it from CCR.

[DennisHeimbigner]

I would prefer to add this as a built-in filter so that we do not need to extend the API.
Also, that would make it easier to use in ncgen and nczarr.

[edwardhartnett]

We should still extend the API to make this easy for users to use...

[DennisHeimbigner]

I leave that question for discussion: new API or use existing def_var_filter?

[edwardhartnett]

I think filters are not the answer.

If we implement this as a filter, we still have all the compatibility problems: such data cannot be read by older versions of netcdf-c, cannot be read by netcdf-java, and cannot be read by HDF5 (without installing the filter, even though it is not doing a thing for readers).

I was thinking this should be implemented in that giant function that does type conversion of data. As the data are converted (or even if the data are not converted) the bit grooming algorithm can be applied.

In this way we get netcdf-java for free, as well as all older versions of netcdf-c and native HDF5. I suspect @lesserwhirls and @dopplershift would prefer it if we did not break netcdf-java reads with this feature, and there is no reason we need to.

If we just forget about filters in this case, it becomes a lot simpler for everyone...

[DennisHeimbigner]

If you do it in the write functions (which are not chunk aware) then you need
to make sure that the algorithm is idempotent because it could end up being
applied to the same data multiple times. Do we know it that true for this algorithm?
Also, when you say "jave" do you mean the pure java read-only implementation
or the mixed implementation using java+netcdf-c?

[dopplershift]

I believe @edhartnett means if it's implemented as a library feature, the pure Java implementation gets to read these files with no code modifications.

If it's left as a filter, netCDF-Java would have to at least recognize and ignore the filter to read the data.

[edwardhartnett]

@DennisHeimbigner how can it be applied to the same data multiple times? (Unless the user re-writes the data?) In the case of HDF5 it would be applied before the data are chunked, and the chunking would not affect the results.

@dopplershift is correct - if we make it part of the library, not just netcdf-java, then all existing versions of netcdf-c will be able to read the data for free. When I say netcdf-java I mean the native Java read-only implementation, which is very important. (However, it would also be readable by all existing java+netcdf-c implementations, without upgrading netcdf-c.)

@czender is of course the expert in this code, which is his. @czender would you like to comment on the idempotence of the bitgroom algorithm? I believe that it is idempotent, as long as the same starting point is chosen each time.

[DennisHeimbigner]

how can it be applied to the same data multiple times? (Unless the user re-writes the data?) In the case of HDF5 it would be applied before the data are chunked, and the chunking would not affect the results.

You have to take all possibilities into acct. So you have to assume some user will write the same data more than once. Possibly in two different programs. Otherwise you have some
kind of write once situation of which the user needs to be aware.

[DennisHeimbigner]

At this point, I am mostly convinced by Ed's arguments that we should build this into the library as a non-filter.
It would need to be added it to nczarr as well.
How about netcdf-3?

[edwardhartnett]

I believe it should also be applied to classic formats. This will benefit users in 2 ways:
1 - files can be compressed as a post processing step before storage
2 - data can be stored with correct precision.

Also it will be easier to document and explain if it applies to all binary formats.

[DennisHeimbigner]

Some issues that need to be resolved:

• What types? Float and double of course.
  What about integer types (for simulated fixed point)?
• From H5ZBitgroom, there are 5 parameters:

0. no. of significant digits to keep -- default is 3
1. typesize (in bytes)
2. missing value flag -- 0 => do not use missing value, !0 => use
3. missing value part 1 (what byte order?)
4. missing value part 2

• We can get the type from the variable's type.
• Can we get the missing value from a "missing_value" attribute?
  Or is it the same as _FillValue?
• Do we want to record the use of quantize + parameters in the file
  using some special attribute like "_Quantized"
• Idempotent? As mentioned we have to take into account multiple writes.
  • Related question: what if the two writes use different parameters?
• What name? NumCodecs uses Quantize, Zender uses Bitgroom.
  I prefer quantize as more descriptive.

[edwardhartnett]

All good questions. Some answers I know now, some remain to be worked out.

Only float and double types can use grooming. For all other types trying to turn on grooming returns error EBADTYPE (or EINVAL?)

As with deflate or other var settings, the settings for grooming cannot be changed once enddef has been called. So the settings are final for the var before any writes are done, and can not be changed, so writes cannot have different parameters.

We do want to record parameters in a non-hidden attribute that is added whenever grooming settings are turned on.

What is in a name? I don't mind what we call it. But quantization has typically referred to bit-shaving (i.e. always using 0s). Bit grooming is when we alternate 0s and 1s.

Questions to consult with @czender on when he returns later this week:

• idempotency
• fill value

[czender]

Howdy all, and sorry for the delayed response due to my off-the-grid vacation. I have just finished reading the last 10 days of this thread which occurred while I was away. I'll try to answer and ask as many questions as I think are still relevant to the discussion.

First, let me clarify that BitGrooming is a pre-processor for compression, and is not itself a compressor. BitGroomed data remain in IEEE format on-disk, hence there is no need for special software for reading. BitGrooming (BG) is a form of quantization, as are BitShaving and BitSetting, and a few other algorithms (DigitRounding, ...) not mentioned here. BG simply eliminates (by alternately shaving and setting) the least significant bits of floating-point mantissas, until the resulting mantissa preserves no more than the user-requested Number of Significant Digits (NSD). An easy to implement alternative would be to no more than the user-requested Number of Significant Bits (NSB). I think most scientists would prefer NSD, since that is the language of instrumental/numerical precision, though computer scientists might prefer NSB.

BG enhances the compression ratio (CR) of subsequent lossless compression algorithms (that for our purposes require netCDF4 format to invoke) by eliminating the randomness beyond the least significant digit. I hope this makes clear that netCDF3 format can contain BG'd data, though its CR is identically 1, so BG'd netCDF3 datasets do not change size. BG'd netCDF4 datasets that are subsequently compressed will be smaller (CR(BG) > CR(no-BG), where CR is input size/output size) than the same datasets were they compressed without first using the BG algorithm.

Idempotency:
My understanding is that BG is idempotent. The output of BG depends solely on the specified NSD. If previously BG'd data are re-BG'd they will not change unless NSD2 (the newer value of NSD) is smaller than NSD1 (the previous value), in which case more bits/digits will be quantized. BG is irreversible since precision can only be eliminated, not created, so attempting NSD2 > NSD1 should probably return an error to avoid confusion, and attempting NSD2 >= NSD1 should be OK, IMHO. The NCO BG implementation includes user-visible variable-metadata to indicate the NSD of BG'd variables. The CCR implementation does not. FWIW, both implementations treat requests to BG integer variables as no-ops (not errors).

Fill Value:
The NCO and CCR implementations of BG sometimes refer to _FillValue as "missing value" in descriptions/documentation. They are synonymous and their usage depends on my own idiosyncratic preferences for distinguishing semantics from syntax.

Please ping me via this thread for any additional clarifications. I am pinging @jtolento, a UCI grad student who has been working to update the BG algorithm to be more efficient.

[edwardhartnett]

@czender I think we all understand that bit-grooming (and it's cousins) does not, in itself, compress the data.

For example, the advantage of applying bit-grooming to a netCDF classic file would be that such a file would compress better. That is, we would write such a file, and then gzip the file. That file would be smaller than if we write the file without bit-grooming, and then compress it.

The question I have with fill value is this: what if the user specifies a fill value with more precision than the bit-groomed data? For example, if I have a fill value of 9.99999 and then specify bit-grooming with NSD=3? Then, the values that were 9.99999 will become 9.99 or 10.0, correct? So neither result will be equal to the fill value.

Does that mean that when bit grooming is applied, we must also apply it to the fill value? How do we handle the fact that alternate values of 9.99999 will be rounded different ways by the bit-groom? Do we have two different fill values? Or do we check for fill value as we bit groom and handle it specially (I think this will be the answer).

In terms of changing the NSD, that will not be allowed. Once enddef is called, the NSD is final. Attempts to change it will return NC_ELATEDEF - Attempt to define var properties, like deflate, after enddef.

[czender]

@edwardhartnett We are on the same page with respect to BG being a pre-filter.

Regarding fill values: The BG algorithm as implemented in both NCO and CCR does not touch any data equal to the _FillValue attribute, if any. Before the (irreversible) BG procedure is applied to any value, that value is compared to the _FillValue, if any. BG only quantizes valid data, not data equal to _FillValue. That is why the _FillValue argument is input to and not changed by the BG algorithm. Of course the comparison takes time, so it is only performed when has_mss_val is true. A wrinkle to note is that the current implementations only respect explicitly defined _FillValue attributes. Values equal to the default fill values (NC_FILL_DOUBLE, NC_FILL_FLOAT) will be BG'd unless the _FillValue attribute is explicitly defined to be one of those values. I think this behavior moots the rest of your FV questions. If not, let me know.

Regarding NSD: I have no problem with making it immutable in libnetcdf. Just pointing out the consequences of more and less strict behaviors.

[edwardhartnett]

OK, but we also need to apply the fill value rule to default fill values. Or figure something else out - perhaps require that a fill value be set before allowing bit-grooming to occur? Or just treat default fill values the same way - that is, check for them, and don't apply the BG algorithm for fill values.

[DennisHeimbigner]

If _FillValue has not been set when bitgrooming is defined, we could automatically
set the _FillValue to the default fill value. Assuming that fill has been set for the variable.

[czender]

It's easy enough to change the implementation to treat all floating pointing variables as having a _FillValue, either explicitly or implicitly defined. If _FillValue is explicitly defined then there would be two comparisons prior to BG'ing a value, first to the explicit _FillValue attribute, and second to the default fill value for that type. If _FillValue is implicitly defined to be the default value for that type, then there is only one comparison. I have never been sure whether this is the intended way to use the default _FillValue, though you are the experts so now it's clearer to me.

[edwardhartnett]

WRT fill values, the CCR code and the code in my PR both ignore data that matches the fill value, when doing quantization. This works with the defined fill value, if the user has defined one, or the default fill value, if the user has not defined one.

On to another question: should quantize work with scalars? I believe the answer is yes...

[edwardhartnett]

Thanks @WardF ! I'm so happy to see this great new feature make it's way into the hands of users! ;-)