ncdump failed assert() when reading NCZarr file

[czender]

Hola Unidata,

Zarr'ification of NCO is proceeding in fits and starts. When I throw a non-trivial file (below) at the file:// scheme, I get the same breakage that I do not understand in both Unidata tools (ncdump) and NCO (ncks). I wonder if this might be an NCZarr library issue. The following demonstrates behavior with today's main tree of netcdf-c development on up-to-date MacOS 12.4, and the behavior seems replicable on other OS's.

The problem manifests as a failed assertion when reading a fairly detailed NCZarr file:

zender@spectral:~$ ncdump "file://${HOME}/in_zarr#mode=nczarr,file"
Assertion failed: (NCJlength(jdimrefs) == rank), function define_vars, file zsync.c, line 1727.
Abort trap: 6
zender@spectral:~$ ncks "file://${HOME}/in_zarr#mode=nczarr,file"
Assertion failed: (NCJlength(jdimrefs) == rank), function define_vars, file zsync.c, line 1727.
Abort trap: 6
zender@spectral:~$ 

The indicated zsync.c code is obscure to me, something to convert JSON dimension lists->netCDF. To create the input NCZarr file I used the latest ncgen on in_zarr.cdl (attached) as follows:

ncgen -lb -o "file://${HOME}/in_zarr#mode=nczarr,file" ${HOME}/in_zarr.cdl

The NCZarr production proceeds nominally. in_zarr.cdl.txt (attached) is the standard netCDF3-based (no netCDF4 atomic types, attributes, or features) test file in.cdl with the single modification that the time dimension has been changed from unlimited to fixed size = 10 (otherwise the conversion to NCZarr would fail). Manual inspection shows that the NCZarr file appears to be correctly created on disk, so I expect that both ncdump and ncks will print its contents just as they both do correctly for much simpler NCZarr files (e.g., one or a few variables). Instead the above error occurs. I hope it is easy for others to reproduce this error. I expect I might be asking NCZarr to handle a feature it does not yet support but I can't discern what that might be since the ncgen conversion works correctly. Any pointers or insight appreciated.

Thanks,
Charlie

[WardF]

I'm able to duplicate this on MacOS 12.5.1; any immediate idea, @DennisHeimbigner ? I'll keep digging in as well.

[DennisHeimbigner]

The problem appears to be the handling of scalars. Pure Zarr does not
support scalar values, but NCZarr does. So I have code to treat a scalar
as of shape "[1]" and then having hacks in various places to recognize
the situation.
Anyway, there is a bug in the handling of scalars.
A temporary work around is defining dimension named "scalar"
and convert the scalar variables such as lat_times_lon_nb
to lat_times_lon_nb[scalar].

[czender]

Thanks for looking into this. I hadn't realized there was anything special about scalars in Zarr vs. NCZarr. I'll continue the Zarr-ification and just test it without scalars until a patch for scalars is merged.

[DennisHeimbigner]

I had better make sure that is mentioned in the documentation.

[czender]

I can confirm that adding a degenerate (size=1) dimension to all the scalars in that test file results in the expected/desired behavior. Also that I do not see the special treatment of scalars mentioned in the NCZarr documentation.

[DennisHeimbigner]

Fixed(?) by #2492