Cleanup DAP4 testing #2555
Merged
Cleanup DAP4 testing #2555
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
NOTE: This PR should not be included in 4.9.1 since additional
DAP4 related PRs will be forthcoming.
This PR makes major changes to libdap4 and dap4_test driven by changes to TDS.
* Enable DAP4
* Clean up the test input files and the test baseline comparison files. This entails:
* Remove a multitude of unused test input and baseline data files; among them are dap4_test/: daptestfiles, dmrtestfiles, nctestfiles, and misctestfiles.
* Define a canonical set of test input files and record in dap4_test/cdltestfiles.
* Use the cdltestfiles to generate the .nc test inputs. This set of .nc files is then moved to the d4ts (DAP4 test server) war file in the tds repository. This set then becomes the canonical set of DAP4 test sources.
* Scrape d4ts to obtain copies of the raw streams of DAP4 encoded data. The .dmr and .dap streams are then stored in dap4_test/rawtestfiles.
* Disable some remote server tests until those servers are fixed.
* Add an option to ncdump (-XF) that forces the type of the _FillValue attribute; this is primarily to simplify testing of fill mismatch.
* Minor bug fixes to ncgen.
* Changes to libdap4:
* Replace old checksum hack with the dap4.checksum flag.
* Support the dap4.XXX controls.
* Cleanup _FillValue handling, especially var-attribute type mismatches.
* Fix enum handling based on changes to netcdf-java.
* Changes to dap4_test:
* Add getopt support to various test support programs.
* Remove unneeded shell scripts.
* Add new scripts: test_curlopt.sh
DennisHeimbigner
added a commit
to DennisHeimbigner/netcdf-c
that referenced
this pull request
Nov 16, 2022
Warning: This PR is a follow on to PR Unidata#2555 and should not be merged until that prior PR has been merged. The changeset for this PR is a delta on the PR Unidata#2555. This PR re-enables the use of the server *remotetest.unidata.ucar.edu/d4ts* to test several features: 1. Show that access over the Internet to servers using the DAP4 protocol works. 2. Test that DAP4 support in the [Thredds Data Server](https://github.com/Unidata/tds) is operating correctly. 4. Test that the DAP4 support in the [netcdf-java library](https://github.com/Unidata/netcdf-java) library and the DAP4 support in the netcdf-c library are consistent and are interoperable. The test inputs (primarily *\*.nc* files) provided in the netcdf-c library are also used by the DAP4 Test Server (aka d4ts) to present web access to a collection of data files accessible via the DAP4 protocol and which can be used for testing Internet access to a working server. To be precise, this version of d4ts is currently in unmerged branches of the *netcdf-java* and *tds* Github repositories and so are not actually in the main repositories *yet*. However, the *d4ts.war* file was created from that branch and used to populate the *remotetest.unidata.ucar.edu* server The two other remote servers that were used in the past are *Hyrax* (OPenDAP.org) and *thredds-test*. These will continue to remain disabled until those servers can be fixed. ## Primary Changes * Rebuild the *baselineremote* directory. This directory contains the validation data needed to test the remote servers. * Re-enable using remotetest.unidata.ucar.edu as part of the DAP4 testing process. * Fix the *dap4_test/test_remote.sh* test script to match the current available test data. * Make some changes to libdap4 to improve the ability to catch malformed data streams [affects a lot of files in libdap4]. ## Misc. Unrelated Changes * Remove a raft of warnings, especially in nc_test4/tst_quantize.c. * Add some additional explanatory information to the NCZarr documentation. * Cleanup some Doxygen errors in the docs file and reorder some files.
DennisHeimbigner
added a commit
to DennisHeimbigner/netcdf-c
that referenced
this pull request
Jan 19, 2023
This change-set modifies PR Unidata#2555 to add the changes listed below. Most of these changes are required by changes to the Java remotetest.unidata.ucar.edu server. ## DAP4 Related Changes * Add tests *dap4_test/test_constraints.sh* and *dap4_test/test_hyrax.sh*. * Provide explicit list of remotetest files to test. * Cleanup local checksum computing and verification. * Define a temporary Hyrax hack flag to deal with the way Hyrax handles checksums and add "#hyrax" fragment flag for it. * Add a hack to get past an LGTM problem with using "http:". * Improve debug support. ## Other Changes * Cleanup the recipe in *docs/nczarr.md* for building *aws-sdk-cpp* library.
|
Hi Dennis, this looks good at a glance, but with 423 changed files, 'at a glance' is subjective. Am I right that the bulk of those changes are to the test files? I want to dig through and wrap my head around this, but should get it (and the subsequent that depends on this) merged in shortish order. |
|
Yes, the primary change is cleaning up the test cases, which required adding new testsets and modifying others. |
|
Just making sure; I spent some time reviewing. Thanks a lot, glad we could get this merged in! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
NOTE: This PR should not be included in 4.9.1 since additional DAP4 related PRs will be forthcoming.
This PR makes major changes to libdap4 and dap4_test driven by changes to TDS.