Make the validation condition for random distributions lenient #550
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
manopapad
reviewed
Aug 19, 2022
tests/integration/utils/random.py
Outdated
| # method, we make the check lenient to avoid random | ||
| # failures in the CI. (we still need the check to catch | ||
| # the cases that are obviously wrong.) | ||
| assert np.abs(theo_stdev - stdev) < stdev_tol * np.abs(theo_stdev) |
There was a problem hiding this comment.
The default stddev_tol = 2 = 200% seems like a very high limit for a relative difference. Since standard deviation is always non-negative, what you are essentially checking here is that stddev < 3 * theo_stdev. Is that your goal here, or is stddev_tol = 2 a typo?
There was a problem hiding this comment.
Oops. I spent much time on writing a comment and didn't do the job for the test itself ;) I revised the test to make it more sensible.
manopapad
approved these changes
Aug 19, 2022
jjwilke
pushed a commit
to jjwilke/cunumeric
that referenced
this pull request
Sep 2, 2022
…gate#550) * Make the validation condition for random distributions lenient * Fix typo * Catch too small standard variations against theoretical values as well * Replace unnecessary NumPy calls with Python primitives * Tighten the tolerance
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current validation check for random distributions is causing random test failures in the CI. It simply checks if the standard deviation of the samples is close enough to the theoretical value, but failing this test doesn't have any statistical significance unless we draw extremely many samples. (i.e., a particular run might be simply very unlucky.) until we find a better way to validate samples, this PR simply relaxes the check to avoid any unexpected and undesirable test failures.