Remove noisy scenarios and add new scenario #327
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current microbenchmark utility uses scenarios to test independent features, this suffers from unstable results due to many factors, such as code layout and alignment, WAF "runtime" changes, etc. This applies to all tests except the global ruleset one.
To simplify things and potentially resolve the constant noise from benchmarks, the current scenario based benchmark will only be used as a macrobenchmark which will contain only wide-ranging scenarios, starting with a single, full ruleset one.
Note that the benchmarks will fail as no scenarios are common to the master branch.