Fix benchmark variances

[Anilm3]

• Fix some issues on valid test generation.
• Removed "mixed" tests.
• Added fixed-size tests with the nomenclature: `random.x.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (d08a9e6) 83.28% compared to head (f03f900) 83.28%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #246   +/-   ##
=======================================
  Coverage   83.28%   83.28%           
=======================================
  Files         106      106           
  Lines        4177     4177           
  Branches     1915     1915           
=======================================
  Hits         3479     3479           
  Misses        279      279           
  Partials      419      419           

Flag	Coverage Δ
waf_test_none	83.18% <ø> (ø)
waf_test_sse2	83.25% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-01-17 17:54:26

Comparing candidate commit f03f900 in PR branch anilm3/benchmark-variance with baseline commit d08a9e6 in branch master.

Some scenarios are present only in baseline or only in candidate runs. If you didn't create or remove some scenarios in your branch, this maybe a sign of crashed benchmarks 💥💥💥
Check Gitlab CI job log to find if any benchmark has crashed.

Scenarios present only in baseline:

• bool_equals_matcher.random.long_strings
• schema_extraction_processor.random.any
• string_equals_matcher.random.long_strings
• phrase_match_matcher.random.deep_containers
• string_equals_matcher.random.any
• lowercase_transformer.random.any
• schema_extraction_processor.random.deep_containers
• remove_nulls_transformer.mixed
• ip_match_matcher.mixed
• regex_match_matcher.lowercase_transformer.random.deep_containers
• string_equals_matcher.random.deep_containers
• signed_equals_matcher.mixed
• unsigned_equals_matcher.random.long_strings
• float_equals_matcher.random.deep_containers
• regex_match_matcher.random.long_strings
• regex_match_matcher.case_insensitive_flag.random.any
• regex_match_matcher.case_insensitive_flag.mixed
• regex_match_matcher.case_insensitive_option.random.any
• exact_match_matcher.random.any
• lowercase_transformer.random.long_strings
• float_equals_matcher.random.long_strings
• lowercase_transformer.mixed
• exact_match_matcher.random.deep_containers
• regex_match_matcher.case_insensitive_option.random.deep_containers
• regex_match_matcher.lowercase_transformer.random.any
• regex_match_matcher.random.deep_containers
• ip_match_matcher.random.any
• ip_match_matcher.random.long_strings
• is_sqli_matcher.random.deep_containers
• phrase_match_matcher.mixed
• regex_match_matcher.random.any
• phrase_match_matcher.random.long_strings
• regex_match_matcher.case_insensitive_flag.random.long_strings
• bool_equals_matcher.random.any
• remove_nulls_transformer.random.any
• is_sqli_matcher.random.long_strings
• lowercase_transformer.random.deep_containers
• ruleset_1.10.0.random.any
• string_equals_matcher.mixed
• signed_equals_matcher.random.long_strings
• ip_match_matcher.random.deep_containers
• exact_match_matcher.random.long_strings
• signed_equals_matcher.random.deep_containers
• regex_match_matcher.case_insensitive_option.random.long_strings
• is_sqli_matcher.mixed
• unsigned_equals_matcher.random.deep_containers
• is_sqli_matcher.random.any
• is_xss_matcher.random.any
• bool_equals_matcher.random.deep_containers
• unsigned_equals_matcher.random.any
• ruleset_1.10.0.random.long_strings
• exact_match_matcher.mixed
• float_equals_matcher.random.any
• regex_match_matcher.lowercase_transformer.random.long_strings
• regex_match_matcher.mixed
• regex_match_matcher.case_insensitive_flag.random.deep_containers
• regex_match_matcher.case_insensitive_option.mixed
• phrase_match_matcher.random.any
• remove_nulls_transformer.random.deep_containers
• bool_equals_matcher.mixed
• is_xss_matcher.random.long_strings
• ruleset_1.10.0.random.deep_containers
• ruleset_1.10.0.mixed
• float_equals_matcher.mixed
• regex_match_matcher.lowercase_transformer.mixed
• is_xss_matcher.random.deep_containers
• unsigned_equals_matcher.mixed
• schema_extraction_processor.random.long_strings
• remove_nulls_transformer.random.long_strings
• signed_equals_matcher.random.any
• is_xss_matcher.mixed
  Scenarios present only in candidate:
• regex_match_matcher.case_insensitive_option.random.1x128.128
• regex_match_matcher.lowercase_transformer.random.1x64.128
• unsigned_equals_matcher.random.1x1.1024
• remove_nulls_transformer.random.5x1.128
• regex_match_matcher.lowercase_transformer.random.1x1.128
• ruleset_1.10.0.random.20x1.128
• lowercase_transformer.random.1x1.128
• phrase_match_matcher.random.1x1.4096
• regex_match_matcher.lowercase_transformer.random.1x1.4096
• regex_match_matcher.case_insensitive_flag.random.1x1.1024
• string_equals_matcher.random.1x1.1024
• regex_match_matcher.case_insensitive_flag.random.1x64.128
• regex_match_matcher.random.20x1.128
• lowercase_transformer.random.5x1.128
• bool_equals_matcher.random.1x1.128
• regex_match_matcher.lowercase_transformer.random.1x16.128
• ip_match_matcher.random.1x1.4096
• regex_match_matcher.lowercase_transformer.random.20x1.128
• ip_match_matcher.random.20x1.128
• signed_equals_matcher.random.1x1.1024
• float_equals_matcher.random.1x1.4096
• exact_match_matcher.random.10x1.128
• float_equals_matcher.random.1x128.128
• signed_equals_matcher.random.1x1.4096
• regex_match_matcher.random.1x1.4096
• is_sqli_matcher.random.1x1.1024
• ruleset_1.10.0.random.1x64.128
• string_equals_matcher.random.1x1.512
• is_sqli_matcher.random.5x1.128
• is_sqli_matcher.random.1x16.128
• is_xss_matcher.random.10x1.128
• ruleset_1.10.0.random.1x1.1024
• regex_match_matcher.case_insensitive_flag.random.5x1.128
• regex_match_matcher.lowercase_transformer.random.10x1.128
• remove_nulls_transformer.random.1x1.1024
• string_equals_matcher.random.5x1.128
• remove_nulls_transformer.random.1x128.128
• bool_equals_matcher.random.1x128.128
• string_equals_matcher.random.1x64.128
• is_xss_matcher.random.20x1.128
• unsigned_equals_matcher.random.1x64.128
• regex_match_matcher.case_insensitive_flag.random.1x16.128
• bool_equals_matcher.random.1x256.128
• bool_equals_matcher.random.5x1.128
• lowercase_transformer.random.1x1.1024
• string_equals_matcher.random.1x256.128
• regex_match_matcher.case_insensitive_option.random.1x1.128
• unsigned_equals_matcher.random.1x16.128
• is_sqli_matcher.random.1x128.128
• bool_equals_matcher.random.20x1.128
• regex_match_matcher.random.1x16.128
• bool_equals_matcher.random.1x1.1024
• is_xss_matcher.random.1x16.128
• phrase_match_matcher.random.1x16.128
• remove_nulls_transformer.random.10x1.128
• remove_nulls_transformer.random.1x256.128
• regex_match_matcher.lowercase_transformer.random.1x1.1024
• lowercase_transformer.random.1x16.128
• is_xss_matcher.random.1x128.128
• ip_match_matcher.random.1x1.128
• float_equals_matcher.random.1x1.1024
• exact_match_matcher.random.1x1.128
• float_equals_matcher.random.1x1.128
• remove_nulls_transformer.random.1x1.128
• exact_match_matcher.random.1x128.128
• ip_match_matcher.random.1x64.128
• regex_match_matcher.random.1x1.1024
• phrase_match_matcher.random.20x1.128
• ruleset_1.10.0.random.1x16.128
• ruleset_1.10.0.random.1x256.128
• unsigned_equals_matcher.random.1x128.128
• lowercase_transformer.random.1x64.128
• phrase_match_matcher.random.1x256.128
• lowercase_transformer.random.20x1.128
• lowercase_transformer.random.1x1.4096
• ip_match_matcher.random.1x16.128
• regex_match_matcher.lowercase_transformer.random.1x256.128
• is_xss_matcher.random.1x1.1024
• regex_match_matcher.lowercase_transformer.random.5x1.128
• unsigned_equals_matcher.random.10x1.128
• exact_match_matcher.random.1x1.512
• is_sqli_matcher.random.1x64.128
• exact_match_matcher.random.20x1.128
• remove_nulls_transformer.random.1x64.128
• bool_equals_matcher.random.1x1.512
• unsigned_equals_matcher.random.20x1.128
• float_equals_matcher.random.1x64.128
• is_sqli_matcher.random.10x1.128
• regex_match_matcher.case_insensitive_option.random.1x16.128
• phrase_match_matcher.random.1x64.128
• regex_match_matcher.case_insensitive_option.random.1x64.128
• exact_match_matcher.random.1x256.128
• regex_match_matcher.case_insensitive_option.random.1x1.512
• ip_match_matcher.random.1x1.512
• regex_match_matcher.case_insensitive_flag.random.1x1.128
• ruleset_1.10.0.random.1x1.512
• signed_equals_matcher.random.1x256.128
• remove_nulls_transformer.random.20x1.128
• float_equals_matcher.random.1x1.512
• is_xss_matcher.random.1x256.128
• float_equals_matcher.random.1x256.128
• regex_match_matcher.random.1x1.512
• ip_match_matcher.random.1x1.1024
• remove_nulls_transformer.random.1x1.512
• is_sqli_matcher.random.20x1.128
• ruleset_1.10.0.random.1x128.128
• string_equals_matcher.random.1x16.128
• lowercase_transformer.random.1x1.512
• exact_match_matcher.random.1x1.4096
• exact_match_matcher.random.1x16.128
• is_sqli_matcher.random.1x1.512
• is_xss_matcher.random.1x1.128
• phrase_match_matcher.random.1x1.1024
• phrase_match_matcher.random.1x1.128
• regex_match_matcher.random.1x64.128
• ruleset_1.10.0.random.1x1.4096
• remove_nulls_transformer.random.1x16.128
• is_xss_matcher.random.1x1.512
• exact_match_matcher.random.5x1.128
• string_equals_matcher.random.20x1.128
• unsigned_equals_matcher.random.1x1.512
• regex_match_matcher.random.10x1.128
• regex_match_matcher.case_insensitive_option.random.1x1.1024
• unsigned_equals_matcher.random.1x1.128
• regex_match_matcher.random.1x256.128
• ip_match_matcher.random.10x1.128
• signed_equals_matcher.random.1x16.128
• regex_match_matcher.lowercase_transformer.random.1x1.512
• ruleset_1.10.0.random.5x1.128
• unsigned_equals_matcher.random.1x1.4096
• ruleset_1.10.0.random.10x1.128
• lowercase_transformer.random.1x128.128
• bool_equals_matcher.random.10x1.128
• phrase_match_matcher.random.1x1.512
• bool_equals_matcher.random.1x1.4096
• regex_match_matcher.case_insensitive_option.random.1x256.128
• regex_match_matcher.case_insensitive_flag.random.20x1.128
• exact_match_matcher.random.1x1.1024
• regex_match_matcher.lowercase_transformer.random.1x128.128
• signed_equals_matcher.random.1x128.128
• remove_nulls_transformer.random.1x1.4096
• unsigned_equals_matcher.random.5x1.128
• regex_match_matcher.case_insensitive_flag.random.1x128.128
• signed_equals_matcher.random.1x64.128
• string_equals_matcher.random.1x1.4096
• regex_match_matcher.case_insensitive_flag.random.1x1.4096
• signed_equals_matcher.random.10x1.128
• is_sqli_matcher.random.1x1.4096
• lowercase_transformer.random.1x256.128
• bool_equals_matcher.random.1x64.128
• regex_match_matcher.case_insensitive_option.random.10x1.128
• ip_match_matcher.random.1x256.128
• signed_equals_matcher.random.20x1.128
• string_equals_matcher.random.1x128.128
• float_equals_matcher.random.5x1.128
• signed_equals_matcher.random.1x1.128
• is_sqli_matcher.random.1x1.128
• is_xss_matcher.random.1x64.128
• regex_match_matcher.case_insensitive_flag.random.10x1.128
• signed_equals_matcher.random.5x1.128
• bool_equals_matcher.random.1x16.128
• is_xss_matcher.random.1x1.4096
• float_equals_matcher.random.20x1.128
• regex_match_matcher.case_insensitive_flag.random.1x256.128
• regex_match_matcher.case_insensitive_option.random.20x1.128
• ruleset_1.10.0.random.1x1.128
• is_sqli_matcher.random.1x256.128
• float_equals_matcher.random.1x16.128
• ip_match_matcher.random.1x128.128
• signed_equals_matcher.random.1x1.512
• unsigned_equals_matcher.random.1x256.128
• exact_match_matcher.random.1x64.128
• regex_match_matcher.case_insensitive_flag.random.1x1.512
• regex_match_matcher.random.1x128.128
• regex_match_matcher.random.5x1.128
• regex_match_matcher.random.1x1.128
• string_equals_matcher.random.10x1.128
• phrase_match_matcher.random.10x1.128
• regex_match_matcher.case_insensitive_option.random.5x1.128
• phrase_match_matcher.random.1x128.128
• is_xss_matcher.random.5x1.128
• ip_match_matcher.random.5x1.128
• regex_match_matcher.case_insensitive_option.random.1x1.4096
• phrase_match_matcher.random.5x1.128
• string_equals_matcher.random.1x1.128
• float_equals_matcher.random.10x1.128
• lowercase_transformer.random.10x1.128

Found 13 performance improvements and 0 performance regressions! Performance is the same for 0 metrics, 4 unstable metrics.

scenario:bool_equals_matcher.valid

• 🟩 execution_time [-25.569µs; -23.295µs] or [-98.862%; -90.068%]

scenario:exact_match_matcher.valid

• 🟩 execution_time [-198.750µs; -183.502µs] or [-103.118%; -95.207%]

scenario:float_equals_matcher.valid

• 🟩 execution_time [-27.957µs; -25.819µs] or [-97.866%; -90.380%]

scenario:ip_match_matcher.valid

• 🟩 execution_time [-41.064µs; -37.457µs] or [-99.515%; -90.774%]

scenario:is_xss_matcher.valid

• 🟩 execution_time [-5.559ms; -5.108ms] or [-104.196%; -95.734%]

scenario:lowercase_transformer.valid

• 🟩 execution_time [-199.743µs; -181.718µs] or [-103.909%; -94.532%]

scenario:phrase_match_matcher.valid

• 🟩 execution_time [-561.934µs; -519.497µs] or [-103.581%; -95.758%]

scenario:regex_match_matcher.case_insensitive_option.valid

• 🟩 execution_time [-474.276µs; -436.873µs] or [-103.670%; -95.494%]

scenario:regex_match_matcher.lowercase_transformer.valid

• 🟩 execution_time [-651.298µs; -595.795µs] or [-104.111%; -95.238%]

scenario:regex_match_matcher.valid

• 🟩 execution_time [-217.383µs; -197.098µs] or [-103.922%; -94.225%]

scenario:ruleset_1.10.0.valid

• 🟩 execution_time [-44.504ms; -43.149ms] or [-101.170%; -98.091%]

scenario:signed_equals_matcher.valid

• 🟩 execution_time [-25.036µs; -22.795µs] or [-98.897%; -90.044%]

scenario:unsigned_equals_matcher.valid

• 🟩 execution_time [-25.987µs; -23.710µs] or [-98.919%; -90.250%]

[Anilm3]

The variability of benchmarks is due to the fact that tests in master are essentially more complex and buggy than the ones in this PR.