Reduce benchmark noise

[Anilm3]

• Remove valid tests due to noise
• Add per-test cache priming
• Add fixture regex (--fixture)
• Increase per-scenario warmup iterations
• Increased object sizes to reduce noise of shorter tests

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (044f675) 83.30% compared to head (a8aaf76) 83.30%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #257   +/-   ##
=======================================
  Coverage   83.30%   83.30%           
=======================================
  Files         106      106           
  Lines        4188     4188           
  Branches     1923     1923           
=======================================
  Hits         3489     3489           
  Misses        280      280           
  Partials      419      419           

Flag	Coverage Δ
waf_test_none	83.20% <ø> (ø)
waf_test_sse2	83.27% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-02-08 14:04:09

Comparing candidate commit a8aaf76 in PR branch anilm3/benchmark-changes with baseline commit 044f675 in branch master.

Some scenarios are present only in baseline or only in candidate runs. If you didn't create or remove some scenarios in your branch, this maybe a sign of crashed benchmarks 💥💥💥
Check Gitlab CI job log to find if any benchmark has crashed.

Scenarios present only in baseline:

• remove_nulls_transformer.eval.valid
• is_xss_matcher.eval.valid
• exact_match_matcher.eval.valid
• phrase_match_matcher.eval.valid
• float_equals_matcher.eval.valid
• ruleset_1.10.0.eval.valid
• is_sqli_matcher.eval.valid
• regex_match_matcher.lowercase_transformer.eval.valid
• phrase_match_matcher.enforce_word_boundary.eval.valid
• regex_match_matcher.case_insensitive_option.eval.valid
• unsigned_equals_matcher.eval.valid
• bool_equals_matcher.eval.valid
• regex_match_matcher.eval.valid
• string_equals_matcher.eval.valid
• lowercase_transformer.eval.valid
• ip_match_matcher.eval.valid
• signed_equals_matcher.eval.valid
• regex_match_matcher.case_insensitive_flag.eval.valid

Found 16 performance improvements and 3 performance regressions! Performance is the same for 0 metrics, 0 unstable metrics.

scenario:bool_equals_matcher.random

• 🟩 execution_time [-3.362µs; -3.342µs] or [-50.038%; -49.751%]

scenario:exact_match_matcher.random

• 🟩 execution_time [-22.241µs; -22.105µs] or [-19.615%; -19.496%]

scenario:float_equals_matcher.random

• 🟩 execution_time [-3.293µs; -3.250µs] or [-49.693%; -49.047%]

scenario:ip_match_matcher.random

• 🟩 execution_time [-8.002µs; -7.950µs] or [-51.911%; -51.575%]

scenario:is_sqli_matcher.random

• 🟥 execution_time [+321.785µs; +322.252µs] or [+59.030%; +59.116%]

scenario:is_xss_matcher.random

• 🟩 execution_time [-661.773µs; -660.730µs] or [-11.716%; -11.697%]

scenario:lowercase_transformer.random

• 🟩 execution_time [-12.756µs; -12.611µs] or [-13.918%; -13.760%]

scenario:phrase_match_matcher.enforce_word_boundary.random

• 🟩 execution_time [-255.926µs; -255.621µs] or [-36.852%; -36.808%]

scenario:phrase_match_matcher.random

• 🟩 execution_time [-35.250µs; -34.661µs] or [-6.918%; -6.802%]

scenario:regex_match_matcher.case_insensitive_flag.random

• 🟩 execution_time [-50.083µs; -49.853µs] or [-14.754%; -14.686%]

scenario:regex_match_matcher.case_insensitive_option.random

• 🟩 execution_time [-49.651µs; -49.423µs] or [-14.640%; -14.573%]

scenario:regex_match_matcher.lowercase_transformer.random

• 🟩 execution_time [-50.833µs; -50.560µs] or [-12.034%; -11.970%]

scenario:regex_match_matcher.random

• 🟩 execution_time [-47.950µs; -47.772µs] or [-34.145%; -34.018%]

scenario:remove_nulls_transformer.random

• 🟩 execution_time [-47.509µs; -47.164µs] or [-5.623%; -5.582%]

scenario:ruleset_1.10.0.random

• 🟥 execution_time [+566.916ms; +566.939ms] or [+268.038%; +268.049%]

scenario:schema_extraction_processor.random

• 🟥 execution_time [+529.007µs; +529.280µs] or [+5998.618%; +6001.723%]

scenario:signed_equals_matcher.random

• 🟩 execution_time [-3.128µs; -3.092µs] or [-46.844%; -46.306%]

scenario:string_equals_matcher.random

• 🟩 execution_time [-5.158µs; -5.110µs] or [-43.675%; -43.269%]

scenario:unsigned_equals_matcher.random

• 🟩 execution_time [-3.238µs; -3.220µs] or [-48.774%; -48.492%]