Bug Fixes for the eosio-test-stability Pipeline

[kj4ezj]

Change Description

While working on service item AUTO-1004 to provide a low-effort stability testing solution to Blockchain for their unit and integration tests, I found a way to satisfy AUTO-495 so that the pipeline upload script natively supports stability testing within the organizational Buildkite job limit.

Pull request 10460 added support for this eosio-test-stability pipeline, but I found a bug while porting that code to our other branches that caused the test scripts to ignore $1 when TEST is set in the environment. It makes more sense for the test name passed to the test scripts to take precedence over a variable set in the environment, because a user might not even know about it. Doing it this way also allows the stability testing pipeline to take advantage of container-level parallelization when a user directs it to run all of the tests. This was my motivation for opening another pull request.

---

Instructions

Stability testing of EOSIO unit and integration tests is done in the eosio-test-stability pipeline. It will take thousands of runs of any given test to identify it as "stable" or "unstable". Runs should be split evenly across "pinned" (fixed dependency version) and "unpinned" (default dependency version) builds because, sometimes, test instability is only expressed in one of these environments. Finally, stability testing should be performed on the Linux fleet first because this fleet is effectively infinite. Once stability is demonstrated on Linux, testing can be performed on the finite macOS Anka fleet.

The eosio-test-stability pipeline uses the same pipeline upload script as eosio, eosio-build-unpinned, and eosio-lrt, so all variables from the pipeline documentation apply. However, there are five primary environment variables relevant to stability testing:

PINNED='true|false'    # whether to perform the test with pinned dependencies, or default dependencies
ROUNDS='ℕ'             # natural number defining the number of gated rounds of tests to generate
ROUND_SIZE='ℕ'         # number of test steps to generate per operating system, per round
SKIP_MAC='true|false'  # conserve finite macOS Anka agents by excluding them from your testing
TEST='name'            # PCRE expression defining the tests to run, preceded by '^' and followed by '$'
TIMEOUT='ℕ'            # set timeout in minutes for all Buildkite steps

The TEST variable is parsed as pearl-compatible regular expression where the expression in TEST is preceded by ^ and followed by $. To specify one test, set TEST equal to the test name (e.g. TEST='read_only_query'). Specify two tests as TEST='(nodeos_short_fork_take_over_lr_test|read_only_query)'. Or, perhaps, you want all of the restart_scenarios tests. Then, you could define TEST='restart-scenario-test-.*' and Buildkite will generate ROUND_SIZE steps each round for each operating system for all three restart scenarios tests.

The number of total test runs will be:

RUNS = ROUNDS * ROUND_SIZE * OS_COUNT * TEST_COUNT # where:
OS_COUNT   = 'ℕ' # the number of supported operating systems
TEST_COUNT = 'ℕ' # the number of tests matching the PCRE filter in TEST

We recommend stability testing one test per build with two builds per test, on Linux at first. Kick off one pinned build on Linux...

PINNED='true'
ROUNDS='42'
ROUND_SIZE'5'
SKIP_MAC='true'
TEST='read_only_query'

...and one unpinned build on Linux:

PINNED='true'
ROUNDS='42'
ROUND_SIZE'5'
SKIP_MAC='true'
TEST='read_only_query'

Once the Linux runs have proven stable, and if instability was observed on macOS, kick off two equivalent builds on macOS instead of Linux. One pinned build on macOS...

PINNED='true'
ROUNDS='42'
ROUND_SIZE'5'
SKIP_LINUX='true'
SKIP_MAC='false'
TEST='read_only_query'

...and one unpinned build on macOS:

PINNED='true'
ROUNDS='42'
ROUND_SIZE'5'
SKIP_LINUX='true'
SKIP_MAC='false'
TEST='read_only_query'

If these runs are against eos:develop and develop has five supported operating systems, this pattern would consist of 2,100 runs per test across all four builds. If the runs are against eos:release/2.1.x which, at the time of this writing, supports eight operating systems, this pattern would consist of 3,360 runs per test across all four builds. This gives you strong confidence that any test instability occurs less than 1% of the time.

---

See Also

• eos
  • Pull Request 9675 -- nodeos_irreversible_mode_lr_test Stability Testing
  • Pull Request 10446 -- Stability Test nodeos_short_fork_take_over_lr_test
  • Pull Request 10460 -- Support Stability Testing for Tests + CI Bug Fixes (eos:develop)
  • Pull Request 10463 -- Support Stability Testing for Tests + CI Bug Fixes (eos:release/2.1.x)
  • Pull Request 10464 -- Support Stability Testing for Tests + CI Bug Fixes (eos:release/2.0.x)
  • Pull Request 10465 -- Bug Fixes for the eosio-test-stability Pipeline (eos:develop)
  • Pull Request 10467 -- Another Bug Fix for the eosio-test-stability Pipeline (eos:develop)
  • Pull Request 10468 -- Another Bug Fix for the eosio-test-stability Pipeline (eos:release/2.1.x)
  • Pull Request 10469 -- Another Bug Fix for the eosio-test-stability Pipeline (eos:release/2.0.x)
• eos-buildkite-pipelines
  • Commit 50cd1ca -- Modernize eosio-test-stability pipeline
  • Commit 326177f -- Reduce number of default rounds so we don't exceed the org-wide Buildkite job limit
  • Commit 8e33ef4 -- Skip macOS by default

Change Type

Select ONE:

• Documentation
• Stability bug fix
• Other
• Other - special case

Support stability testing of EOSIO tests.

Testing Changes

Select ANY that apply:

• New Tests
• Existing Tests
• Test Framework
• CI System
• Other

Support stability testing of EOSIO tests, plus other bug fixes to the pipeline upload script.

Consensus Changes

• Consensus Changes

None.

API Changes

• API Changes

None.

Documentation Additions

• Documentation Additions

None.