Skip to content
This repository has been archived by the owner on Aug 2, 2022. It is now read-only.

Support Stability Testing for Tests + CI Bug Fixes #10460

Merged
merged 19 commits into from
Jun 28, 2021
Merged

Conversation

kj4ezj
Copy link
Contributor

@kj4ezj kj4ezj commented Jun 25, 2021

Change Description

While working on service item AUTO-1004 to provide a low-effort stability testing solution to Blockchain for their unit and integration tests, I found a way to satisfy AUTO-495 so that the pipeline upload script natively supports stability testing within the organizational Buildkite job limit.

This pull request introduces the following:

  • Restores the ability to run the pipeline upload script locally to see what YAML it generates in your shell or editor of choice
  • Fixes the code that prevents pipeline documentation from being uploaded when the pipeline upload step is retried
  • Fixes the alignment of an fi statement
  • Alphabetizes the serial and long-running tests by test name so it is easier to find tests in the list of jobs
  • Fixes null SKIP variable bug
  • Removes extraneous newlines
  • Requires explicit action from the user to run all test suites more than once

Additionally, stability testing of EOSIO tests is supported natively by the Buildkite pipelines, without having to edit the pipeline upload script or cherry-pick code from pull request 9675.


Instructions

Stability testing of EOSIO unit and integration tests is done in the eosio-test-stability pipeline. It will take thousands of runs of any given test to identify it as "stable" or "unstable". Runs should be split evenly across "pinned" (fixed dependency version) and "unpinned" (default dependency version) builds because, sometimes, test instability is only expressed in one of these environments. Finally, stability testing should be performed on the Linux fleet first because this fleet is effectively infinite. Once stability is demonstrated on Linux, testing can be performed on the finite macOS Anka fleet.

The eosio-test-stability pipeline uses the same pipeline upload script as eosio, eosio-build-unpinned, and eosio-lrt, so all variables from the pipeline documentation apply. However, there are five primary environment variables relevant to stability testing:

PINNED='true|false'    # whether to perform the test with pinned dependencies, or default dependencies
ROUNDS=''             # natural number defining the number of gated rounds of tests to generate
ROUND_SIZE=''         # number of test steps to generate per operating system, per round
SKIP_MAC='true|false'  # conserve finite macOS Anka agents by excluding them from your testing
TEST='name'            # PCRE expression defining the tests to run, preceded by '^' and followed by '$'
TIMEOUT=''            # set timeout in minutes for all Buildkite steps

The TEST variable is parsed as pearl-compatible regular expression where the expression in TEST is preceded by ^ and followed by $. To specify one test, set TEST equal to the test name (e.g. TEST='read_only_query'). Specify two tests as TEST='(nodeos_short_fork_take_over_lr_test|read_only_query)'. Or, perhaps, you want all of the restart_scenarios tests. Then, you could define TEST='restart-scenario-test-.*' and Buildkite will generate ROUND_SIZE steps each round for each operating system for all three restart scenarios tests.

The number of total test runs will be:

RUNS = ROUNDS * ROUND_SIZE * OS_COUNT * TEST_COUNT # where:
OS_COUNT   = '' # the number of supported operating systems
TEST_COUNT = '' # the number of tests matching the PCRE filter in TEST

We recommend stability testing one test per build with two builds per test, on Linux at first. Kick off one pinned build on Linux...

PINNED='true'
ROUNDS='42'
ROUND_SIZE'5'
SKIP_MAC='true'
TEST='read_only_query'

...and one unpinned build on Linux:

PINNED='true'
ROUNDS='42'
ROUND_SIZE'5'
SKIP_MAC='true'
TEST='read_only_query'

Once the Linux runs have proven stable, and if instability was observed on macOS, kick off two equivalent builds on macOS instead of Linux. One pinned build on macOS...

PINNED='true'
ROUNDS='42'
ROUND_SIZE'5'
SKIP_LINUX='true'
SKIP_MAC='false'
TEST='read_only_query'

...and one unpinned build on macOS:

PINNED='true'
ROUNDS='42'
ROUND_SIZE'5'
SKIP_LINUX='true'
SKIP_MAC='false'
TEST='read_only_query'

If these runs are against eos:develop and develop has five supported operating systems, this pattern would consist of 2,100 runs per test across all four builds. If the runs are against eos:release/2.1.x which, at the time of this writing, supports eight operating systems, this pattern would consist of 3,360 runs per test across all four builds. This gives you strong confidence that any test instability occurs less than 1% of the time.


See Also

Change Type

Select ONE:

  • Documentation
  • Stability bug fix
  • Other
  • Other - special case

Support stability testing of EOSIO tests.

Testing Changes

Select ANY that apply:

  • New Tests
  • Existing Tests
  • Test Framework
  • CI System
  • Other

Support stability testing of EOSIO tests, plus other bug fixes to the pipeline upload script.

Consensus Changes

  • Consensus Changes

None.

API Changes

  • API Changes

None.

Documentation Additions

  • Documentation Additions

None.

@kj4ezj kj4ezj changed the title Minor Pipeline Upload Fixes Support Stability Testing for Tests Jun 25, 2021
@kj4ezj kj4ezj changed the title Support Stability Testing for Tests Support Stability Testing for Tests + CI Bug Fixes Jun 25, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants