Add recovery-stats telemetry device

[danielmitterdorfer]

With this commit we add a new telemetry device called recovery-stats
which can be used to retrieve statistics about ongoing recoveries while
running a benchmark.

Closes #635

[danielmitterdorfer]

I've tested the changes with the attached test-track. That track restores an index taxis from a snapshot.

Test scenarios (all with an ES metrics store):

• esrally --pipeline=from-sources-skip-build --telemetry=recovery-stats --track-path=/path/to/test-track
• esrally --pipeline=from-sources-skip-build --telemetry=recovery-stats --track-path=/path/to/test-track --telemetry-params="recovery-stats-indices:'ta*'"
• esrally --pipeline=from-sources-skip-build --telemetry=recovery-stats --track-path=/path/to/test-track --telemetry-params="recovery-stats-indices:'bla*'"
• esrally --pipeline=from-sources-skip-build --telemetry=recovery-stats --track-path=/path/to/test-track --telemetry-params="recovery-stats-indices:'ta*',recovery-stats-sample-interval:2"
• esrally --pipeline=from-sources-skip-build --telemetry=recovery-stats --track-path=/path/to/test-track --telemetry-params="telemetry-params.json"

telemetry-params.json:

{
  "recovery-stats-indices": {
    "default": "taxi*"
  },
  "recovery-stats-sample-interval": 2
}

[dliappis]

LGTM thanks! I left a small note for the docs.

[dliappis]

This table seems to be indented by +1 character to the right hence not aligned with the title. Is it on purpose?

[danielmitterdorfer]

No. IIRC I just pressed tab and assumed it did the right thing.

[danielmitterdorfer]

I've addressed this in b61b64e

[dliappis]

💯 thanks for this.

[danielmitterdorfer]

I've tested this also with the in-memory metrics store and although this telemetry device is not useful there (it only writes raw metrics entries which get lost if the metrics store is transient), it should at least not break it. This is addressed in db94883. @dliappis can you please have another look at those changes?

[dliappis]

Tested it with both internal and external metric store. Works great, thanks!

LGTM

[danielmitterdorfer]

Thanks! Merged.