Allow to manage Elasticsearch nodes separately from benchmarking

[danielmitterdorfer]

Currently Rally can be used as a load generator and to manage Elasticsearch instances (building a distribution from sources, installing and configuring Elasticsearch as well as launching / stopping it). It is possible to use Rally as a standalone load generator (--pipeline=benchmark-only) but it is not possible to use Rally only to manage Elasticsearch nodes. In order to do that, we should allow to call the provisioning and launch phases as separate subcommands in Rally. We should only support a local mode though, i.e. it is not possible to only setup a cluster from the coordinator node but users must invoke Rally instead on each of the target nodes. As a corollary, each command applies to only one node.

We will add the following subcommands:

• provision: Provisions a single node. The artefact might be built from sources or a distribution that has been downloaded.
• start: Starts an already provisioned node.
• stop: Stops an already started node.

Implementation note: None of these commands should rely on the actor system and instead run the commands in the main Rally process.

Provisioning

This covers all steps from retrieving the binary (either download or build it via Gradle) until a fully configured node is installed. Note that there are essentially no new steps, it's just that we expose this now as a dedicated subcommand. One tricky aspect is that we are currently able to create a unique node name by increasing a counter (rally-node-{N}), see also:

rally/esrally/mechanic/mechanic.py

Lines 178 to 186 in f575790

	def nodes_by_host(ip_port_pairs):
	nodes = {}
	node_id = 0
	for ip_port in ip_port_pairs:
	if ip_port not in nodes:
	nodes[ip_port] = []
	nodes[ip_port].append(node_id)
	node_id += 1
	return nodes

)

This will not be possible anymore with the provision subcommand as it does not have a global view of the cluster but rather a per-node view.

Post condition

After the provisioner has (successfully) run the following conditions are met:

• An Elasticsearch node is installed in a well-known directory ~/.rally/benchmarks/nodes/$ID/install where $ID is a globally unique id generated by the provisioner.
• A meta-data file has been written to ~/.rally/benchmarks/nodes/$ID/provisioner.json which contains the necessary data that needs to be exchanged between the provisioner and the launcher (see NodeConfiguration for data that are exchanged between those two).
• $ID is written to stdout. This id can then be used later on to start and stop this node.

Note: It might make sense that we specify all telemetry devices as part of the provisioning process regardless when they get attached (some are only attached on launch) and exchange the necessary information via the provisioner metadata.

Start

This will launch a single node. The launcher process will not be a parent process of Elasticsearch (this is a change to the previous behavior) and will instead terminate once the node has successfully started. As input it will use the provisioner id to read the provisioner metadata (NodeConfiguration) to startup the node. Depending on the launcher type we may need to persist the PID of the Elasticsearch process. For plain vanilla Elasticsearch we might just run it as a daemon. Any files should go to ~/.rally/benchmarks/races/$ID if needed.

Post condition

After the launcher has (successfully) run the following conditions are met:

• An Elasticsearch process is running
• Its PID is recorded in a well-known location that can be retrieved again via $ID.

Stop

This will stop a node that has previously been started. If --preserve-install=true is specified on the command line, the node's installation and data directory will be cleaned up. Logs and telemetry data should always be kept (all of this is no change to previous behavior).

Post condition

• The corresponding Elasticsearch process is stopped
• All files except any telemetry data and logs have been deleted (unless --preserve-install=true).

Preparatory work

As a preparation we should:

• Switch from installation directories based on the race timestamp to installation directories based on unique ids (~/.rally/benchmarks/nodes/$ID) - see Change filestore to be indexed by unique ID #720
• Move telemetry devices that currently rely on ES_JAVA_OPTS from the launcher to the provisioner and instead persist the necessary information in config/jvm.options. - see Change telemetry devices to rely on jvm.config instead of ES_JAVA_OPTS #711
• Revisit all remaining telemetry devices and ensure they will not rely on Rally being a parent process of Elasticsearch. Specifically for the DiskIo telemetry device we should ensure that it persists the disk counters at the beginning of a benchmark and can read it again afterwards. - see Update DiskIo telemetry device to persist the counters #731
• Run Elasticsearch as a daemon (we should do the same for the container launcher). This will simplify exposing the start command separately. - see Implement ES daemon-mode in process launcher #701

Tasks

• Expose race id as command line parameter - see Expose race-id as command line parameter #778
• Calculate system metrics per node - see Calculate system metrics per node #803
• Implement start, stopand provision as subcommands - see Manage Elasticsearch nodes with dedicated subcommands #830

Follow-up work

We will continue to support the actor-system-based approach for now but we intend to remove this coordination layer at some point and let users choose freely how they want to manage coordination (e.g. via Ansible) as this also allows for more complex setups. Doing this will require additional preparation though so we will tackle this in a separate issue.