client-options "verify_certs:false" seems to be ignored

[suckowbiz]

Rally version (get with esrally --version):
0.10.0

Invoked command:

rally --on-error="abort" \
    --report-file=/root/.rally/report.md \
    --track="http_logs" \
    --target-hosts="https://1.2.3.4:9200,https://1.2.3.5:9200" \
    --pipeline=benchmark-only \
    --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'${ESRALLY_USERNAME}',basic_auth_password:'${ESRALLY_PASSWORD}'"

Configuration file (located in ~/.rally/rally.ini)):

>_ cat rally.ini
[meta]
config.version = 15

[system]
env.name = local

[node]
root.dir = /root/.rally/benchmarks

[runtime]
java.home = /usr/lib/jvm/java-8-openjdk-amd64

[benchmarks]
local.dataset.cache = ${node:root.dir}/data

[reporting]
datastore.type = in-memory
datastore.host =
datastore.port =
datastore.secure =
datastore.user =
datastore.password =

[tracks]
default.url = https://github.com/elastic/rally-tracks

[teams]
default.url = https://github.com/elastic/rally-teams

[defaults]
preserve_benchmark_candidate = False

[distributions]
release.1.url = https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-{{VERSION}}.tar.gz
release.2.url = https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/{{VERSION}}/elasticsearch-{{VERSION}}.tar.gz
release.url = https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{{VERSION}}.tar.gz
release.cache = true

JVM version:

>_ java -version
openjdk version "1.8.0_162"
OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-1~bpo8+1-b12)
OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)

OS version:
Debian GNU/Linux 8 (jessie)

Description of the problem including expected versus actual behavior:
Expected: Rally runs as 0.9.4 did.
Actual: Rally fails with CERTIFICATE_VERIFY_FAILED exception.

Steps to reproduce:

1. Run rally as stated

Provide logs (if relevant):

[ERROR] Cannot race. ConnectionError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)) caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify f
ailed (_ssl.c:833))
        Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/local/lib/python3.6/ssl.py", line 814, in __init__
    self.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

Describe the feature:

[danielmitterdorfer]

Can you please paste the respective snippet from your log file when the client is created? It should look similar to the one below:

2018-04-18 12:59:55,886 PID:21548 rally.client INFO Creating ES client connected to [{'host': '127.0.0.1', 'port': 39200}] with options [{'basic_auth_user': 'rally', 'basic_auth_password': '*****', 'use_ssl': True, 'verify_certs': False, 'timeout': 60}]
2018-04-18 12:59:55,886 PID:21548 rally.client INFO SSL support: on
2018-04-18 12:59:55,893 PID:21548 rally.client INFO SSL certificate verification: off
2018-04-18 12:59:55,893 PID:21548 rally.client WARNING User has enabled SSL but disabled certificate verification. This is dangerous but may be ok for a benchmark. Disabling urllib warnings now to avoid a logging storm. See https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings for details.
2018-04-18 12:59:55,894 PID:21548 rally.client INFO HTTP basic authentication: on
2018-04-18 12:59:55,895 PID:21548 rally.client INFO HTTP compression: off

Can you please also paste the output of pip3 list?

[danielmitterdorfer]

I think I know the root cause. I have pushed a change which hopefully fixes this issue. It is contained in Rally 0.10.1. Can you please upgrade to Rally 0.10.1 and retry?

[suckowbiz]

Your fix solved the bug. ;)

[danielmitterdorfer]

Thanks for the quick verification @suckowbiz! The reason was that we did not copy the client options but modified it. So the next time a client was created, it was done based on a modified version of the client options which missed a few properties. Fixed in c6198fe.

[gittygoo]

I seem to be getting this same problem with esrally 1.4.1 and 2.0.0
esrally --pipeline benchmark-only --track=nyc_taxis --challenge append-no-conflicts-index-only --target-host=https://a.b.c.d:30537 --client-options="use_ssl:true,basic_auth_user:'admin',basic_auth_password:'admin',verify_certs:false"

[dliappis]

@gittygoo Would you mind moving this observation in https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/6/rally? We also use verify_certs:false and haven't observed the issue you are reporting, but let's move the discussion to a dedicated discuss topic. Also while at it please include relevant parts of ~/.rally/logs/rally.log

[ZijunXu]

I find that the extra quotation marks can be problematic in some cases. I tried to run the elastic rally in a k8s job.
The extra double quotes around --client-options gave me the same symptom as this issue. It cannot parse the --client-options properly.

containers:
        - name: rally
          image: elastic/rally:2.0.2
          imagePullPolicy: Always
          args:
          - --track=percolator
          - --pipeline=benchmark-only
          - --target-hosts=il-search-elasticsearch-es-ingest:9200
          - --client-options="basic_auth_user:'rdeniro',basic_auth_password:'mypassword',use_ssl:true,verify_certs:false"

The correct parameter syntax should be like this: (without any double quote mark)

apiVersion: batch/v1
kind: Job
metadata:
  name: elastic-rally-job
spec:
  backoffLimit: 1
  template:
    metadata:
      labels:
        app: elastic-rally
    spec:
      restartPolicy: Never
      containers:
        - name: rally
          image: elastic/rally:2.0.2
          imagePullPolicy: Always
          args:
          - --track=percolator
          - --pipeline=benchmark-only
          - --target-hosts=il-search-elasticsearch-es-ingest:9200
          - --client-options=timeout:60,use_ssl:true,verify_certs:false,basic_auth_user:'rdeniro',basic_auth_password:'mypassword'

This may not be an issue if you run the rally in a terminal.