Can fluent-bit talk with Fluentd v0.14 in_forward using TLS? #350

okkez · 2017-08-09T08:01:42Z

I'm trying to use fluent-bit with Fluentd v0.14.20 in_forward using TLS with private CA.
fluent-bit: 70d0594 (build myself)

fluent-bit.conf:

[SERVICE]
    Flush 1
    Daemon off
    Log_Level info

[INPUT]
    Name tail
    Path dummy.log
    Tag tag.test

[OUTPUT]
    Name forward
    Match tag.test
    Host 127.0.0.1
    Port 24224
    Shared_Key fluentd
    Self_Hostname localhost
    tls on
    tls.verify off
    tls.crt_file ca_cert.pem

fluent.conf:

<source>
  @type forward
  port 24224
  <transport tls>
    ca_cert_path /home/kenji/wc/influent/out/ca_cert.pem
    ca_private_key_path /home/kenji/wc/influent/out/ca_key.pem
    ca_private_key_passphrase password
  </transport>
  <security>
    self_hostname localhost
    shared_key fluentd
  </security>
</source>

<match test>
  @type stdout
  <buffer>
    flush_interval 1s
  </buffer>
</match>

I got errors from fluent-bit:

2017/08/09 16:52:04] [error] [out_fw] error writing content body
[2017/08/09 16:52:04] [error] [out_fw] error writing content body
[2017/08/09 16:52:04] [error] [out_fw] error writing content body
[2017/08/09 16:52:04] [error] [out_fw] error writing content body

I got errors from Fluentd in_forward:

2017-08-09 16:52:04 +0900 [trace]: #0 plugin_helper/server.rb:669:rescue in try_tls_accept: unexpected error before accepting TLS connection error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 state=SSLv2/v3 read client hello A: unknown protocol"
2017-08-09 16:52:04 +0900 [trace]: #0 plugin_helper/server.rb:669:rescue in try_tls_accept: unexpected error before accepting TLS connection error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 state=SSLv2/v3 read client hello A: unknown protocol"
2017-08-09 16:52:04 +0900 [trace]: #0 plugin_helper/server.rb:669:rescue in try_tls_accept: unexpected error before accepting TLS connection error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 state=SSLv2/v3 read client hello A: unknown protocol"
2017-08-09 16:52:04 +0900 [trace]: #0 plugin_helper/server.rb:669:rescue in try_tls_accept: unexpected error before accepting TLS connection error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 state=SSLv2/v3 read client hello A: unknown protocol"

Configuration for in_forward works with Fluentd's out_forward using following configuration:

<source>
  @type dummy
  dummy {"message": "this is test"}
  tag test
</source>

<filter test>
  @type stdout
</filter>
<match test>
  @type forward
  # insecure fasle
  transport tls
  tls_cert_path ca_cert.pem
  tls_verify_hostname false # for test
  heartbeat_type none
  <server>
    # first server
    host 127.0.0.1
    port 24224
  </server>
  <buffer>
    flush_interval 1
  </buffer>
</match>

I've created CA files following command:

openssl req -new -x509 -newkey rsa:2048 -out ca_cert.pem -keyout ca_key.pem -days 36500 \
-passout pass:password -subj "/C=JP/ST=CA/L=Tokyo/O=Influent/OU=Influent/CN=localhost"

Additional information, without TLS works well.

The text was updated successfully, but these errors were encountered:

edsiper · 2017-08-10T04:04:05Z

@okkez

we were doing some troubleshooting today. We found that Fluentd v0.14 uses the Basic Constraints extension when configuring SSL/TLS that makes the problem visible, in Fluentd v0.12 this is not enabled and the communication works well.

No solution is yet available as we are figuring out where the fix or improvement should be done. More updates within the week.

okkez · 2017-08-10T05:38:55Z

Thank you for your information. I will check updates.

edsiper · 2017-08-10T15:41:52Z

@okkez

I managed to make it work, it was a setup issue. I've generated a self-signed cert as this:

$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365

I've added a password 'test'

then in Fluentd 0.14.20 I started it with the following config:

<source>
  @type forward
  port 24224
  <transport tls>
     ca_cert_path /home/edsiper/coding/fluent-bit/certs/tmp/cert.pem
     ca_private_key_path /home/edsiper/coding/fluent-bit/certs/tmp/key.pem
     ca_private_key_passphrase test
  </transport>
  <security>
    self_hostname localhost
    shared_key fluentd
  </security>
</source>

<match **>
  @type stdout
  <buffer>
    flush_interval 1s
  </buffer>
</match>

then started Fluent Bit as follows:

$ bin/fluent-bit  \
         -i random \
         -o forward://127.0.0.1:24224 \
         -p tls=on -p tls.verify=off \
         -p tls.cert_file=../certs/tmp/cert.pem \
         -p tls.key_file=../certs/tmp/key.pem \
         -p tls.key_passwd=test \
         -p shared_key=fluentd -f 1

let me know how it goes

okkez · 2017-08-14T01:29:13Z

I could not confirm working with TLS.
I got following errors:

$ bin/fluent-bit  \
         -i random \
         -o forward://127.0.0.1:24224 \
         -p tls=on -p tls.verify=off \
         -p tls.cert_file=tmp/cert.pem \
         -p tls.key_file=tmp/key.pem \
         -p tls.key_passwd=password \
         -p shared_key=fluentd -f 1
Fluent-Bit v0.12.0
Copyright (C) Treasure Data

[2017/08/14 10:13:17] [ info] [engine] started
[2017/08/14 10:13:17] [error] [TLS] Invalid Key file: tmp/key.pem
[2017/08/14 10:13:17] [error] [output forward.0] error initializing TLS context

My openssl command version is 1.1.0f-4.

I removed tls.key_file and tls.key_passwd option and got following errors:

$ bin/fluent-bit  \
          -i random \
          -o forward://127.0.0.1:24224 \
          -p tls=on -p tls.verify=off \
          -p tls.cert_file=tmp/cert.pem \
          -p shared_key=fluentd -f 1
Fluent-Bit v0.12.0
Copyright (C) Treasure Data

[2017/08/14 10:23:43] [ info] [engine] started
[2017/08/14 10:23:44] [error] [io_tls] flb_io_tls.c:287 X509 - The CRT/CRL/CSR format is invalid, e.g. different type expected
[2017/08/14 10:23:44] [error] [out_fw] no upstream connections available
... repeat above 2 lines ...

okkez · 2017-08-14T01:49:12Z

My key.pem start with -----BEGIN ENCRYPTED PRIVATE KEY----- and end with -----END ENCRYPTED PRIVATE KEY-----.

edsiper · 2017-08-14T18:47:38Z

what command did you use to generate the certificates ?

okkez · 2017-08-14T23:36:33Z

I've tried following 4 commands:

enerian.local is my hostname.

$ openssl req -new -x509 -newkey rsa:4096 -out ca_cert.pem -keyout ca_key.pem -days 36500 \
-passout pass:password -subj "/C=JP/ST=CA/L=Tokyo/O=Influent/OU=Influent/CN=enerian.local"

$ openssl req -new -x509 -newkey rsa:4096 -out cert.pem -keyout key.pem -days 365

$ openssl req -x509 -newkey rsa:4096 -out cert.pem -keyout key.pem -days 365

$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 \
-passout pass:password -subj "/C=JP/ST=CA/L=Tokyo/O=Influent/OU=Influent/CN=enerian.local"

tarokkk · 2017-09-20T12:30:10Z

Why is this issue mark as fixed? This is not working for me either. I tried the method described here: #350 (comment) and got the exact same errors as @okkez above...

edsiper · 2017-09-20T15:36:40Z

I've removed the fixed label.

would you please re-run Fluent Bit but adding the option '-p tls.debug=3' on forward and paste here the output ?

tarokkk · 2017-09-22T08:42:21Z

My exact setup was this:

OpenSSL command:
openssl req -x509 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 365

Fluentd with version fluentd-0.12.32 with gem 'fluent-plugin-secure-forward' version '0.4.5'

The config

<source>
  type         secure_forward
  self_hostname myserver.local
  shared_key    xxxx
  secure yes
  ca_cert_path        /etc/fluent/certs/ca.crt
  ca_private_key_path /etc/fluent/certs/ca.key
  ca_private_key_passphrase
  allow_self_signed_certificate true
  port    24231
</source>

On the client side:

./fluent-bit  \                                                                                                                                                                                     *[master]
         -i random \
         -o forward://fluentd-forwarder01:24231 \
         -p tls=on -p tls.verify=off \
         -p tls.ca_file=./ca.crt \
         -p shared_key=xxxx -f 1 \
       -p tls.debug=4
Fluent-Bit v0.13.0
Copyright (C) Treasure Data

[2017/09/22 10:45:10] [ info] [engine] started
[2017/09/22 10:45:13] [error] [io_tls] flb_io_tls.c:287 X509 - The CRT/CRL/CSR format is invalid, e.g. different type expected
[2017/09/22 10:45:13] [error] [out_fw] no upstream connections available
[2017/09/22 10:45:14] [error] [io_tls] flb_io_tls.c:287 X509 - The CRT/CRL/CSR format is invalid, e.g. different type expected
[2017/09/22 10:45:14] [error] [out_fw] no upstream connections available```

tarokkk · 2017-09-26T13:08:48Z

Just a bit more information:

Unsecure fluent-bit source and Unsecure fluentd destination working
Secure fluentd source and secure fluentd destination working
The mixed secure fluent-bit and secure fluentd destination still not working

edsiper · 2017-09-26T14:41:20Z

@tarokkk

I was dealing with the same thing yesterday, my scenario is as follows (using Fluentd to isolate the issue):

Fluentd 0.14
Generated root CA cert, server cert, server key, client cert and client key according to these steps
Start one Fluentd 0.14 with out_secure forward using Client side certificates
Start another Fluentd 0.14 using in_secure_forward

is anybody able to make work that kind of local/self-signed certs ?

mpeltonen · 2017-10-06T11:17:31Z

We also faced this issue. Server side is Fluentd 0.14.21 with TLS in_forward, essentially the exact same <source> setup as in the original question. When trying to connect with Fluent Bit we get:

/usr/local/bin/td-agent-bit -f 1 -i random -o forward://10.0.0.1:24224 -p tls=on -p shared_key=fluentd -p tls.verify=off
Fluent-Bit v0.12.4
Copyright (C) Treasure Data

[2017/10/06 20:01:32] [ info] [engine] started
[2017/10/06 20:01:34] [error] [io_tls] flb_io_tls.c:300 X509 - The CRT/CRL/CSR format is invalid, e.g. different type expected
[2017/10/06 20:01:34] [error] [out_fw] no upstream connections available
[2017/10/06 20:01:35] [error] [io_tls] flb_io_tls.c:300 X509 - The CRT/CRL/CSR format is invalid, e.g. different type expected
[2017/10/06 20:01:35] [error] [out_fw] no upstream connections available

mpeltonen · 2017-10-09T05:46:56Z

@edsiper: I dug deeper and it seems that there's two problems in fluentd certificate creation that cause this:

Certificate created by fluentd has version 1 (i.e. not explicitly set in cert_option_generate_pair), which causes the CRT/CRL/CSR format is invalid error in mbedtls parsing.
Extensions are not correctly set and their format is invalid, which causes mbedtls extension parsing to fail.

Here's a sample patch to fluentd that seems to fix the problem for me:

diff --git a/lib/fluent/plugin_helper/cert_option.rb b/lib/fluent/plugin_helper/cert_option.rb
index 30e3cc36..3dc6f320 100644
--- a/lib/fluent/plugin_helper/cert_option.rb
+++ b/lib/fluent/plugin_helper/cert_option.rb
@@ -104,6 +104,7 @@ module Fluent
         cert.not_before = Time.at(0)
         cert.not_after = Time.now + opts[:expiration]
         cert.public_key = key
+        cert.version = 2
         cert.serial = 1
         cert.issuer = issuer
         cert.subject  = subject
@@ -127,9 +128,10 @@ module Fluent
         cert, key = cert_option_generate_pair(generate_opts, ca_cert.subject)
         raise "BUG: certificate digest algorithm not set" unless generate_opts[:digest]

+        ef = OpenSSL::X509::ExtensionFactory.new
         # basicConstraints: this cert is for CA or not
-        cert.add_extension OpenSSL::X509::Extension.new('basicConstraints', OpenSSL::ASN1.Sequence([OpenSSL::ASN1::Boolean(false)]))
-        cert.add_extension OpenSSL::X509::Extension.new('nsCertType', 'server')
+        cert.add_extension ef.create_extension('basicConstraints', 'CA:FALSE')
+        cert.add_extension ef.create_extension('nsCertType', 'server')

         cert.sign(ca_key, generate_opts[:digest].to_s)
         return cert, key, nil

edsiper · 2017-10-09T13:34:45Z

@mpeltonen thanks so much for helping to troubleshoot/fix this.

for now waiting for some feedback on fluent/fluentd#1711

nahi · 2017-10-09T22:25:30Z

I agree with @mpeltonen that this Certificate generation code is the cause.

cert.version should be 0x2 (X509v3) for Certificate extensions
basicConstraints must have 2 values: CA and pathlen. And you can use ExtensionFactory to avoid similar inconsistent data generation problem in the future.

In addition to it, my suggestions;

serial must be unique. Use random?
It also should have keyUsage and extendedKeyUsage.

cf. https://github.com/nahi/ruby-crypt/blob/master/CA.rb/gen_cert.rb#L61-L115

edsiper · 2017-10-09T22:40:10Z

@nahi hey! thanks for comments.

at this point we need to move the conversation to fluent/fluentd#1711

nahi · 2017-10-09T22:50:17Z

@edsiper Indeed. As @repeatedly asked me I posted here but it should be an issue of fluentd.

edsiper · 2017-10-13T19:11:32Z

Issue have been fixed on Fluentd side.

This adds a basic explanation on how to register & manipulate Fluent Bit as Windows Service. This should be useful for users who want to install Fluent Bit to their Windows servers. Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>

edsiper self-assigned this Aug 10, 2017

edsiper added question fixed labels Aug 10, 2017

edsiper removed the fixed label Sep 20, 2017

mpeltonen mentioned this issue Oct 9, 2017

TLS connection from fluent-bit to Fluentd 0.14.21 not working fluent/fluentd#1711

Closed

mpeltonen mentioned this issue Oct 11, 2017

Improve generated certificates' conformance to X.509 specification fluent/fluentd#1714

Merged

edsiper closed this as completed Oct 13, 2017

edsiper added the not-an-issue label Oct 13, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can fluent-bit talk with Fluentd v0.14 in_forward using TLS? #350

Can fluent-bit talk with Fluentd v0.14 in_forward using TLS? #350

okkez commented Aug 9, 2017

edsiper commented Aug 10, 2017

okkez commented Aug 10, 2017

edsiper commented Aug 10, 2017 •

edited

Loading

okkez commented Aug 14, 2017

okkez commented Aug 14, 2017

edsiper commented Aug 14, 2017

okkez commented Aug 14, 2017

tarokkk commented Sep 20, 2017

edsiper commented Sep 20, 2017

tarokkk commented Sep 22, 2017 •

edited

Loading

tarokkk commented Sep 26, 2017 •

edited

Loading

edsiper commented Sep 26, 2017

mpeltonen commented Oct 6, 2017

mpeltonen commented Oct 9, 2017

edsiper commented Oct 9, 2017

nahi commented Oct 9, 2017

edsiper commented Oct 9, 2017

nahi commented Oct 9, 2017

edsiper commented Oct 13, 2017

Can fluent-bit talk with Fluentd v0.14 in_forward using TLS? #350

Can fluent-bit talk with Fluentd v0.14 in_forward using TLS? #350

Comments

okkez commented Aug 9, 2017

edsiper commented Aug 10, 2017

okkez commented Aug 10, 2017

edsiper commented Aug 10, 2017 • edited Loading

okkez commented Aug 14, 2017

okkez commented Aug 14, 2017

edsiper commented Aug 14, 2017

okkez commented Aug 14, 2017

tarokkk commented Sep 20, 2017

edsiper commented Sep 20, 2017

tarokkk commented Sep 22, 2017 • edited Loading

tarokkk commented Sep 26, 2017 • edited Loading

edsiper commented Sep 26, 2017

mpeltonen commented Oct 6, 2017

mpeltonen commented Oct 9, 2017

edsiper commented Oct 9, 2017

nahi commented Oct 9, 2017

edsiper commented Oct 9, 2017

nahi commented Oct 9, 2017

edsiper commented Oct 13, 2017

edsiper commented Aug 10, 2017 •

edited

Loading

tarokkk commented Sep 22, 2017 •

edited

Loading

tarokkk commented Sep 26, 2017 •

edited

Loading