Use pipe command sender mechanism of ServerEngine on Windows

[unak]

The pipe command sender mechanism has been merged into ServerEngine (treasure-data/serverengine#50).
This pull request makes fluentd to use it on Windows, to fix #1084.

[tagomoris]

I'm checking this patch on Windows environment too... will paste all logs.

[tagomoris]

I ran fluentd on cmd.exe, and Ctrl-c to stop it. (success: confirmed that both two processes exited)

C:\>fluentd -c C:\fluent_test.conf
2016-08-30 03:04:39 +0000 [info]: reading config file path="C:\\fluent_test.conf"
2016-08-30 03:04:39 +0000 [info]: starting fluentd-0.14.2
2016-08-30 03:04:39 +0000 [info]: spawn command to main: C:/Ruby22/bin/ruby.exe -Eascii-8bit:ascii-8bit  "C:/Ruby22/bin/fluentd" "-c" "C:\fluent_test.conf" --under-supervisor
2016-08-30 03:04:40 +0000 [info]: reading config file path="C:\\fluent_test.conf"
2016-08-30 03:04:40 +0000 [info]: starting fluentd-0.14.2 without supervision
2016-08-30 03:04:40 +0000 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-08-30 03:04:40 +0000 [info]: gem 'fluentd' version '0.14.2'
2016-08-30 03:04:40 +0000 [info]: gem 'fluentd' version '0.14.1'
2016-08-30 03:04:40 +0000 [info]: gem 'fluentd' version '0.14.0.pre.1'
2016-08-30 03:04:40 +0000 [info]: adding match in @dump pattern="**" type="file"
2016-08-30 03:04:40 +0000 [info]: adding source type="forward"
2016-08-30 03:04:41 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type forward
    @label @dump
  </source>
  <label @dump>
    <match **>
      @type file
      path "C:/mydump.*.log"
      time_slice_format "%Y%m%d%H%M%S"
      time_slice_wait 1s
      buffer_path C:/mydump.*.log
      <buffer time>
        flush_mode lazy
        retry_type exponential_backoff
        path C:/mydump.*.log
        timekey_wait 1s
        timekey 1
      </buffer>
    </match>
  </label>
  <system>
    rpc_endpoint "127.0.0.1:24444"
  </system>
</ROOT>
2016-08-30 03:04:41 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2016-08-30 03:04:43 +0000 [info]: Received graceful stop
2016-08-30 03:04:43 +0000 [info]: Worker 0 finished with
Terminate batch job (Y/N)? 2016-08-30 03:04:43 +0000 [info]: shutting down fluentd
2016-08-30 03:04:43 +0000 [info]: preparing shutdown input plugin type=:forward plugin_id="object:13ec4bc"
2016-08-30 03:04:43 +0000 [info]: preparing shutdown output plugin type=:file plugin_id="object:171036c"
2016-08-30 03:04:43 +0000 [info]: shutting down input plugin type=:forward plugin_id="object:13ec4bc"
2016-08-30 03:04:43 +0000 [info]: shutting down output plugin type=:file plugin_id="object:171036c"
2016-08-30 03:04:43 +0000 [info]: closing input plugin type=:forward plugin_id="object:13ec4bc"
2016-08-30 03:04:43 +0000 [info]: closing output plugin type=:file plugin_id="object:171036c"
Y

C:\>

[tagomoris]

Windows service (success: 3 ruby process executed, and stopped correctly).

I registered Fluentd as a Windows service:

fluentd --reg-winsvc i
fluentd --reg-winsvc-fluentdopt '-c C:/fluent_test.conf -o C:/fluent.log'

Then, I started & stopped "Fluentd Windows Service". Logs:

C:\>more fluent.log
2016-08-30 03:15:36 +0000 [info]: reading config file path="C:/fluent_test.conf"
2016-08-30 03:15:36 +0000 [info]: starting fluentd-0.14.2
2016-08-30 03:15:36 +0000 [info]: spawn command to main: C:/Ruby22/bin/ruby.exe
-Eascii-8bit:ascii-8bit  "C:/Ruby22/bin/fluentd" "-c" "C:/fluent_test.conf" "-o"
 "C:/fluent.log" "-x" "fluentdwinsvc" --under-supervisor
2016-08-30 03:15:37 +0000 [info]: reading config file path="C:/fluent_test.conf"
2016-08-30 03:15:37 +0000 [info]: starting fluentd-0.14.2 without supervision
2016-08-30 03:15:37 +0000 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-08-30 03:15:37 +0000 [info]: gem 'fluentd' version '0.14.2'
2016-08-30 03:15:37 +0000 [info]: gem 'fluentd' version '0.14.1'
2016-08-30 03:15:37 +0000 [info]: gem 'fluentd' version '0.14.0.pre.1'
2016-08-30 03:15:37 +0000 [info]: adding match in @dump pattern="**" type="file"
2016-08-30 03:15:37 +0000 [info]: adding source type="forward"
2016-08-30 03:15:38 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type forward
    @label @dump
  </source>
  <label @dump>
    <match **>
      @type file
      path "C:/mydump.*.log"
      time_slice_format "%Y%m%d%H%M%S"
      time_slice_wait 1s
      buffer_path C:/mydump.*.log
      <buffer time>
        flush_mode lazy
        retry_type exponential_backoff
        path C:/mydump.*.log
        timekey_wait 1s
        timekey 1
      </buffer>
    </match>
  </label>
  <system>
    rpc_endpoint "127.0.0.1:24444"
  </system>
</ROOT>
2016-08-30 03:15:38 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2016-08-30 03:16:37 +0000 [info]: Received graceful stop
2016-08-30 03:16:37 +0000 [info]: Worker 0 finished with status 0

[tagomoris]

@unak Just one thing I want to confirm:
When I launched Fluentd as a service, I couldn't see any logs on fluent.log file (its size was 0).
I could see whole logs after I stopped Fluentd service.

Can we fix this behavior to see logs as soon as possible? Or not?

[tagomoris]

I retried the configuration, and confirmed that:

• the size of log file is 0 bytes
• but we can see logs actually (by type fluent.log or tail -f)

Note: fsync may help this case if needed https://github.com/ruby/ruby/blob/v2_2_0/NEWS#L110-L113

There are no problem to merge this change!

[tagomoris]

I'll merge this branch besides CI red, because it is not related with this patch, and already known at #1115.