Fix lifecycle control

[tagomoris]

This change includes some changes and bug fixes about lifecycle of process and plugins.

• plugins should call super in start and shutdown
• plugins sometimes raise non-standard errors: these should be rescued and logged
• ServerEngine sends SIGTERM to stop worker process, but it wasn't rescued
  • shutdown was called in ensure clause, and it invokes shutdown sequence unexpectedly

This change also includes removing default_loop from Engine. It wasn't used in anywhere, so I simplified main loop by removing it.

[sonots]

I never rescue Exception (non StandardError) because we can do nothing for NoMemoryError, SystemStackError, SystemExit, and so on.

plugins sometimes raise non-standard errors: these should be rescued and logged

What did they raise? I guess they are doing something wrong.

[tagomoris]

@sonots I did it mainly for SignalException, which can be raised from anywhere. And I also thought about kind of ScriptError (we can raise it by eval at anywhere in ruby).

[sonots]

@tagomoris Then, I think we should rescue only SignalException and StandardError.

[tagomoris]

@sonots If plugin.shutdown occurs infinite recursion call in 3rd party plugins, then it raises SystemStackError, but it can be rescued and we can execute shutdown for another plugins.
Same with exit 0 in #shutdown of 3rd party plugins. Isn't it?

[sonots]

Do we rescue exit? If I write exit 0 on plugin (I never wite though), I expect that fluentd dies immediately without rescuing.

[sonots]

Also, it looks root_agent.rb#L166 does not propagate error, so it recsues and ignores SystemExit. Ignoring SystemExit is not intuitive for me.

[tagomoris]

Yes, that is what I will do. I think that we should not forgive plugins to exit.
When a plugin will do exit, other plugins may have memory buffer with events which will be lost with exit.

[sonots]

Someone may want to write exit just for developing purpose. Non standard, non-intuitive behavior brings troubles.

[sonots]

I am very negative about rescuing entire Exception. It is practical manner in ruby world. Please read articles like

• http://daniel.fone.net.nz/blog/2013/05/28/why-you-should-never-rescue-exception-in-ruby/
• https://robots.thoughtbot.com/rescue-standarderror-not-exception
• http://stackoverflow.com/questions/10048173/why-is-it-bad-style-to-rescue-exception-e-in-ruby
• http://kwfsws.g.hatena.ne.jp/kiwofusi/20111231/1325314356 this is wrong

[repeatedly]

plugins should be sandbox so calling exit should not affect other plugin behaviour.
If we support exit like feature, it should be provided via plugin helper like mechanizm.

[repeatedly]

If we go with rescue Exception, we should write a comment because other developers can't understand why catch all exceptions instead of standard recoverable errors.

[tagomoris]

Generally speaking, I say "Don't rescue Exception" if this is a Rails application.
But this is middleware, and making data safe is most important.

How many non-standard exceptions If we don't rescue Exception? Non-standard exceptions are:

• NoMemoryError
• ScriptError (LoadError, NotImplementedError, SyntaxError)
• SignalException
• SystemExit
• SystemStackError
• fatal

#shutdown of plugins will do:

• infinite recursive call (SystemStackError)
• unexpected exit (SystemExit)
• unexpected deletion of signal handler (SignalException)
• evaluate script with syntax error or something else (ScriptError)
• huge object generation in #write at shutdown (NoMemoryError)

[tagomoris]

@repeatedly okay, will do.

[sonots]

I think that we should not forgive plugins to exit.

I agree this. So, I think we do not need to take care of SystemExit. Rescuing SystemExit rather looks as forgiving plugins to call exit as a API design.

[tagomoris]

I meant that "we should not forgive plugins to exit entire Fluentd process", not about just calling "exit" method.

[sonots]

Okay, you meant that API supports (takes cares safely) plugins call exit method. If it is the design, rescuing SystemExit is okay. But, I also think we need to document as repeatedly says.

[tagomoris]

I added a comment.

[tagomoris]

Merged.