Issue 1726: Fix for in_tcp log corruption under load. #1729
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The TCP input plugin shares a single buffer across all connections. Under load the buffer sometimes gets truncated. The suspicion is that concurrent connections race to modify it. Long message is received. Short message is received while the long message is still being parsed. The short message is parsed so the buffer will be truncated. The buffer, which now contains "longmessage\nshort\n", is truncated by the length of the short message. It is now "longm". Another message arrives. Now the buffer is "longmanother\n", which does not parse. The whole buffer is thrown away and subsequent messages are received and handled as usual. Eventually the pattern repeats. The fix is to use a per-connection buffer.
|
Reporter confirmed this patch fixed the issue. Thanks! |
|
It would be a great help if this could be tagged as |
|
I am a colleague of @AM-iain. We've tested this on our infrastructure successfully. |
mururu
pushed a commit
to mururu/fluentd
that referenced
this pull request
Dec 28, 2017
fluent#1729 introduced @buffer to TCPCallbackSocket. TLSCallbackSocket has to have it to be used for in_tcp.
mururu
pushed a commit
to mururu/fluentd
that referenced
this pull request
Dec 28, 2017
fluent#1729 introduced @buffer to TCPCallbackSocket. TLSCallbackSocket has to have it to be used for in_tcp.
mururu
pushed a commit
to mururu/fluentd
that referenced
this pull request
Dec 28, 2017
fluent#1729 introduced @buffer to TCPCallbackSocket. TLSCallbackSocket also has to have it to be used for in_tcp.
mururu
pushed a commit
to mururu/fluentd
that referenced
this pull request
Dec 28, 2017
fluent#1729 introduced @buffer to TCPCallbackSocket. TLSCallbackSocket also has to have it to be used for in_tcp.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The TCP input plugin shares a single buffer across all connections. Under load
the buffer sometimes gets truncated. The suspicion is that concurrent
connections race to modify it.
Long message is received.
Short message is received while the long message is still being parsed.
The short message is parsed so the buffer will be truncated.
The buffer, which now contains "longmessage\nshort\n", is truncated by the
length of the short message. It is now "longm".
Another message arrives. Now the buffer is "longmanother\n", which does not
parse.
The whole buffer is thrown away and subsequent messages are received and
handled as usual.
Eventually the pattern repeats.
The fix is to use a per-connection buffer.