Add buf_file_single plugin #2579
Conversation
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
repeatedly
added
the
feature request
|
|
||
| @p.start | ||
| assert File.exist?(@bufdir) | ||
| assert { File.stat(@bufdir).mode.to_s(8).end_with?('755') } |
There was a problem hiding this comment.
assert_match? or assert_equal is better?
There was a problem hiding this comment.
assert_equal is not fit because result is 40755 like string. This equal seems good for test.
| end | ||
| end | ||
|
|
||
| def restore_metadata |
There was a problem hiding this comment.
Doesn't file_single_chunk support file_chunk's metadata restore?(this is just confirmation)
There was a problem hiding this comment.
We can support it with metadata header in the future.
Just now, support only one field key or tag.
| assert_equal (d1.to_json + "\n" + d2.to_json + "\n"), File.read(@c.path) | ||
| end | ||
|
|
||
| test 'can #rollback to revert non-committed data from #concat' do |
There was a problem hiding this comment.
I think it's better to add a test case for 'can not #rollback a committed data'
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
test/plugin/test_buf_file_single.rb
Outdated
| @d = create_driver | ||
| p = @d.instance.buffer | ||
| assert_equal File.join(@bufdir, 'fsb.*.buf'), p.path | ||
| end | ||
|
|
||
| data('text based chunk' => [FluentPluginFileSingleBufferTest::DummyOutputPlugin, :text], | ||
| 'msgpack based chunk' => [FluentPluginFileSingleBufferTest::DummyOutputMPPlugin, :msgpack]) |
lib/fluent/plugin/buf_file_single.rb
Outdated
| desc 'The permission of chunk file. If no specified, <system> setting or 0644 is used' | ||
| config_param :file_permission, :string, default: nil # '0644' | ||
| desc 'The permission of chunk directory. If no specified, <system> setting or 0644 is used' | ||
| config_param :dir_permission, :string, default: nil # '0755' |
There was a problem hiding this comment.
L48 says 0644 but this line says 0755. which is one correct?
There was a problem hiding this comment.
ah, forgot it. Will fix.
|
|
||
| @state = :queued | ||
| @bytesize = @chunk.size | ||
| @commit_position = @chunk.size |
There was a problem hiding this comment.
Are there any reasons not to use @chunk.pos?
| return nil unless res | ||
|
|
||
| key = base[4..res - 1] # remove 'fsb.' and '.' | ||
| hex_id = base[res + 2..-5] # remove '.' and '.buf' |
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
Which issue(s) this PR fixes:
Ref #2153
What this PR does / why we need it:
buf_fileworks fine for typical cases, e.g.in_tail/in_forward, but it is not good for lots of small emit cases. It issues lots of IO operation.So to reduce IO tps on high traffic environment, introduce
buf_file_single. This is similar to v0.12 buf_file and this is lightweight.buf_file_singlehas one limitation that supports only tag or one field chunk key,<buffer tag>or<buffer key>. This limitation will be relaxed by supporting metadata header in the file.Docs Changes:
Add
buf_file_singlearticle. Will send a patch later.Release Note:
Same as title