filter deletion in compaction filter #344
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: tabokie <xy.tao@outlook.com>
tabokie
force-pushed
the
fix-compact-range
branch
from
5632fe8
to
66dcaa3
Compare
Signed-off-by: tabokie <xy.tao@outlook.com>
tonyxuqqi
approved these changes
Aug 4, 2023
overvenus
reviewed
Aug 4, 2023
| @@ -195,7 +195,8 @@ class WritableFileWriter { | |||
| TEST_SYNC_POINT_CALLBACK("WritableFileWriter::WritableFileWriter:0", | |||
| reinterpret_cast<void*>(max_buffer_size_)); | |||
| buf_.Alignment(writable_file_->GetRequiredBufferAlignment()); | |||
| buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_)); | |||
| // Removed to reduce WAL writer memory usage. | |||
| // buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_)); | |||
There was a problem hiding this comment.
Maybe we can keep the code and use a smaller max_buffer_size in raftstore v2.
There was a problem hiding this comment.
The allocation is delayed to actual write, which I think is good for both v1 and v2. Because the initialization is also on critical path, moving it elsewhere amortizes the overhead.
There was a problem hiding this comment.
I see. How about skipping alloc is max_buffer_size_ == 0?
Suggested change
| // buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_)); | |
| if max_buffer_size_ != 0 { | |
| buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_)); | |
| } |
There was a problem hiding this comment.
Oh, I forgot this flag works for both WAL and SST. So we can't simply change the config.
overvenus
reviewed
Aug 7, 2023
| @@ -195,7 +195,8 @@ class WritableFileWriter { | |||
| TEST_SYNC_POINT_CALLBACK("WritableFileWriter::WritableFileWriter:0", | |||
| reinterpret_cast<void*>(max_buffer_size_)); | |||
| buf_.Alignment(writable_file_->GetRequiredBufferAlignment()); | |||
| buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_)); | |||
| // Removed to reduce WAL writer memory usage. | |||
| // buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_)); | |||
There was a problem hiding this comment.
I see. How about skipping alloc is max_buffer_size_ == 0?
Suggested change
| // buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_)); | |
| if max_buffer_size_ != 0 { | |
| buf_.AllocateNewBuffer(std::min((size_t)65536, max_buffer_size_)); | |
| } |
| @@ -212,6 +217,21 @@ class CompactionFilter : public Customizable { | |||
| skip_until); | |||
| } | |||
|
|
|||
| // This interface is reserved for TiKV's region range filter. Only this | |||
| // interface can accept `value_type=kTypeDeletion`. | |||
| virtual Decision UnsafeFilter(int level, const Slice& key, | |||
There was a problem hiding this comment.
Is is possible to add a test case to show its usage?
Signed-off-by: tabokie <xy.tao@outlook.com>
overvenus
approved these changes
Aug 7, 2023
v01dstar
pushed a commit
to v01dstar/rocksdb
that referenced
this pull request
Feb 21, 2024
And delay the buffer initialization of writable file to first actual write. --------- Signed-off-by: tabokie <xy.tao@outlook.com>
v01dstar
pushed a commit
to v01dstar/rocksdb
that referenced
this pull request
Feb 22, 2024
compaction_filter: add bottommost_level into context (tikv#160) Signed-off-by: qupeng <qupeng@pingcap.com> Signed-off-by: tabokie <xy.tao@outlook.com> add range for compaction filter context (tikv#192) * add range for compaction filter context Signed-off-by: qupeng <qupeng@pingcap.com> Signed-off-by: tabokie <xy.tao@outlook.com> allow no_io for VersionSet::GetTableProperties (tikv#211) * allow no_io for VersionSet::GetTableProperties Signed-off-by: qupeng <qupeng@pingcap.com> Signed-off-by: tabokie <xy.tao@outlook.com> expose seqno from compaction filter and iterator (tikv#215) This PR supports to access `seqno` for every key/value pairs in compaction filter or iterator. It's helpful to enhance GC in compaction filter in TiKV. Signed-off-by: qupeng <qupeng@pingcap.com> Signed-off-by: tabokie <xy.tao@outlook.com> allow to query DB stall status (tikv#226) This PR adds a new property is-write-stalled to query whether the column family is in write stall or not. In TiKV there is a compaction filter used for GC, in which DB::write is called. So if we can query whether the DB instance is stalled or not, we can skip to create more compaction filter instances to save some resources. Signed-off-by: qupeng <qupeng@pingcap.com> Signed-off-by: tabokie <xy.tao@outlook.com> Fix compatibilty issue with Titan Signed-off-by: v01dstar <yang.zhang@pingcap.com> filter deletion in compaction filter (tikv#344) And delay the buffer initialization of writable file to first actual write. --------- Signed-off-by: tabokie <xy.tao@outlook.com> Adjustments for compaptibilty with 8.10.facebook Signed-off-by: v01dstar <yang.zhang@pingcap.com> Adjust tikv related changes with upstream Signed-off-by: v01dstar <yang.zhang@pingcap.com>
v01dstar
pushed a commit
to v01dstar/rocksdb
that referenced
this pull request
May 29, 2024
And delay the buffer initialization of writable file to first actual write. --------- Signed-off-by: tabokie <xy.tao@outlook.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
And delay the buffer initialization of writable file to first actual write.