dump all keys; count size of all/subset of keys based on prefix #150

thinkingfish · 2017-04-01T08:25:05Z

I got a request to find out how many keys there are in cache that belong to a particular dataset (represented by a specific prefix). I thought about it for a bit and realized this feature might be generally useful to perform a total key dump for analysis or get an dataset level overview of the cache, so I checked this code it.

Not sure if this is ready for merge at this point: big issue is thread-safety. Since we perform the scanning on the admin thread, to avoid reading dirty keys (keys that are being updated by the worker thread) requires some kind of locking/synchronization. This we can do rather cheaply by using an atomic instruction to let worker thread alter a flag when an hashbucket is being actively modified, and having the admin thread blocked if it runs into a dirty hashtable entry. But the one-off usage I'm asked to provide can be done on a staging host after live traffic is shut off, so it doesn't seem to warrant the extra rigor for now.

I just want to have a place where this feature is visible/checked in.

seppo0010 · 2017-04-01T14:34:42Z

re: race condition, could we do what Redis does for saving (a fork and copy-on-write)?

seppo0010

add tests

seppo0010 · 2017-04-01T14:43:59Z

src/storage/slab/item.c

+}
+
+bool
+item_dump(void)


why not receive a prefix here as well?

727905b adding a new API to duration timer (twitter#152) 9406717 add missing "extern C" qualifiers to a few headers for C++ linking; (twitter#150) a0aafdf Add missing semicolon to ring array example code in documentation. (twitter#149) 9264bbb Zero byte (twitter#147) (emergency fix needed for pelikan) d4002d7 simplify cc_print_int64 (twitter#146) b164fcf Clean-up hash functions and introduce MurmurHash3 (twitter#145) b1babb2 change wheel's sleep timer to make it less flaky (twitter#143) ce0b9ea allow printing negative integers in cc_print (twitter#141) ab0edc8 add metrics to track buf_sock objects (twitter#138) ae02038 add travis ci (copied from pelikan) (twitter#139) 964645a Merge pull request twitter#135 from paegun/fix_cmake_install 70710c2 fixed and re-added cmake install instructions, w/ following notes: include directory made proper relative; opened pattern match b/c include directory should only contain files meant for inclusion. 5b095bc Merge pull request twitter#126 from kevyang/kevyang/120 426d56a return NULL when cc_alloc/cc_realloc is called with size == 0 ad271d4 Merge pull request twitter#133 from kevyang/132 47dbdba suppress unused parameter warning in debug_log_flush 648d19e Merge pull request twitter#127 from kevyang/56 780941a Merge pull request twitter#130 from kevyang/129 b8af6c0 Merge pull request twitter#131 from kevyang/128 6ecc318 fix duplicate symbols in cc_signal 080c41d cc_array - stop doing arithmetic on void * d526f7a add debug oriented memory management a4fb927 Update bool member rules in style guide 05c6e1e explicitly make ccommon a C project to avoid checking for CXX related variables git-subtree-dir: deps/ccommon git-subtree-split: 727905b

727905b adding a new API to duration timer (#152) 9406717 add missing "extern C" qualifiers to a few headers for C++ linking; (#150) a0aafdf Add missing semicolon to ring array example code in documentation. (#149) 9264bbb Zero byte (#147) (emergency fix needed for pelikan) d4002d7 simplify cc_print_int64 (#146) b164fcf Clean-up hash functions and introduce MurmurHash3 (#145) b1babb2 change wheel's sleep timer to make it less flaky (#143) ce0b9ea allow printing negative integers in cc_print (#141) ab0edc8 add metrics to track buf_sock objects (#138) ae02038 add travis ci (copied from pelikan) (#139) 964645a Merge pull request #135 from paegun/fix_cmake_install 70710c2 fixed and re-added cmake install instructions, w/ following notes: include directory made proper relative; opened pattern match b/c include directory should only contain files meant for inclusion. 5b095bc Merge pull request #126 from kevyang/kevyang/120 426d56a return NULL when cc_alloc/cc_realloc is called with size == 0 ad271d4 Merge pull request #133 from kevyang/132 47dbdba suppress unused parameter warning in debug_log_flush 648d19e Merge pull request #127 from kevyang/56 780941a Merge pull request #130 from kevyang/129 b8af6c0 Merge pull request #131 from kevyang/128 6ecc318 fix duplicate symbols in cc_signal 080c41d cc_array - stop doing arithmetic on void * d526f7a add debug oriented memory management a4fb927 Update bool member rules in style guide 05c6e1e explicitly make ccommon a C project to avoid checking for CXX related variables git-subtree-dir: deps/ccommon git-subtree-split: 727905b

dd56abd break up ASSERT in buf_sock_read (twitter#153) 727905b adding a new API to duration timer (twitter#152) 9406717 add missing "extern C" qualifiers to a few headers for C++ linking; (twitter#150) a0aafdf Add missing semicolon to ring array example code in documentation. (twitter#149) 9264bbb Zero byte (twitter#147) (emergency fix needed for pelikan) d4002d7 simplify cc_print_int64 (twitter#146) b164fcf Clean-up hash functions and introduce MurmurHash3 (twitter#145) b1babb2 change wheel's sleep timer to make it less flaky (twitter#143) ce0b9ea allow printing negative integers in cc_print (twitter#141) ab0edc8 add metrics to track buf_sock objects (twitter#138) ae02038 add travis ci (copied from pelikan) (twitter#139) 964645a Merge pull request twitter#135 from paegun/fix_cmake_install 70710c2 fixed and re-added cmake install instructions, w/ following notes: include directory made proper relative; opened pattern match b/c include directory should only contain files meant for inclusion. 5b095bc Merge pull request twitter#126 from kevyang/kevyang/120 426d56a return NULL when cc_alloc/cc_realloc is called with size == 0 ad271d4 Merge pull request twitter#133 from kevyang/132 47dbdba suppress unused parameter warning in debug_log_flush 648d19e Merge pull request twitter#127 from kevyang/56 780941a Merge pull request twitter#130 from kevyang/129 b8af6c0 Merge pull request twitter#131 from kevyang/128 6ecc318 fix duplicate symbols in cc_signal 080c41d cc_array - stop doing arithmetic on void * d526f7a add debug oriented memory management a4fb927 Update bool member rules in style guide 05c6e1e explicitly make ccommon a C project to avoid checking for CXX related variables git-subtree-dir: deps/ccommon git-subtree-split: dd56abd

* consolidate code a bit * Squashed 'deps/ccommon/' changes from bb298bc..dd56abd dd56abd break up ASSERT in buf_sock_read (#153) 727905b adding a new API to duration timer (#152) 9406717 add missing "extern C" qualifiers to a few headers for C++ linking; (#150) a0aafdf Add missing semicolon to ring array example code in documentation. (#149) 9264bbb Zero byte (#147) (emergency fix needed for pelikan) d4002d7 simplify cc_print_int64 (#146) b164fcf Clean-up hash functions and introduce MurmurHash3 (#145) b1babb2 change wheel's sleep timer to make it less flaky (#143) ce0b9ea allow printing negative integers in cc_print (#141) ab0edc8 add metrics to track buf_sock objects (#138) ae02038 add travis ci (copied from pelikan) (#139) 964645a Merge pull request #135 from paegun/fix_cmake_install 70710c2 fixed and re-added cmake install instructions, w/ following notes: include directory made proper relative; opened pattern match b/c include directory should only contain files meant for inclusion. 5b095bc Merge pull request #126 from kevyang/kevyang/120 426d56a return NULL when cc_alloc/cc_realloc is called with size == 0 ad271d4 Merge pull request #133 from kevyang/132 47dbdba suppress unused parameter warning in debug_log_flush 648d19e Merge pull request #127 from kevyang/56 780941a Merge pull request #130 from kevyang/129 b8af6c0 Merge pull request #131 from kevyang/128 6ecc318 fix duplicate symbols in cc_signal 080c41d cc_array - stop doing arithmetic on void * d526f7a add debug oriented memory management a4fb927 Update bool member rules in style guide 05c6e1e explicitly make ccommon a C project to avoid checking for CXX related variables git-subtree-dir: deps/ccommon git-subtree-split: dd56abd

a2d5c01 add a function that will reject all pending incoming connections (twitter#154) dd56abd break up ASSERT in buf_sock_read (twitter#153) 727905b adding a new API to duration timer (twitter#152) 9406717 add missing "extern C" qualifiers to a few headers for C++ linking; (twitter#150) a0aafdf Add missing semicolon to ring array example code in documentation. (twitter#149) 9264bbb Zero byte (twitter#147) (emergency fix needed for pelikan) d4002d7 simplify cc_print_int64 (twitter#146) b164fcf Clean-up hash functions and introduce MurmurHash3 (twitter#145) b1babb2 change wheel's sleep timer to make it less flaky (twitter#143) ce0b9ea allow printing negative integers in cc_print (twitter#141) ab0edc8 add metrics to track buf_sock objects (twitter#138) ae02038 add travis ci (copied from pelikan) (twitter#139) 964645a Merge pull request twitter#135 from paegun/fix_cmake_install 70710c2 fixed and re-added cmake install instructions, w/ following notes: include directory made proper relative; opened pattern match b/c include directory should only contain files meant for inclusion. 5b095bc Merge pull request twitter#126 from kevyang/kevyang/120 426d56a return NULL when cc_alloc/cc_realloc is called with size == 0 ad271d4 Merge pull request twitter#133 from kevyang/132 47dbdba suppress unused parameter warning in debug_log_flush 648d19e Merge pull request twitter#127 from kevyang/56 780941a Merge pull request twitter#130 from kevyang/129 b8af6c0 Merge pull request twitter#131 from kevyang/128 6ecc318 fix duplicate symbols in cc_signal 080c41d cc_array - stop doing arithmetic on void * d526f7a add debug oriented memory management a4fb927 Update bool member rules in style guide 05c6e1e explicitly make ccommon a C project to avoid checking for CXX related variables git-subtree-dir: deps/ccommon git-subtree-split: a2d5c01

6b59401 Rust build tweak (twitter#165) b39086d fix docs to new ring_array_destroy semantics (twitter#163) 95e04b0 Additional ring_array API (twitter#159) 3c994cc a single job in the travis-ci matrix for building with rust enabled (twitter#161) 1fe907e rust support - Native bstrings (twitter#160) b3de2ee add HAVE_RUST option defaulted to off (twitter#158) ffc70a8 Port travis ci changes - closes twitter#186 (twitter#155) a2d5c01 add a function that will reject all pending incoming connections (twitter#154) dd56abd break up ASSERT in buf_sock_read (twitter#153) 727905b adding a new API to duration timer (twitter#152) 9406717 add missing "extern C" qualifiers to a few headers for C++ linking; (twitter#150) a0aafdf Add missing semicolon to ring array example code in documentation. (twitter#149) 9264bbb Zero byte (twitter#147) (emergency fix needed for pelikan) d4002d7 simplify cc_print_int64 (twitter#146) b164fcf Clean-up hash functions and introduce MurmurHash3 (twitter#145) b1babb2 change wheel's sleep timer to make it less flaky (twitter#143) ce0b9ea allow printing negative integers in cc_print (twitter#141) ab0edc8 add metrics to track buf_sock objects (twitter#138) ae02038 add travis ci (copied from pelikan) (twitter#139) 964645a Merge pull request twitter#135 from paegun/fix_cmake_install 70710c2 fixed and re-added cmake install instructions, w/ following notes: include directory made proper relative; opened pattern match b/c include directory should only contain files meant for inclusion. 5b095bc Merge pull request twitter#126 from kevyang/kevyang/120 426d56a return NULL when cc_alloc/cc_realloc is called with size == 0 ad271d4 Merge pull request twitter#133 from kevyang/132 47dbdba suppress unused parameter warning in debug_log_flush 648d19e Merge pull request twitter#127 from kevyang/56 780941a Merge pull request twitter#130 from kevyang/129 b8af6c0 Merge pull request twitter#131 from kevyang/128 6ecc318 fix duplicate symbols in cc_signal 080c41d cc_array - stop doing arithmetic on void * d526f7a add debug oriented memory management a4fb927 Update bool member rules in style guide 05c6e1e explicitly make ccommon a C project to avoid checking for CXX related variables git-subtree-dir: deps/ccommon git-subtree-split: 6b59401

f5efe29 forgot to remove unused include (#167) 441934b update ring_array docs for new API, add multi threaded unit test for ring_array (#166) 67ce9c2 Adding issue and pull request templates to comply with twitter OSS policies (#151) 6b59401 Rust build tweak (#165) b39086d fix docs to new ring_array_destroy semantics (#163) 95e04b0 Additional ring_array API (#159) 3c994cc a single job in the travis-ci matrix for building with rust enabled (#161) 1fe907e rust support - Native bstrings (#160) b3de2ee add HAVE_RUST option defaulted to off (#158) ffc70a8 Port travis ci changes - closes #186 (#155) a2d5c01 add a function that will reject all pending incoming connections (#154) dd56abd break up ASSERT in buf_sock_read (#153) 727905b adding a new API to duration timer (#152) 9406717 add missing "extern C" qualifiers to a few headers for C++ linking; (#150) a0aafdf Add missing semicolon to ring array example code in documentation. (#149) 9264bbb Zero byte (#147) (emergency fix needed for pelikan) d4002d7 simplify cc_print_int64 (#146) b164fcf Clean-up hash functions and introduce MurmurHash3 (#145) b1babb2 change wheel's sleep timer to make it less flaky (#143) ce0b9ea allow printing negative integers in cc_print (#141) ab0edc8 add metrics to track buf_sock objects (#138) ae02038 add travis ci (copied from pelikan) (#139) 964645a Merge pull request #135 from paegun/fix_cmake_install 70710c2 fixed and re-added cmake install instructions, w/ following notes: include directory made proper relative; opened pattern match b/c include directory should only contain files meant for inclusion. 5b095bc Merge pull request #126 from kevyang/kevyang/120 426d56a return NULL when cc_alloc/cc_realloc is called with size == 0 ad271d4 Merge pull request #133 from kevyang/132 47dbdba suppress unused parameter warning in debug_log_flush 648d19e Merge pull request #127 from kevyang/56 780941a Merge pull request #130 from kevyang/129 b8af6c0 Merge pull request #131 from kevyang/128 6ecc318 fix duplicate symbols in cc_signal 080c41d cc_array - stop doing arithmetic on void * d526f7a add debug oriented memory management a4fb927 Update bool member rules in style guide 05c6e1e explicitly make ccommon a C project to avoid checking for CXX related variables git-subtree-dir: deps/ccommon git-subtree-split: f5efe29

kevyang · 2018-09-15T06:54:15Z

src/storage/slab/item.c

+        struct item *it;
+
+        SLIST_FOREACH(it, entry, i_sle) {
+            if (write(fd, item_key(it), it->klen) < it->klen) {


would we also want to check for expiry?

kevyang

not really sure how this can work without interfering with the worker thread; i am probably missing something. can you offer some insights here? possibly comment the explanation somewhere?

edit: nvm, just read your comment above.

…cache

…izes in addition to counts & total sizes

…witter#150) fix a few typos/style deviations

CLAassistant · 2019-11-17T00:03:40Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Yao Yue seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

seppo0010 reviewed Apr 1, 2017

View reviewed changes

This comment was marked as spam.

Sign in to view

kevyang reviewed Sep 15, 2018

View reviewed changes

Yao Yue added 6 commits July 18, 2019 13:50

add key dumping and counting logic; enable commands on admin for twem…

af47c2c

…cache

let dump command take a prefix as well

b55fd09

rename admin command count to census, report min/max on key & value s…

6386e3c

…izes in addition to counts & total sizes

seperating debugging commands so we can ban them on the admin thread

116bcf7

add debug thread

9fdc42b

temp

79f7fb0

thinkingfish force-pushed the dump_keys branch from 6532ecf to 79f7fb0 Compare July 18, 2019 20:51

swlynch99 pushed a commit to swlynch99/pelikan-twitter that referenced this pull request Sep 30, 2019

add missing "extern C" qualifiers to a few headers for C++ linking; (t…

9406717

…witter#150) fix a few typos/style deviations

Merge branch 'master' into dump_keys

409d712

thinkingfish added the enhancement label Nov 11, 2020

theqazi force-pushed the master branch from e252250 to e7a76c9 Compare January 18, 2023 03:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dump all keys; count size of all/subset of keys based on prefix #150

dump all keys; count size of all/subset of keys based on prefix #150

thinkingfish commented Apr 1, 2017

seppo0010 commented Apr 1, 2017

seppo0010 left a comment

seppo0010 Apr 1, 2017

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

kevyang Sep 15, 2018

This comment was marked as spam.

kevyang left a comment •

edited

Loading

CLAassistant commented Nov 17, 2019 •

edited

Loading

dump all keys; count size of all/subset of keys based on prefix #150

Are you sure you want to change the base?

dump all keys; count size of all/subset of keys based on prefix #150

Conversation

thinkingfish commented Apr 1, 2017

seppo0010 commented Apr 1, 2017

seppo0010 left a comment

Choose a reason for hiding this comment

seppo0010 Apr 1, 2017

Choose a reason for hiding this comment

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

kevyang Sep 15, 2018

Choose a reason for hiding this comment

This comment was marked as spam.

kevyang left a comment • edited Loading

Choose a reason for hiding this comment

CLAassistant commented Nov 17, 2019 • edited Loading

kevyang left a comment •

edited

Loading

CLAassistant commented Nov 17, 2019 •

edited

Loading