Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

relax size check #36

Merged
merged 3 commits into from
Oct 4, 2020
Merged

relax size check #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
326b127
fix compression
jasperzhong Jul 7, 2020
c5ac870
add docs
jasperzhong Jul 7, 2020
c819c68
Merge branch 'byteps' into patch
jasperzhong Oct 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions docs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,10 @@ The data communicated are presented as key-value
index and the value might be the according `float` gradient.
1. Basic synchronization functions: \ref ps::KVWorker::Push, \ref
ps::KVWorker::Pull, and \ref ps::KVWorker::Wait
2. Dynamic length value push and pull: \ref ps::KVWorker::VPush and \ref
ps::KVWorker::VPull
3. Zero-copy versions: \ref ps::KVWorker::ZPush, \ref
ps::KVWorker::ZPull, \ref ps::KVWorker::ZVPush and \ref
ps::KVWorker::ZVPull
2. Zero-copy versions: \ref ps::KVWorker::ZPush, \ref
ps::KVWorker::ZPull

To support dynamic length, pull operations(`Pull` and `ZPull`), do not require the buffer(`vals`) to be the same size as the total data size of pulling down. Larger buffer is allowed while `lens` records the actual size of each key. So the reliable way to read a valid message is to read `lens` bytes. If you ensure that the data size of a key does not change during push or pull, you can verify it by checking whether `lens` of the key is equal to the fixed size.

often server *i* handles the keys (feature indices) within the i-th
segment of <em>[0, uint64_max]</em>. The server node allows user-defined handles to
Expand Down
18 changes: 15 additions & 3 deletions include/ps/kv_app.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,12 @@ struct KVPairs {
* \brief A worker node that can \ref Push (\ref Pull) key-value pairs to (from) server
* nodes
*
* 1. Basic synchronization functions: \ref ps::KVWorker::Push,
* \ref ps::KVWorker::Pull, and \ref ps::KVWorker::Wait
*
* 2. Zero-copy versions: \ref ps::KVWorker::ZPush,
* \ref ps::KVWorker::ZPull *
*
* \tparam Val the type of value, which should be primitive types such as
* int32_t and float
*/
Expand Down Expand Up @@ -160,6 +166,12 @@ class KVWorker : public SimpleApp {
* namely \a vals (and \a lens) is filled with pulled values, only
* if \ref Wait returns or the callback is called.
*
* Note that \a vals can be larger than the length of the pulled values.
* \a lens is the actual length of the pulled values. The reliable way to
* read a valid message is to read \a lens bytes. If you ensure that the
* data size of a key does not change during push or pull, you can verify
* it by checking whether \a lens of the key is equal to the fixed size. *
*
* @param keys a list of keys, must be unique and sorted in increasing order
* @param vals the buffer for the pulled values. It can be 0 size.
* @param lens optional buffer for the value length. If set, it can be 0 size.
Expand Down Expand Up @@ -640,10 +652,10 @@ int KVWorker<Val>::Pull_(
if (vals->empty()) {
vals->resize(total_val);
} else {
CHECK_EQ(vals->size(), total_val);
CHECK_GE(vals->size(), total_val);
}

if (!is_worker_zpull_) { // otherwise do nothing
if (!is_worker_zpull_) { // otherwise do nothing
Val* p_vals = vals->data();
int *p_lens = nullptr;
if (lens) {
Expand All @@ -670,7 +682,7 @@ int KVWorker<Val>::Pull_(
if (cb) cb();
});

KVPairs<Val> kvs;
KVPairs<Val> kvs;
kvs.keys = keys;
kvs.vals = *vals;
Send(ts, false, cmd, kvs);
Expand Down