UCP/API/TEST: Add non-blocking endpoint flush and use it for RMA tests #1912

yosefe · 2017-10-13T21:26:55Z

mellanox-github · 2017-10-13T22:52:46Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/4815/ for details (Mellanox internal link).

yosefe · 2017-10-13T23:06:59Z

00:39:25 [ RUN      ] dcx/test_ucp_perf.envelope/0
00:39:26 [     INFO ]                tag latency : 1.536 usec
00:39:27 [     INFO ]            tag iov latency : 4.529 usec
00:39:28 [     INFO ]                     tag mr : 2.083 Mpps
00:39:30 [     INFO ]                tag sync mr : 1.827 Mpps
00:39:31 [     INFO ]                tag wild mr : 1.597 Mpps
00:39:32 [1507930771.748390] [hpc-test-node2:21162:0]          flush.c:336  UCX  WARN  flush failed: No resources are available to initiate the operation
00:39:32 [     INFO ]                     tag bw : 992.233 MB/sec
00:39:32 [     INFO ]         tag bw_zcopy_multi : 902.404 MB/sec
00:39:33 [     INFO ]                put latency : 2.281 usec
00:39:33 [     INFO ]                   put rate : 5.266 Mpps
00:39:34 [     INFO ]                     put bw : 4232.287 MB/sec
00:39:34 [     INFO ]                get latency : 2.175 usec
00:39:35 [     INFO ]                     get bw : 3634.934 MB/sec
00:39:36 [     INFO ]            atomic add rate : 1.008 Mpps
00:39:36 [     INFO ]        atomic fadd latency : 2.069 usec
00:39:37 [     INFO ]        atomic swap latency : 1.982 usec
00:39:37 [     INFO ]       atomic cswap latency : 2.111 usec
00:39:37 /scrap/jenkins/scrap/workspace/hpc-ucx-pr/label/hpc-test-node2/worker/3/contrib/../test/gtest/common/test.cc:244: Failure
00:39:37 Failed
00:39:37 Got 1 warnings during the test
00:39:37 
[  FAILED  ] dcx/test_ucp_perf.envelope/0, where GetParam() = \dc_mlx5 (11925 ms)

swx-jenkins1 · 2017-10-14T01:07:51Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/2810/ for details.

The blocking version of endpoint flush may potentially cause a deadlock because it does not progress communications on anything except the current worker. Introduce a non-blocking flush and use it for unit tests.

swx-jenkins1 · 2017-10-14T18:24:45Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/2816/ for details.

mellanox-github · 2017-10-14T19:04:31Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/4820/ for details (Mellanox internal link).

swx-jenkins1 · 2017-10-14T20:26:28Z

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/2820/ for details.

yosefe · 2017-10-14T20:28:05Z

bot:bgate:retest

mellanox-github · 2017-10-14T20:54:13Z

Test PASSed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/4823/ for details (Mellanox internal link).

swx-jenkins1 · 2017-10-15T08:42:41Z

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/2821/ for details.

swx-jenkins1 · 2017-10-15T11:07:31Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/2827/ for details.

shamisp · 2017-10-17T13:49:46Z

We need an example how block flush can be implemented, since the non-blocking calls almost always we be used in a blocking way. otherwise, it looks good

yosefe · 2017-10-18T17:21:53Z

@shamisp there is example in flush.c (blocking flush is now implemented over non-blocking one), and also in the unit tests

shamisp · 2017-10-19T14:23:51Z

I think we need to put an example that is exposed in ucp.h , so it goes to the spec,

shamisp · 2017-10-19T14:28:14Z

@bbenton please take a look. My only comment for now - we have some good examples in the spec to cover the concept. People not used to work with non-blocking flush.

+ Add API example for how to implement blocking flush. + Fix return status from blocking flush compatibility functions. + Use non-blocking flush in hello_world example.

swx-jenkins1 · 2017-10-21T09:47:01Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/2864/ for details.

mellanox-github · 2017-10-21T10:30:08Z

Test PASSed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/4885/ for details (Mellanox internal link).

yosefe · 2017-10-21T11:01:02Z

@shamisp fixed

shamisp · 2017-10-21T19:25:27Z

+1

…

On Sat, Oct 21, 2017 at 06:01 Yossi Itigin ***@***.***> wrote: @shamisp <https://github.com/shamisp> fixed — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1912 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACIe2MT7TujPNBPiNZiJQSYWjICVFVU2ks5suc7vgaJpZM4P5EuG> .

yosefe · 2017-10-25T01:05:13Z

@bbenton can you pls take a look?

shamisp · 2017-10-27T20:46:51Z

@bbenton - can you please take a look

gmegan · 2017-10-30T20:36:45Z

+1

UCP/API/TEST: Add non-blocking endpoint flush and use it for RMA tests

e7a73f7

The blocking version of endpoint flush may potentially cause a deadlock because it does not progress communications on anything except the current worker. Introduce a non-blocking flush and use it for unit tests.

yosefe force-pushed the topic/ucp-nb-flush branch from 136ee66 to e7a73f7 Compare October 14, 2017 17:41

yosefe added API Bugfix Feature New feature labels Oct 14, 2017

UCP/FLUSH: Fix return value check.

e0df0cd

yosefe added the Ready for Review label Oct 15, 2017

UCP/TEST/EXAMPLES: Code review fixes for non-blocking flush.

b7cc386

+ Add API example for how to implement blocking flush. + Fix return status from blocking flush compatibility functions. + Use non-blocking flush in hello_world example.

shamisp approved these changes Oct 26, 2017

View reviewed changes

yosefe merged commit c9151b1 into openucx:master Oct 31, 2017

yosefe deleted the topic/ucp-nb-flush branch October 31, 2017 07:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UCP/API/TEST: Add non-blocking endpoint flush and use it for RMA tests #1912

UCP/API/TEST: Add non-blocking endpoint flush and use it for RMA tests #1912

yosefe commented Oct 13, 2017 •

edited

Loading

mellanox-github commented Oct 13, 2017

yosefe commented Oct 13, 2017

swx-jenkins1 commented Oct 14, 2017

swx-jenkins1 commented Oct 14, 2017

mellanox-github commented Oct 14, 2017

swx-jenkins1 commented Oct 14, 2017

yosefe commented Oct 14, 2017

mellanox-github commented Oct 14, 2017

swx-jenkins1 commented Oct 15, 2017

swx-jenkins1 commented Oct 15, 2017

shamisp commented Oct 17, 2017

yosefe commented Oct 18, 2017

shamisp commented Oct 19, 2017

shamisp commented Oct 19, 2017

swx-jenkins1 commented Oct 21, 2017

mellanox-github commented Oct 21, 2017

yosefe commented Oct 21, 2017

shamisp commented Oct 21, 2017 via email

yosefe commented Oct 25, 2017

shamisp commented Oct 27, 2017

gmegan commented Oct 30, 2017

UCP/API/TEST: Add non-blocking endpoint flush and use it for RMA tests #1912

UCP/API/TEST: Add non-blocking endpoint flush and use it for RMA tests #1912

Conversation

yosefe commented Oct 13, 2017 • edited Loading

mellanox-github commented Oct 13, 2017

yosefe commented Oct 13, 2017

swx-jenkins1 commented Oct 14, 2017

swx-jenkins1 commented Oct 14, 2017

mellanox-github commented Oct 14, 2017

swx-jenkins1 commented Oct 14, 2017

yosefe commented Oct 14, 2017

mellanox-github commented Oct 14, 2017

swx-jenkins1 commented Oct 15, 2017

swx-jenkins1 commented Oct 15, 2017

shamisp commented Oct 17, 2017

yosefe commented Oct 18, 2017

shamisp commented Oct 19, 2017

shamisp commented Oct 19, 2017

swx-jenkins1 commented Oct 21, 2017

mellanox-github commented Oct 21, 2017

yosefe commented Oct 21, 2017

shamisp commented Oct 21, 2017 via email

yosefe commented Oct 25, 2017

shamisp commented Oct 27, 2017

gmegan commented Oct 30, 2017

yosefe commented Oct 13, 2017 •

edited

Loading