UCX/RNDV: multirail - updated EP configuration #1981

hoopoepg · 2017-11-08T08:47:48Z

added array of lanes to be used by rndv-mrail
added array of memory handles for multirail

this is split PR for #1894

jenkinsornl · 2017-11-08T09:16:51Z

Build finished.

swx-jenkins1 · 2017-11-08T09:30:32Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/3021/ for details.

mellanox-github · 2017-11-08T11:45:27Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5094/ for details (Mellanox internal link).

hoopoepg · 2017-11-08T11:47:55Z

bot:mlx:retest

swx-jenkins1 · 2017-11-08T13:50:02Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/3028/ for details.

jenkinsornl · 2017-11-08T14:35:16Z

Build finished.

mellanox-github · 2017-11-08T14:51:42Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5100/ for details (Mellanox internal link).

hoopoepg · 2017-11-08T14:54:06Z

bot:ornl:retest

mellanox-github · 2017-11-08T15:56:39Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5101/ for details (Mellanox internal link).

hoopoepg · 2017-11-08T16:46:26Z

bot:mlx:retest

jenkinsornl · 2017-11-08T16:56:39Z

Build finished.

mellanox-github · 2017-11-08T19:46:45Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5107/ for details (Mellanox internal link).

jenkinsornl · 2017-11-09T08:10:39Z

Build finished.

swx-jenkins1 · 2017-11-09T08:18:22Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/3035/ for details.

mellanox-github · 2017-11-09T09:39:37Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5114/ for details (Mellanox internal link).

swx-jenkins1 · 2017-11-09T11:16:35Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/3038/ for details.

mellanox-github · 2017-11-09T12:01:22Z

Test PASSed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5117/ for details (Mellanox internal link).

jenkinsornl · 2017-11-09T12:05:12Z

Build finished.

hoopoepg · 2017-11-09T12:08:09Z

bot:ornl:retest

jenkinsornl · 2017-11-09T14:00:30Z

Build finished.

hoopoepg · 2017-11-09T14:06:08Z

@MattBBaker could you look what is wrong?

MattBBaker · 2017-11-09T14:26:57Z

So the test actually succeeds, but then has a hiccup on the JSON it gets from github when setting the build to success.

MattBBaker · 2017-11-09T14:27:04Z

bot:ornl:retest

hoopoepg · 2017-11-09T14:27:57Z

ok, thank you

jenkinsornl · 2017-11-09T15:55:12Z

Build finished.

hoopoepg · 2017-11-09T16:16:46Z

@MattBBaker something wrong again

hoopoepg · 2017-11-09T16:25:34Z

bot:ornl:retest

jenkinsornl · 2017-11-10T18:15:17Z

Build finished.

swx-jenkins1 · 2017-11-10T18:28:00Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/3052/ for details.

mellanox-github · 2017-11-10T18:57:51Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5132/ for details (Mellanox internal link).

- additional fix for lane config

jenkinsornl · 2017-11-10T20:06:15Z

Build finished.

swx-jenkins1 · 2017-11-10T20:17:18Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/3053/ for details.

mellanox-github · 2017-11-10T20:58:05Z

Test PASSed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5133/ for details (Mellanox internal link).

brminich

Maybe it is better to remove RNDV lane and use RMA lane instead at first? Otherwise you would need to remove half of newly written code in the next PRs

brminich · 2017-11-10T21:13:56Z

src/ucp/core/ucp_context.c

@@ -100,6 +100,10 @@ static ucs_config_field_t ucp_config_table[] = {
   "the eager_zcopy protocol",
   ucs_offsetof(ucp_config_t, ctx.rndv_perf_diff), UCS_CONFIG_TYPE_DOUBLE},

+  {"MAX_RNDV_LANES", "1",
+   "Set max multirail-get rendezvous lane numbers",


Should we get rid of multirail word here as well?

multi-rail is still here

@brminich @yosefe
"Set max rendezvous-get lane numbers"
will work?

"Maximal number of devices on which a rendezvous operation may be executed in parallel."

brminich · 2017-11-10T21:14:33Z

src/ucp/core/ucp_context.h

@@ -53,6 +53,8 @@ typedef struct ucp_context_config {
    int                                    use_mt_mutex;
    /** On-demand progress */
    int                                    adaptive_progress;
+    /** Rendezvous multirail support */


multilanes?

Do you agree with multilanes instead of multi-rail?

brminich · 2017-11-10T21:18:00Z

src/ucp/core/ucp_ep.inl

+    return ucp_ep_config(ep)->key.rndv_lanes[idx] != UCP_NULL_LANE;
+}
+
+static inline int ucp_ep_rndv_num_lanes(ucp_ep_h ep)


better use UCS_F_ALWAYS_INLINE

all other funcs there are static inline
will add separate PR to set UCS_F_ALWAYS_INLINE

brminich · 2017-11-10T21:23:05Z

src/ucp/dt/dt.h

@@ -41,4 +42,26 @@ typedef struct ucp_dt_state {
 size_t ucp_dt_pack(ucp_datatype_t datatype, void *dest, const void *src,
                   ucp_dt_state_t *state, size_t length);

-#endif
+static UCS_F_ALWAYS_INLINE void
+ucp_dt_clear_rndv_lanes(ucp_dt_state_t *state)


maybe it is worth not to mention RNDV, as the code is rather generic and is in dt.h

brminich · 2017-11-10T21:27:24Z

src/ucp/tag/rndv.c

@@ -340,7 +357,7 @@ static void ucp_rndv_handle_recv_contig(ucp_request_t *rndv_req, ucp_request_t *
    } else {
        if (rndv_rts_hdr->flags & UCP_RNDV_RTS_FLAG_PACKED_RKEY) {
            UCS_PROFILE_CALL(uct_rkey_unpack, rndv_rts_hdr + 1,
-                             &rndv_req->send.rndv_get.rkey_bundle);
+                    &rndv_req->send.rndv_get.rkey_bundle);


indentation

- code beautify

jenkinsornl · 2017-11-11T04:08:03Z

Build finished.

swx-jenkins1 · 2017-11-11T04:18:05Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/3055/ for details.

mellanox-github · 2017-11-11T04:59:15Z

Test PASSed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5135/ for details (Mellanox internal link).

yosefe · 2017-11-13T12:41:28Z

@brminich pls re-review

- updated comment & help wording

jenkinsornl · 2017-11-13T13:41:59Z

Build finished.

swx-jenkins1 · 2017-11-13T13:51:27Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/3069/ for details.

mellanox-github · 2017-11-13T16:10:23Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5152/ for details (Mellanox internal link).

hoopoepg · 2017-11-13T16:12:21Z

bot:mlx:retest

mellanox-github · 2017-11-13T19:05:01Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5155/ for details (Mellanox internal link).

hoopoepg · 2017-11-14T08:02:30Z

bot:mlx:retest

mellanox-github · 2017-11-15T11:34:01Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5183/ for details (Mellanox internal link).

hoopoepg · 2017-11-15T14:01:22Z

bot:mlx:retest

mellanox-github · 2017-11-15T17:01:39Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5189/ for details (Mellanox internal link).

hoopoepg · 2017-11-15T17:05:30Z

bot:mlx:retest

mellanox-github · 2017-11-15T18:22:07Z

Test PASSed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/5191/ for details (Mellanox internal link).

hoopoepg · 2017-11-16T07:15:09Z

@miked-mellanox @alinask are you ok to merge this PR?
@yosefe and @brminich are approved it

this PR really blocks my work

hoopoepg force-pushed the topic/mrail-get-memh branch from 93d6eb9 to 04b7eee Compare November 8, 2017 13:07

hoopoepg force-pushed the topic/mrail-get-memh branch from 04b7eee to cb9336b Compare November 9, 2017 07:40

hoopoepg force-pushed the topic/mrail-get-memh branch from cb9336b to f6947c8 Compare November 9, 2017 10:37

UCX/RNDV: multirail - updated EP configuration (2)

22596c1

- additional fix for lane config

hoopoepg force-pushed the topic/mrail-get-memh branch from f9fcbcd to 22596c1 Compare November 10, 2017 19:37

brminich reviewed Nov 10, 2017

View reviewed changes

UCX/RNDV: multirail - updated EP configuration (3)

8cf6173

- code beautify

UCX/RNDV: multirail - updated EP configuration (4)

d989538

- updated comment & help wording

brminich approved these changes Nov 14, 2017

View reviewed changes

mike-dubman merged commit 4bd4c13 into openucx:master Nov 16, 2017

yosefe mentioned this pull request Dec 21, 2017

UCP/TAG: Fix RNDV get_zcopy progression #2097

Merged

UCX/RNDV: multirail - updated EP configuration #1981

UCX/RNDV: multirail - updated EP configuration #1981

Conversation

hoopoepg commented Nov 8, 2017

jenkinsornl commented Nov 8, 2017

swx-jenkins1 commented Nov 8, 2017

mellanox-github commented Nov 8, 2017

hoopoepg commented Nov 8, 2017

swx-jenkins1 commented Nov 8, 2017

jenkinsornl commented Nov 8, 2017

mellanox-github commented Nov 8, 2017

hoopoepg commented Nov 8, 2017

mellanox-github commented Nov 8, 2017

hoopoepg commented Nov 8, 2017

jenkinsornl commented Nov 8, 2017

mellanox-github commented Nov 8, 2017

jenkinsornl commented Nov 9, 2017

swx-jenkins1 commented Nov 9, 2017

mellanox-github commented Nov 9, 2017

swx-jenkins1 commented Nov 9, 2017

mellanox-github commented Nov 9, 2017

jenkinsornl commented Nov 9, 2017

hoopoepg commented Nov 9, 2017

jenkinsornl commented Nov 9, 2017

hoopoepg commented Nov 9, 2017

MattBBaker commented Nov 9, 2017

MattBBaker commented Nov 9, 2017

hoopoepg commented Nov 9, 2017

jenkinsornl commented Nov 9, 2017

hoopoepg commented Nov 9, 2017

hoopoepg commented Nov 9, 2017

jenkinsornl commented Nov 10, 2017

swx-jenkins1 commented Nov 10, 2017

mellanox-github commented Nov 10, 2017

jenkinsornl commented Nov 10, 2017

swx-jenkins1 commented Nov 10, 2017

mellanox-github commented Nov 10, 2017

brminich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jenkinsornl commented Nov 11, 2017

swx-jenkins1 commented Nov 11, 2017

mellanox-github commented Nov 11, 2017

yosefe commented Nov 13, 2017

jenkinsornl commented Nov 13, 2017

swx-jenkins1 commented Nov 13, 2017

mellanox-github commented Nov 13, 2017

hoopoepg commented Nov 13, 2017

mellanox-github commented Nov 13, 2017

hoopoepg commented Nov 14, 2017

mellanox-github commented Nov 15, 2017

hoopoepg commented Nov 15, 2017

mellanox-github commented Nov 15, 2017

hoopoepg commented Nov 15, 2017

mellanox-github commented Nov 15, 2017

hoopoepg commented Nov 16, 2017 • edited Loading

hoopoepg commented Nov 16, 2017 •

edited

Loading