UCT/IB: Use separate resource domain for mlx5 trasnports #2338

yosefe · 2018-02-21T16:55:43Z

Fixes #1926

mlx5 transports manage their own blue-flame register state. In order to avoid inconsistencies with verbs transports, they must not share that register. This change is using the verbs "resource domain" API in order to force the separation, since QPs created on different resource domains will have separate blue-flame register space.

mellanox-github · 2018-02-21T18:21:14Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/6336/ for details (Mellanox internal link).

swx-jenkins1 · 2018-02-21T18:38:06Z

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/4059/ for details.

mellanox-github · 2018-02-21T19:54:03Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/6339/ for details (Mellanox internal link).

mellanox-github · 2018-02-21T20:24:30Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/6340/ for details (Mellanox internal link).

brminich · 2018-02-22T06:53:43Z

src/uct/ib/base/ib_iface.c

+
+    return res_domain->ibv_domain->context == dev->ibv_context;
+}
+


missing #if HAVE_IBV_EXP_RES_DOMAIN

brminich · 2018-02-22T07:06:29Z

src/uct/ib/rc/base/rc_iface.c

@@ -919,7 +920,7 @@ ucs_status_t uct_rc_iface_qp_create(uct_rc_iface_t *iface, int qp_type,

 #  if HAVE_DECL_IBV_EXP_ATOMIC_HCA_REPLY_BE
    if (dev->dev_attr.exp_atomic_cap == IBV_EXP_ATOMIC_HCA_REPLY_BE) {
-        qp_init_attr.comp_mask       |= IBV_EXP_QP_INIT_ATTR_CREATE_FLAGS;
+        qp_init_attr.comp_mask      |= IBV_EXP_QP_INIT_ATTR_CREATE_FLAGS;


previous alignment was correct

brminich · 2018-02-22T07:08:57Z

src/uct/ib/rc/accel/rc_mlx5_iface.c

@@ -236,7 +236,8 @@ static UCS_CLASS_INIT_FUNC(uct_rc_mlx5_iface_t, uct_md_h md, uct_worker_h worker

    UCS_CLASS_CALL_SUPER_INIT(uct_rc_iface_t, &uct_rc_mlx5_iface_ops, md, worker,
                              params, &config->super, 0,
-                              sizeof(uct_rc_fc_request_t), IBV_EXP_TM_CAP_RC);
+                              sizeof(uct_rc_fc_request_t), IBV_EXP_TM_CAP_RC,
+                              UCT_IB_MLX5_RES_DOMAIN_KEY);


Is it ok that all accelerated transports use the same domain key?

yes because they share the BF register structure as well (with the same key)

shamisp · 2018-02-26T17:37:04Z

And if you don't separate these ? I guess verbs and mlx5 will step on each other and you will end up having more dropped doorbells than you want ?

yosefe · 2018-02-26T17:38:32Z

@shamisp yes, they step on each other. i've observed data corruption, though dropped doorbells can happen too

shamisp · 2018-02-26T17:58:51Z

ouch... this is critical one.

BTW, Can we add some check (ucx info ?) validating that majority of BF doorbells are successful ?

yosefe · 2018-02-26T18:16:42Z

reading the counters is not possible without root privileges..

shamisp · 2018-02-26T18:29:48Z

yes...

…

On Mon, Feb 26, 2018 at 6:16 PM, Yossi Itigin ***@***.***> wrote: reading the counters is not possible without root privileges.. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2338 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACIe2FO24eCYSpmvz3_-jnLZ46M813d7ks5tYvULgaJpZM4SN-tc> .

…-res-domain Conflicts: src/uct/ib/mlx5/ib_mlx5.h

swx-jenkins1 · 2018-03-08T11:32:42Z

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/4183/ for details.

mellanox-github · 2018-03-08T12:26:53Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/6491/ for details (Mellanox internal link).

swx-jenkins1 · 2018-03-08T19:31:17Z

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/4191/ for details.

mellanox-github · 2018-03-08T21:00:33Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/6499/ for details (Mellanox internal link).

yosefe · 2018-03-21T23:35:13Z

bot:mlx:retest

mellanox-github · 2018-03-22T02:08:39Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/6600/ for details (Mellanox internal link).

…-res-domain

mellanox-github · 2018-03-22T13:57:38Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/6610/ for details (Mellanox internal link).

shamisp · 2018-03-22T14:31:32Z

config/m4/ib.m4

+                       ibv_exp_destroy_res_domain],
+                      [AC_DEFINE([HAVE_IBV_EXP_RES_DOMAIN], 1, [IB resource domain])],
+                      [AC_MSG_WARN([Cannot use mlx5 accel because resource domains are not supported])
+                       AC_MSG_WARN([Please upgrade MellanoxOFED to 3.1 or above])


swx-jenkins1 · 2018-03-22T15:40:04Z

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/4290/ for details.

mellanox-github · 2018-03-22T16:34:28Z

Test PASSed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/6612/ for details (Mellanox internal link).

yosefe · 2018-03-23T10:48:29Z

bot:bgate:retest

swx-jenkins1 · 2018-03-23T11:44:20Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/4301/ for details.

…-res-domain

swx-jenkins1 · 2018-03-25T12:24:49Z

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/4304/ for details.

mellanox-github · 2018-03-25T14:05:17Z

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/6629/ for details (Mellanox internal link).

mellanox-github · 2018-03-25T17:08:48Z

Test PASSed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/6631/ for details (Mellanox internal link).

yosefe · 2018-03-26T08:15:20Z

@brminich pls take a look

artpol84 · 2018-03-29T20:07:17Z

@yosefe
I confirm that lost doorbells was the case in my multi-threaded tests

brminich reviewed Feb 22, 2018

View reviewed changes

yosefe added the WIP-DNM Work in progress / Do not review label Feb 22, 2018

yosefe added 2 commits March 8, 2018 11:41

UCT/IB: Fix compilation without resource_domain, fix indentation

0819e65

Merge remote-tracking branch 'upstream/master' into topic/uct-ib-mlx5…

3cc615d

…-res-domain Conflicts: src/uct/ib/mlx5/ib_mlx5.h

yosefe added Bugfix and removed WIP-DNM Work in progress / Do not review labels Mar 8, 2018

CONTRIB: Add leak suppression for ibv_exp_create_res_domain

0929040

UCT/DC/MLX5: Add missing cleanup for DCI workqueues

40431c7

yosefe changed the title ~~UCT/IB: Use separate resourcve domain for mlx5 trasnports~~ UCT/IB: Use separate resource domain for mlx5 trasnports Mar 12, 2018

yosefe added 2 commits March 22, 2018 12:16

Merge remote-tracking branch 'upstream/master' into topic/uct-ib-mlx5…

2e77d71

…-res-domain

CONTRIB/JENKINS: Generate valgrind suppressions

997f9c9

shamisp reviewed Mar 22, 2018

View reviewed changes

Merge remote-tracking branch 'upstream/master' into topic/uct-ib-mlx5…

fe17377

…-res-domain

yosefe force-pushed the topic/uct-ib-mlx5-res-domain branch from b17b836 to fe17377 Compare March 25, 2018 11:30

brminich approved these changes Mar 26, 2018

View reviewed changes

yosefe merged commit 81939df into openucx:master Mar 26, 2018

yosefe deleted the topic/uct-ib-mlx5-res-domain branch March 26, 2018 09:59

hoopoepg mentioned this pull request Mar 27, 2018

hang in rcx/test_ucp_peer_failure_2pairs.status_after_error/0 on arm #2450

Closed

yosefe mentioned this pull request Mar 29, 2018

ib/mlx5 BlueFlame issue for MT applications #2464

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UCT/IB: Use separate resource domain for mlx5 trasnports #2338

UCT/IB: Use separate resource domain for mlx5 trasnports #2338

yosefe commented Feb 21, 2018 •

edited

Loading

mellanox-github commented Feb 21, 2018

swx-jenkins1 commented Feb 21, 2018

mellanox-github commented Feb 21, 2018

mellanox-github commented Feb 21, 2018

brminich Feb 22, 2018

yosefe Mar 8, 2018

brminich Feb 22, 2018

yosefe Mar 8, 2018

brminich Feb 22, 2018

yosefe Mar 8, 2018

shamisp commented Feb 26, 2018

yosefe commented Feb 26, 2018 •

edited

Loading

shamisp commented Feb 26, 2018

yosefe commented Feb 26, 2018

shamisp commented Feb 26, 2018 via email

swx-jenkins1 commented Mar 8, 2018

mellanox-github commented Mar 8, 2018

swx-jenkins1 commented Mar 8, 2018

mellanox-github commented Mar 8, 2018

yosefe commented Mar 21, 2018

mellanox-github commented Mar 22, 2018

mellanox-github commented Mar 22, 2018

shamisp Mar 22, 2018

swx-jenkins1 commented Mar 22, 2018

mellanox-github commented Mar 22, 2018

yosefe commented Mar 23, 2018

swx-jenkins1 commented Mar 23, 2018

swx-jenkins1 commented Mar 25, 2018

mellanox-github commented Mar 25, 2018

mellanox-github commented Mar 25, 2018

yosefe commented Mar 26, 2018

artpol84 commented Mar 29, 2018


		return res_domain->ibv_domain->context == dev->ibv_context;
		}

UCT/IB: Use separate resource domain for mlx5 trasnports #2338

UCT/IB: Use separate resource domain for mlx5 trasnports #2338

Conversation

yosefe commented Feb 21, 2018 • edited Loading

mellanox-github commented Feb 21, 2018

swx-jenkins1 commented Feb 21, 2018

mellanox-github commented Feb 21, 2018

mellanox-github commented Feb 21, 2018

brminich Feb 22, 2018

Choose a reason for hiding this comment

yosefe Mar 8, 2018

Choose a reason for hiding this comment

brminich Feb 22, 2018

Choose a reason for hiding this comment

yosefe Mar 8, 2018

Choose a reason for hiding this comment

brminich Feb 22, 2018

Choose a reason for hiding this comment

yosefe Mar 8, 2018

Choose a reason for hiding this comment

shamisp commented Feb 26, 2018

yosefe commented Feb 26, 2018 • edited Loading

shamisp commented Feb 26, 2018

yosefe commented Feb 26, 2018

shamisp commented Feb 26, 2018 via email

swx-jenkins1 commented Mar 8, 2018

mellanox-github commented Mar 8, 2018

swx-jenkins1 commented Mar 8, 2018

mellanox-github commented Mar 8, 2018

yosefe commented Mar 21, 2018

mellanox-github commented Mar 22, 2018

mellanox-github commented Mar 22, 2018

shamisp Mar 22, 2018

Choose a reason for hiding this comment

swx-jenkins1 commented Mar 22, 2018

mellanox-github commented Mar 22, 2018

yosefe commented Mar 23, 2018

swx-jenkins1 commented Mar 23, 2018

swx-jenkins1 commented Mar 25, 2018

mellanox-github commented Mar 25, 2018

mellanox-github commented Mar 25, 2018

yosefe commented Mar 26, 2018

artpol84 commented Mar 29, 2018

yosefe commented Feb 21, 2018 •

edited

Loading

yosefe commented Feb 26, 2018 •

edited

Loading