UCS/CALLBACKQ: fix recursive one-shot #5926

evgeny-leksikov · 2020-11-18T20:48:03Z

What

Fix recursive one-shot

Why ?

To avoid a stuck until out-of-memory

How ?

limit iteration count by actual value before the loop

yosefe

in general LGTM, just need to fix commit title short->shot and build issues

dmitrygx · 2020-11-20T13:52:33Z

src/ucp/rma/flush.c

@@ -115,7 +118,7 @@ static void ucp_ep_flush_progress(ucp_request_t *req)
            }
        } else {
            ucp_ep_flush_error(req, status);
-            break;
+            req->send.flush.started_lanes |= UCS_BIT(lane);


should we do
--req->send.state.uct_comp.count;?

move req->send.flush.started_lanes |= UCS_BIT(lane); to ucp_ep_flush_error()?

should we do
--req->send.state.uct_comp.count;?

it's done in ucp_ep_flush_error

move req->send.flush.started_lanes |= UCS_BIT(lane); to ucp_ep_flush_error()?

it will be wrong for pending_add failure

evgeny-leksikov · 2020-11-26T07:46:31Z

bot:pipe:retest

evgeny-leksikov · 2020-11-26T07:46:45Z

/azp run

azure-pipelines · 2020-11-26T07:46:56Z

Azure Pipelines successfully started running 1 pipeline(s).

evgeny-leksikov · 2020-11-26T13:57:02Z

bot:pipe:retest

yosefe · 2020-11-27T11:39:09Z

bot:pipe:retest

evgeny-leksikov · 2020-11-27T11:53:24Z

@yosefe the failure is relevant, reproduced locally when network interface is TCP but selected transport is RCX

evgeny-leksikov · 2020-11-30T07:12:54Z

bot:pipe:retest

evgeny-leksikov · 2020-12-02T07:20:04Z

bot:pipe:retest

yosefe · 2020-12-03T21:09:38Z

src/ucp/proto/proto_am.inl

@@ -348,6 +348,9 @@ ucs_status_t ucp_do_am_zcopy_multi(uct_pending_req_t *self, uint8_t am_id_first,
        ucp_send_request_add_reg_lane(req, req->send.lane);
    } else {
        req->send.lane = ucp_ep_get_am_lane(ep);
+        if (req->send.state.dt.offset == 0) {
+            ucp_send_request_add_reg_lane(req, req->send.lane);


maybe move this after line 354?
seems only case ucp_send_request_add_reg_lane is not called, is when enable_am_bw is false

yosefe · 2020-12-04T15:06:13Z

bot:pipe:retest

src/ucp/proto/proto_am.inl

yosefe · 2021-01-11T08:56:09Z

pls squash

+ reg send lane on first iter in UCP zcopy multy protocol

yosefe · 2021-01-12T09:56:31Z

@evgeny-leksikov failure could be related. AFAIR, didn't see such failure before

[2021-01-11T12:36:36.864Z] [ RUN      ] all/test_ucp_sockaddr_protocols.stream_bcopy_4k_exp/0 <all>
[2021-01-11T12:36:37.120Z] [     INFO ] < rdma_bind_addr(addr=10.209.45.93:0) failed: No such device >
[2021-01-11T12:36:37.120Z] [     INFO ] server listening on 10.209.45.93:36814
[2021-01-11T12:36:38.488Z] [r-vmb-ppc-jenkins:26934:0:26934] ucp_request.inl:254  Fatal: unexpected error: Invalid parameter
[2021-01-11T12:36:38.785Z] 
[2021-01-11T12:36:38.785Z] /scrap/jenkins/workspace/ucx-5/contrib/../src/ucp/core/ucp_request.inl: [ ucp_request_try_send() ]
[2021-01-11T12:36:38.785Z]       ...
[2021-01-11T12:36:38.785Z]       251     }
[2021-01-11T12:36:38.785Z]       252 
[2021-01-11T12:36:38.785Z]       253     ucs_fatal("unexpected error: %s", ucs_status_string(status));
[2021-01-11T12:36:38.785Z] ==>   254 }
[2021-01-11T12:36:38.785Z]       255 
[2021-01-11T12:36:38.785Z]       256 /**
[2021-01-11T12:36:38.785Z]       257  * Start sending a request.
[2021-01-11T12:36:38.785Z] 
[2021-01-11T12:36:39.366Z] ==== backtrace (tid:  26934) ====
[2021-01-11T12:36:39.366Z]  0 0x000000000005f848 ucs_debug_print_backtrace()  /scrap/jenkins/workspace/ucx-5/contrib/../src/ucs/debug/debug.c:656
[2021-01-11T12:36:39.366Z]  1 0x00000000000af96c ucp_request_try_send()  /scrap/jenkins/workspace/ucx-5/contrib/../src/ucp/core/ucp_request.inl:254
[2021-01-11T12:36:39.366Z]  2 0x00000000000af96c ucp_request_send()  /scrap/jenkins/workspace/ucx-5/contrib/../src/ucp/core/ucp_request.inl:267
[2021-01-11T12:36:39.366Z]  3 0x00000000000af96c ucp_wireup_replay_pending_request()  /scrap/jenkins/workspace/ucx-5/contrib/../src/ucp/wireup/wireup.c:753
[2021-01-11T12:36:39.366Z]  4 0x00000000000af96c ucp_wireup_replay_pending_requests()  /scrap/jenkins/workspace/ucx-5/contrib/../src/ucp/wireup/wireup.c:763
[2021-01-11T12:36:39.366Z]  5 0x00000000000ab7cc ucp_wireup_ep_progress()  /scrap/jenkins/workspace/ucx-5/contrib/../src/ucp/wireup/wireup_ep.c:99
[2021-01-11T12:36:39.366Z]  6 0x0000000000051d04 ucs_callbackq_slow_proxy()  /scrap/jenkins/workspace/ucx-5/contrib/../src/ucs/datastruct/callbackq.c:402
[2021-01-11T12:36:39.366Z]  7 0x0000000000047894 ucs_callbackq_dispatch()  /scrap/jenkins/workspace/ucx-5/contrib/../src/ucs/datastruct/callbackq.h:211
[2021-01-11T12:36:39.366Z]  8 0x0000000000047894 uct_worker_progress()  /scrap/jenkins/workspace/ucx-5/contrib/../src/uct/api/uct.h:2436
[2021-01-11T12:36:39.366Z]  9 0x0000000000047894 ucp_worker_progress()  /scrap/jenkins/workspace/ucx-5/contrib/../src/ucp/core/ucp_worker.c:2403

evgeny-leksikov · 2021-01-12T13:55:58Z

@evgeny-leksikov failure could be related. AFAIR, didn't see such failure before

yep, reproduced locally, this is the EP reconfiguration issue when request is in pending queue in case of UCX_CM_USE_ALL_DEVICES, initial AM lane is TCP, then switching to IB

yosefe · 2021-01-12T17:41:47Z

yep, reproduced locally, this is the EP reconfiguration issue when request is in pending queue in case of UCX_CM_USE_ALL_DEVICES, initial AM lane is TCP, then switching to IB

Did this PR introduce the issue, or just "uncover" it?

evgeny-leksikov · 2021-01-12T18:41:52Z

yep, reproduced locally, this is the EP reconfiguration issue when request is in pending queue in case of UCX_CM_USE_ALL_DEVICES, initial AM lane is TCP, then switching to IB

Did this PR introduce the issue, or just "uncover" it?

uncover

evgeny-leksikov · 2021-01-26T06:59:12Z

failure is #6194

evgeny-leksikov · 2021-01-26T06:59:19Z

/azp run

azure-pipelines · 2021-01-26T06:59:31Z

Azure Pipelines successfully started running 1 pipeline(s).

evgeny-leksikov · 2021-01-26T15:08:48Z

@yosefe ok to merge?

evgeny-leksikov requested a review from yosefe November 18, 2020 20:48

yosefe reviewed Nov 18, 2020

View reviewed changes

evgeny-leksikov force-pushed the ucs_cbq branch from d4b22b8 to 1fb7e41 Compare November 19, 2020 07:07

evgeny-leksikov changed the title ~~UCS/CALLBACKQ: fix recursive one-short~~ UCS/CALLBACKQ: fix recursive one-shot Nov 19, 2020

evgeny-leksikov force-pushed the ucs_cbq branch from 1fb7e41 to 9c30b34 Compare November 19, 2020 08:32

dmitrygx mentioned this pull request Nov 19, 2020

UCP/CORE: Fix uncompleted close request #5928

Closed

evgeny-leksikov mentioned this pull request Nov 20, 2020

UCP/CONTEXT: add DEFER_ERR_HANDLER_TIME configuration #5931

Closed

dmitrygx approved these changes Nov 20, 2020

View reviewed changes

dmitrygx reviewed Nov 20, 2020

View reviewed changes

yosefe approved these changes Nov 20, 2020

View reviewed changes

This was referenced Nov 23, 2020

UCP/EP/FLUSH: fix error handling flow yosefe/ucx#93

Merged

UCP/EP/FLUSH: fix started_lanes mask on failure #5946

Merged

evgeny-leksikov force-pushed the ucs_cbq branch from 0a83780 to 9c30b34 Compare November 25, 2020 11:58

yosefe approved these changes Nov 26, 2020

View reviewed changes

yosefe reviewed Dec 3, 2020

View reviewed changes

evgeny-leksikov requested a review from yosefe December 17, 2020 15:03

yosefe reviewed Dec 17, 2020

View reviewed changes

src/ucp/proto/proto_am.inl Show resolved Hide resolved

yosefe approved these changes Jan 11, 2021

View reviewed changes

UCS/CALLBACKQ: fix recursive one-shot

b8ab8b0

+ reg send lane on first iter in UCP zcopy multy protocol

evgeny-leksikov force-pushed the ucs_cbq branch from 087e922 to b8ab8b0 Compare January 11, 2021 10:41

yosefe approved these changes Jan 12, 2021

View reviewed changes

evgeny-leksikov mentioned this pull request Jan 13, 2021

UCP/WIREUP/CM: delay client EP restoring for all devices case #6127

Merged

yosefe closed this in #6127 Jan 22, 2021

evgeny-leksikov reopened this Jan 25, 2021

yosefe merged commit 07b22d4 into openucx:master Jan 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UCS/CALLBACKQ: fix recursive one-shot #5926

UCS/CALLBACKQ: fix recursive one-shot #5926

evgeny-leksikov commented Nov 18, 2020 •

edited

Loading

yosefe left a comment

dmitrygx Nov 20, 2020

dmitrygx Nov 20, 2020

evgeny-leksikov Nov 20, 2020

evgeny-leksikov Nov 20, 2020

evgeny-leksikov commented Nov 26, 2020

evgeny-leksikov commented Nov 26, 2020

azure-pipelines bot commented Nov 26, 2020

evgeny-leksikov commented Nov 26, 2020

yosefe commented Nov 27, 2020

evgeny-leksikov commented Nov 27, 2020

evgeny-leksikov commented Nov 30, 2020

evgeny-leksikov commented Dec 2, 2020

yosefe Dec 3, 2020

yosefe commented Dec 4, 2020

yosefe commented Jan 11, 2021

yosefe commented Jan 12, 2021 •

edited

Loading

evgeny-leksikov commented Jan 12, 2021

yosefe commented Jan 12, 2021

evgeny-leksikov commented Jan 12, 2021

evgeny-leksikov commented Jan 26, 2021

evgeny-leksikov commented Jan 26, 2021

azure-pipelines bot commented Jan 26, 2021

evgeny-leksikov commented Jan 26, 2021

UCS/CALLBACKQ: fix recursive one-shot #5926

UCS/CALLBACKQ: fix recursive one-shot #5926

Conversation

evgeny-leksikov commented Nov 18, 2020 • edited Loading

What

Why ?

How ?

yosefe left a comment

Choose a reason for hiding this comment

dmitrygx Nov 20, 2020

Choose a reason for hiding this comment

dmitrygx Nov 20, 2020

Choose a reason for hiding this comment

evgeny-leksikov Nov 20, 2020

Choose a reason for hiding this comment

evgeny-leksikov Nov 20, 2020

Choose a reason for hiding this comment

evgeny-leksikov commented Nov 26, 2020

evgeny-leksikov commented Nov 26, 2020

azure-pipelines bot commented Nov 26, 2020

evgeny-leksikov commented Nov 26, 2020

yosefe commented Nov 27, 2020

evgeny-leksikov commented Nov 27, 2020

evgeny-leksikov commented Nov 30, 2020

evgeny-leksikov commented Dec 2, 2020

yosefe Dec 3, 2020

Choose a reason for hiding this comment

yosefe commented Dec 4, 2020

yosefe commented Jan 11, 2021

yosefe commented Jan 12, 2021 • edited Loading

evgeny-leksikov commented Jan 12, 2021

yosefe commented Jan 12, 2021

evgeny-leksikov commented Jan 12, 2021

evgeny-leksikov commented Jan 26, 2021

evgeny-leksikov commented Jan 26, 2021

azure-pipelines bot commented Jan 26, 2021

evgeny-leksikov commented Jan 26, 2021

evgeny-leksikov commented Nov 18, 2020 •

edited

Loading

yosefe commented Jan 12, 2021 •

edited

Loading