UCT/TCP: Implement flush of all outstanding operations [v1.11.x]

[dmitrygx]

backport of #7140 PR to v1.11.x branch to be included in 1.11.1 release version

What

Implement flush of all outstanding operations.

Why ?

To fix uct_ep_flush() which don't wait for all operations being completed. As a result if fixes the following case:

/* client */
for (i= 0; i < 1000; ++i) { 
    req = ucp_ep_tag_send_nb(ep)
    reqs.push_back(req);
}
wait_for_reqs(reqs);

req = ucp_ep_close_nb(ep, FLUSH);
wait_for_req(req);

exit(0);

/* server */
for (i = 0; i < 1000; ++i) {
    req = ucp_worker_tag_recv_nb(worker);
    reqs.push_back(req);
}
wait_for_reqs(reqs);

/* 998-999 completed successfully, but 1-2 completed with error  */

How ?

1. Do PUT operation if last_acked_sn != tx.sn in EP
2. Fix EP flush conditions when checking resources. Return UCS_OK if connection has already been closed.
3. Fix tests.

[yosefe]

any conflicts?

[dmitrygx]

any conflicts?

@yosefe only here -

ucx/src/tools/perf/lib/libperf_int.h

Line 161 in eb2f218

void ucp_perf_thread_barrier(ucx_perf_context_t *perf);

here is what we we have in master (no empty lines between declarations):

ucx/src/tools/perf/lib/libperf_int.h

Line 152 in 9d62432

void ucp_perf_thread_barrier(ucx_perf_context_t *perf);

[dmitrygx]

@yosefe ok to merge?