UCT: fix hang when using polling fd #1492

[evgeny-leksikov]

Fixes #1492

[mellanox-github]

Test FAILed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/1787/ for details.

[mellanox-github]

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/1788/ for details.

[mellanox-github]

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/3724/ for details (Mellanox internal link).

[mellanox-github]

Test PASSed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/3725/ for details (Mellanox internal link).

[evgeny-leksikov]

@yosefe @brminich pls review

[yosefe]

@evgeny-leksikov why need to rearm inside progress, if the user has to explicitly call arm after progress anyway?
e.g use flow is:

1. progress
2. arm
3. epoll_wait/...

so how is adding "arm" after (1) helps, given that we already have arm in (2)?

[evgeny-leksikov]

@yosefe because arm did not re-arm if there are any events. Pls, take a look now.

[yosefe]

@evgeny-leksikov user should call arm again when getting BUSY, until he gets UCS_OK, which would mean the cq is really re-armed.

[yosefe]

@MattBBaker can you please post ORNL failure?

[evgeny-leksikov]

@yosefe yes, but some events can be missed between BUSY and next re-arm but not polled from the CQ by progress. So, we have to have re-armed transport all the time.

[mellanox-github]

Test PASSed.
See http://bgate.mellanox.com/jenkins/job/gh-ucx-pr/1794/ for details.

[yosefe]

@evgeny-leksikov so you mean that if there are unpolled CQEs, and then arm is done, no event is generated?

[evgeny-leksikov]

yes

[mellanox-github]

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/3733/ for details (Mellanox internal link).

[evgeny-leksikov]

bot:mlx:retest

[yosefe]

@evgeny-leksikov what if the event arrives just before arming is done, and the cq is not armed yet? for example, on the first time?

[mellanox-github]

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/3735/ for details (Mellanox internal link).

[mellanox-github]

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/3738/ for details (Mellanox internal link).

[MattBBaker]

@yosefe Looks like a build environment failure.

[MattBBaker]

bot:ornl:retest

[MattBBaker]

bot:ornl:retest

[shamisp]

probably should go to v1.2 as well

[evgeny-leksikov]

bot:mlx:retest

[mellanox-github]

Test FAILed.
See http://hpc-master.lab.mtl.com:8080/job/hpc-ucx-pr/3788/ for details (Mellanox internal link).

[yosefe]

the reported issue is not a bug