UCP/WIREUP: Consider local distance during slow lanes dropping - v1.16.x - revert

[ivankochin]

What

Reverting the backport of #9512 since it breaks wire compatibility.

[yosefe]

Do we need to revert it on master branch?
How did you find the issue? Why we didn't catch it in CI?

[ivankochin]

Do we need to revert it on master branch?

No, we didn't, the fix works OK on master.

How did you find the issue?

It was just a logical conclusion. The original fixes that works for master checked that UCX version is >= 17 and then applied new logic, with older UCX version it leaves the logic as is to avoid different lanes performance on old UCX. If we somehow change that for UCX 1.16, it will break the logic for master branch.

Why we didn't catch it in CI?

I think this error can require some specific setup to be reproduced.

[yosefe]

I think this error can require some specific setup to be reproduced.

Maybe because CI doesn't test current v1.16.x vs master branch?
@brminich WDYT?

[ivankochin]

I think this error can require some specific setup to be reproduced.

Maybe because CI doesn't test current v1.16.x vs master branch? @brminich WDYT?

Yes, that should be the case, but also it should be the system with several NICs and these NICs should have different distance to GPU.

The original issue can be reproduced on DGX1

[yosefe]

ok
but probably need to fix the commit title by force push

[ivankochin]

ok but probably need to fix the commit title by force push

What's wrong with the commit title?

[brminich]

I think this error can require some specific setup to be reproduced.

Maybe because CI doesn't test current v1.16.x vs master branch? @brminich WDYT?

we do test it https://github.com/openucx/ucx/blob/master/buildlib/pr/wire_compat.yml#L11
i think we need a specific server config to get it

[yosefe]

I think this error can require some specific setup to be reproduced.

Maybe because CI doesn't test current v1.16.x vs master branch? @brminich WDYT?

we do test it https://github.com/openucx/ucx/blob/master/buildlib/pr/wire_compat.yml#L11 i think we need a specific server config to get it

I think in the current CI config - future master PRs would have failed after we merged something bad to v1.16.x
but we need a bad v1.16.x PR to fail. maybe need to test master branch PRs vs. release tags, and release branch PRs vs. master?

[ivankochin]

@yosefe pls review