[SHArP] about the intranode allreduce performance with SHArP

[shh2000]

Hello, @sjeaugey nccl team!

ENV:
8 * H100_SXM, ngctorch enviornment

Test Method:
NCCL_ALGO=[ALGO] ./all_reduce_perf -b 2G -e 4G -f 2 -g [gpus], where all_reduce_perf is compiled from nccl-test github repo

Result:

1. when using ALGO=RING, busbw is about 365-370 GBps when gpus from 2 to 8, about 80% of 450GBps(NVLink)
2. when using ALGO=NVLS, things become interest. Busbw remains about 365-370 GBps when gpus from 2 to 4. However, Busbw increase from 370 to 420, 440, 460, and 480 GBps when gpus increase from 4 to 5, 6, 7, and 8.
3. Besides, when gpus from 5 to 6, 7, and 8 at ALGO=NVLS, algbw also increases from 258 to about 265, 270, and 273.

Question:

1. how does 480GBps been calculated? SHArP ALU @nvswitch3 has 400GFLOPS@float32. Is 400GFLOPS any relationship with 480GBps?
2. each H100 should be connected directly with all nvswitches, so how could algbw@5gpus lower than algbw@8gpus? In addition, why gpus=2, 3, and 4 remain 370GBps like Ring allreduce?

Looking forward to you reply