Manual progress performance with fi_cq_sread()

[theWayofthecode]

Hi,

Intuitively it seems to me that the count param of the fi_cq_sread() should affect the performance because it tells the runtime that, e.g. for a receiving endpoint, "we are ready to accept count number of transfers". So it seems the higher the count the better.

What tips do you have to tune this count parameter or the manual progress in general so that we improve the performance?

Thanks,
Aram

[shefty]

sread is a blocking call, which will impact raw latency numbers. The count parameter is the max number of completions that will be returned. The provider isn't expecting to wait until that number of completions are ready.

I would need more details on how you are measuring performance (latency, bandwidth, message rate, CPU utilization, etc.) and what provider you're using to provide much help on possible tuning parameters. There are a lot of variables available, depending on the provider. If you run 'fi_info -e', that will list runtime variables that you can adjust. 'fi_info -g <search_term>' can also be useful to filter the list down to a specific provider name or option.

[theWayofthecode]

Currently we're using the verbs provider and the shm (the latter for development purposes). Our integration is at an early stage so we don't have detailed performance measurements, and neither we're focusing yet on the final fine-tuning of the performance. We just want to avoid big mistakes that will hurt the performance significantly. We're concentrating on throughput (we'll check latency in the near future), and we want to make sure that the receiver (which is receiving from many), is not stalling the senders. At high level, is it ok then to assume that, if sread_cq (or read_cq) is called regularly then we shouldn't worry much about the count parameter? So sread_cq() returns as soon as entries are generated in the queue, without waiting actually to reach count number of entries?

PS: Thank you for the hints for searching the tuning params. I'll check them to better understand our options.

[shefty]

Yes, sread_cq will return as soon as entries are available. Additionally, if entries are available when the call is invoked, the thread will not block and return immediately.