prov/verbs: add param for setting RDMA CM ToS #10360
Open
+47
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
j-xiong
reviewed
Sep 4, 2024
|
Please add a description of the new variable to the verbs provider man page. You can also drop the "RFC" part from the title. |
jakemoroni
force-pushed
the
main
branch
2 times, most recently
from
80995b1
to
20fac51
Compare
jakemoroni
changed the title
Done. Also rebased and retested. |
By default, if the RDMA CM application does not explicitly set a ToS, then the system default is used. This system default can be changed via the default_roce_tos configfs param, but this is global to the system and is somewhat cumbersome to deal with. This change introduces a new environment param: FI_VERBS_TOS This param can be used to explicitly set the ToS via the rdma_set_option API. If the ToS is set, then the system default is ignored and the new value is used instead. This allows for multiple concurrent workloads to use different ToS values. Valid range is -1 through 255. If unset or set to -1, then the call to rdma_set_option is omitted, thus preserving existing behavior. Not supported/tested on Windows. Signed-off-by: Jacob Moroni <jmoroni@google.com>
|
The fi_tx_attr includes a tclass value that can be used to control the service level. It supports DSCP via helper functions. Can this be used instead? |
|
bot:aws:retest |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Opens:
rdma_set_optionso it should build and have no effect if no ToS is set. If a ToS is set, then it should result in a warning.verbs_domain_xrc.c. They may not all be necessary.FI_VERBS_TOSparam takes a range of -1 through 255, with -1 being a special value to indicate "no ToS set" (default), which allows the call tordma_set_optionto be completely skipped. My original plan was to just have 0 be the default and always callrdma_set_optionbut this would introduce a behavioral change on systems where the default is non-0 and would add extra overhead in scenarios where the ToS is a don't care.By default, if the RDMA CM application does not explicitly set a ToS, then the system default is used. This system default can be changed via the default_roce_tos configfs param, but this is global to the system and is somewhat cumbersome to deal with.
This change introduces a new environment param: FI_VERBS_TOS
This param can be used to explicitly set the ToS via the rdma_set_option API. If the ToS is set, then the system default is ignored and the new value is used instead.
This allows for multiple concurrent workloads to use different ToS values.
Valid range is -1 through 255. If unset or set to -1, then the call to rdma_set_option is omitted, thus preserving existing behavior.