rdma/fi_domain: Add page size MR attr field

Page size allows applications to optionally notify the provider the page
size to be used for an MR allocation. Typically, providers can select
the optimal page size. In cases where a VA range has zero pages backing
it, the provider may not know the optimal page size during registration.
Rather than always use a less efficient page size, allow apps to specify
the page size to be used. If page size is zero, provider will select the
page size.

If non-zero, page size must be page size support by OS. If a specific
page size is specified for a memory region during creation, all pages
later associated with the region must be of the given size. Attaching a
memory page of a different size to a region may result in failed
transfers to or from the region.

Signed-off-by: Ian Ziemba <ian.ziemba@hpe.com>

include/rdma/fi_domain.h

@@ -175,6 +175,7 @@ struct fi_mr_attr {
 		int		synapseai;
 	} device;
 	void			*hmem_data;
+	size_t			page_size;
 };
 
 struct fi_mr_modify {

man/fi_mr.3.md

@@ -593,6 +593,7 @@ struct fi_mr_attr {
         int            synapseai;
     } device;
     void               *hmem_data;
+    size_t             page_size;
 };
 
 struct fi_mr_auth_key {
@@ -788,6 +789,26 @@ field is determined by the value specified through iface.
 ## hmem_data
 The hmem_data field is reserved for future use and must be null.
 
+## page_size
+Page size allows applications to optionally provide a hint at what the
+optimal page size is for the an MR allocation. Typically, providers can
+select the optimal page size. In cases where VA range has zero pages
+backing it, which is supported with FI_MR_ALLOCATED unset, the provider
+may not know the optimal page size during registration. Rather than use
+a less efficient page size, this attribute allows applications to specify
+the page size to be used.
+
+If page size is zero, provider will select the page size.
+
+If non-zero, page size must be supported by OS. If a specific page size
+is specified for a memory region during creation, all pages later
+associated with the region must be of the given size. Attaching a memory
+page of a different size to a region may result in failed transfers to
+or from the region.
+
+Providers may choose to ignore page size. This will result in a provider
+selected page size always being used.
+
 ## fi_hmem_ze_device
 
 Returns an hmem device identifier for a level zero <driver, device> tuple.

prov/util/src/util_mr_map.c

@@ -280,6 +280,11 @@ void ofi_mr_update_attr(uint32_t user_version, uint64_t caps,
 		cur_abi_attr->device.reserved = 0;
 		cur_abi_attr->hmem_data = NULL;
 	}
+
+	if (FI_VERSION_GE(user_version, FI_VERSION(1, 22)))
+		cur_abi_attr->page_size = user_attr->page_size;
+	else
+		cur_abi_attr->page_size = 0;
 }
 
 int ofi_mr_regattr(struct fid *fid, const struct fi_mr_attr *attr,