Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>

Currently FI_MULTI_RECV is effectively only defined for untagged message
only. Simply expanding the definition to tagged message would cause
difficulties in either provider support or discovery.

Define FI_TAGGED_MULTI_RECV to indicate that multi recv is supported in
tagged message as well. This is only used as a capability bit. The op
flag and cq flag continues to use FI_MULTI_RECV.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>

FI_DIRECTED_RECV covers both untagged and tagged message. However, the most
often used case is for tagged message. Having a saparate bit for tagged
message allows the provider to optimize non-tagged messsage implementation
while maintain support directed recv over tagged message.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>

The new bit FI_EXACT_DIRECTED_RECV is similar to FI_DIRECTED_RECV, but
requires exact source address. I.e., the wildcard address FI_ADDR_UNSPEC
is not allowed.

It can be used alone, or be used together with FI_DIRECTED_RECV or
FI_TAGGED_DIRECTED_RECV as a modifier.

Not allowing wildcard source address allows the provider to better
optmize the receive handling.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>

Memory registration consists of two parts: map/pin the memory for local
access and export with a key for remote access. The first part is usually
heavyweight and requries kernel involvement. The second part is less
expensive and can be further separated into key allocation and key
assignment. Key allocation may needs kernel involvement, but key assignment
can be done in user space. Here sub-MR is introduced as a way to allow
separattion of the forementioned two parts, and key reservation is added
to further optimize sub-MR creation.

A sub-MR is created from an existing MR (the base MR).  It inherits the
memory mapping/pinning of the base MR but has its own access key.  The
address range exposed can be same as the the base MR or a subpart of that.
The access rights can be different, too.

Now the base MR can be created with a few extra keys reserved. These reserved
keys will be automatically used for sub-MR registration. This only applies
to FI_MR_PROV_KEY mode.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>

Add text to clarify that only FI_HMEM_SYSTEM is allowed for inject calls
if FI_MR_HMEM is required.

Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>