Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages


home | help
fi_mr(3)			   @VERSION@			      fi_mr(3)

       fi_mr - Memory region operations

       fi_mr_reg / fi_mr_regv /	fi_mr_regattr
	      Register local memory buffers for	direct fabric access

	      Deregister registered memory buffers.

	      Return  a	 local	descriptor associated with a registered	memory

	      Return the remote	key needed to access a registered  memory  re-

	      Return raw memory	region attributes.

	      Converts	a  raw memory region key into a	key that is usable for
	      data transfer operations.

	      Releases a previously mapped raw memory region key.

	      Associate	a registered memory region with	a  completion  counter
	      or an endpoint.

	      Updates the memory pages associated with a memory	region.

	      Enables a	memory region for use.

	      #include <rdma/fi_domain.h>

	      int fi_mr_reg(struct fid_domain *domain, const void *buf,	size_t len,
		  uint64_t access, uint64_t offset, uint64_t requested_key,
		  uint64_t flags, struct fid_mr	**mr, void *context);

	      int fi_mr_regv(struct fid_domain *domain,	const struct iovec * iov,
		  size_t count,	uint64_t access, uint64_t offset, uint64_t requested_key,
		  uint64_t flags, struct fid_mr	**mr, void *context);

	      int fi_mr_regattr(struct fid_domain *domain, const struct	fi_mr_attr *attr,
		  uint64_t flags, struct fid_mr	**mr);

	      int fi_close(struct fid *mr);

	      void * fi_mr_desc(struct fid_mr *mr);

	      uint64_t fi_mr_key(struct	fid_mr *mr);

	      int fi_mr_raw_attr(struct	fid_mr *mr, uint64_t *base_addr,
		  uint8_t *raw_key, size_t *key_size, uint64_t flags);

	      int fi_mr_map_raw(struct fid_domain *domain, uint64_t base_addr,
		  uint8_t *raw_key, size_t key_size, uint64_t *key, uint64_t flags);

	      int fi_mr_unmap_key(struct fid_domain *domain, uint64_t key);

	      int fi_mr_bind(struct fid_mr *mr,	struct fid *bfid, uint64_t flags);

	      int fi_mr_refresh(struct fid_mr *mr, const struct	iovec *iov, size, count,
		  uint64_t flags)

	      int fi_mr_enable(struct fid_mr *mr);

       domain Resource domain

       mr     Memory region

       bfid   Fabric identifier	of an associated resource.

	      User specified context associated	with the memory	region.

       buf    Memory buffer to register	with the fabric	hardware

       len    Length of	memory buffer to register

       iov    Vectored memory buffer.

       count  Count of vectored	buffer entries.

       access Memory access permissions	associated with	registration

       offset Optional	specified  offset  for	accessing specified registered
	      buffers.	This parameter is reserved for future use and must  be

	      Optional	requested  remote  key associated with registered buf-

       attr   Memory region attributes

       flags  Additional flags to apply	to the operation.

       Registered memory regions associate  memory  buffers  with  permissions
       granted for access by fabric resources.	A memory buffer	must be	regis-
       tered with a resource domain before it can be used as the target	 of  a
       remote  RMA  or	atomic data transfer.  Additionally, a fabric provider
       may require that	data buffers be	registered before being	used in	 local
       transfers.   Memory  registration  restrictions	are controlled using a
       separate	set of mode bits,  specified  through  the  domain  attributes
       (mr_mode	field).

       The following apply to memory registration.

       Scalable	Memory Registration
	      By  default,  memory  registration is considered scalable.  (For
	      library versions 1.4 and earlier,	this is	indicated  by  setting
	      mr_mode to FI_MR_SCALABLE, with the fi_info mode bit FI_LOCAL_MR
	      set to 0).  For versions 1.5 and later, scalable is  implied  by
	      the  lack	of any mr_mode bits being set.	The setting of mr_mode
	      bits therefore adjusts application behavior as described	below.
	      Default, scalable	registration has several properties.

       In  scalable  mode,  registration occurs	on memory address ranges.  Be-
       cause registration refers to memory regions, versus data	 buffers,  the
       address	ranges	given for a registration request do not	need to	map to
       data buffers allocated by the application at the	time the  registration
       call  is	 made.	 That is, an application can register any range	of ad-
       dresses in their	virtual	address	space, whether or not those  addresses
       are backed by physical pages or have been allocated.

       The resulting memory regions are	accessible by peers starting at	a base
       address of 0.  That is, the target address that is specified is a  byte
       offset into the registered region.

       The  application	 also  selects	the access key associated with the MR.
       The key size is restricted to a maximum of 8 bytes.

       With scalable registration, locally accessed data buffers are not  reg-
       istered.	  This	includes source	buffers	for all	transmit operations --
       sends, tagged sends, RMA, and atomics --	as well	as buffers posted  for
       receive and tagged receive operations.

	      When the FI_MR_LOCAL mode	bit is set, applications must register
	      all data buffers that will be accessed by	the local hardware and
	      provide a	valid desc parameter into applicable data transfer op-
	      erations.	 When FI_MR_LOCAL is zero, applications	 are  not  re-
	      quired  to register data buffers before using them for local op-
	      erations (e.g.  send and receive data buffers).  The desc	param-
	      eter into	data transfer operations will be ignored in this case,
	      unless otherwise required	(e.g.  se FI_MR_HMEM).	It  is	recom-
	      mended  that  applications  pass	in  NULL for desc when not re-

       A provider may hide local registration requirements  from  applications
       by  making use of an internal registration cache	or similar mechanisms.
       Such mechanisms,	however, may negatively	impact	performance  for  some
       applications, notably those which manage	their own network buffers.  In
       order to	support	as broad range of applications	as  possible,  without
       unduly  affecting  their	 performance, applications that	wish to	manage
       their own local memory registrations may	do so by using the memory reg-
       istration calls.

       Note: the FI_MR_LOCAL mr_mode bit replaces the FI_LOCAL_MR fi_info mode
       bit.  When FI_MR_LOCAL is set, FI_LOCAL_MR is ignored.

	      Raw memory regions are used to support providers with keys larg-
	      er  than	64-bits	 or  require  setup  at	 the  peer.   When the
	      FI_MR_RAW	bit is set, applications must use fi_mr_raw_attr() lo-
	      cally  and fi_mr_map_raw() at the	peer before targeting a	memory
	      region as	part of	any data transfer request.

	      The FI_MR_VIRT_ADDR bit indicates	that the  provider  references
	      memory regions by	virtual	address, rather	than a 0-based offset.
	      Peers that target	memory regions registered with FI_MR_VIRT_ADDR
	      specify the destination memory buffer using the target's virtual
	      address, with any	offset into the	region	specified  as  virtual
	      address  +  offset.   Support of this bit	typically implies that
	      peers must exchange addressing data prior	to initiating any  RMA
	      or atomic	operation.

	      When set,	all registered memory regions must be backed by	physi-
	      cal memory pages at the time the registration call is made.

	      This memory region mode indicates	that  the  provider  does  not
	      support  application requested MR	keys.  MR keys are returned by
	      the provider.  Applications that support FI_MR_PROV_KEY can  ob-
	      tain the provider	key using fi_mr_key(), unless FI_MR_RAW	is al-
	      so set.  The returned key	should then be	exchanged  with	 peers
	      prior to initiating an RMA or atomic operation.

	      FI_MR_MMU_NOTIFY is typically set	by providers that support mem-
	      ory registration against memory regions that are not necessarily
	      backed by	allocated physical pages at the	time the memory	regis-
	      tration occurs.  (That  is,  FI_MR_ALLOCATED  is	typically  0).
	      However,	such  providers	 require  that applications notify the
	      provider prior to	the MR being accessed as part of a data	trans-
	      fer  operation.  This notification informs the provider that all
	      necessary	physical pages now back	the region.  The  notification
	      is  necessary  for  providers that cannot	hook directly into the
	      operating	system page tables or  memory  management  unit.   See
	      fi_mr_refresh() for notification details.

	      This  mode bit indicates that the	provider must configure	memory
	      regions that are associated with RMA events prior	to their  use.
	      This  includes  all memory regions that are associated with com-
	      pletion counters.	 When set, applications	 must  indicate	 if  a
	      memory  region  will  be associated with a completion counter as
	      part of the region's creation.  This is done by passing  in  the
	      FI_RMA_EVENT flag	to the memory registration call.

       Such memory regions will	be created in a	disabled state and must	be as-
       sociated	with all completion counters prior to being enabled.   To  en-
       able  a memory region, the application must call	fi_mr_enable().	 After
       calling fi_mr_enable(), no further resource bindings may	be made	to the
       memory region.

	      This  mode bit indicates that the	provider associates memory re-
	      gions with endpoints rather than domains.	 Memory	 regions  that
	      are registered with the provider are created in a	disabled state
	      and must be bound	to an endpoint prior  to  being	 enabled.   To
	      bind   the  MR  with  an	endpoint,  the	application  must  use
	      fi_mr_bind().  To	enable the memory region, the application must
	      call fi_mr_enable().

	      This  mode  bit  is  associated with the FI_HMEM capability.  If
	      FI_MR_HMEM is set, the application must  register	 buffers  that
	      were  allocated using a device call and provide a	valid desc pa-
	      rameter into applicable data transfer operations	even  if  they
	      are  only	used for local operations (e.g.	 send and receive data
	      buffers).	 Device	memory must be registered using	the  fi_mr_re-
	      gattr call, with the iface and device fields filled out.

       If  FI_MR_HMEM  is  set,	 but FI_MR_LOCAL is unset, only	device buffers
       must be registered when used locally.  In this case, the	desc parameter
       passed  into  data  transfer  operations	 must either be	valid or NULL.
       Similarly, if FI_MR_LOCAL is set, but FI_MR_HMEM	is not,	the  desc  pa-
       rameter must either be valid or NULL.

       Basic Memory Registration
	      Basic  memory  registration  is  indicated  by  the  FI_MR_BASIC
	      mr_mode bit.  FI_MR_BASIC	is maintained for  backwards  compati-
	      bility  (libfabric version 1.4 or	earlier).  The behavior	of ba-
	      sic registration is equivalent to	setting	the following  mr_mode
	      bits    to    one:    FI_MR_VIRT_ADDR,	FI_MR_ALLOCATED,   and
	      FI_MR_PROV_KEY.  Additionally, providers that support basic reg-
	      istration	usually	required the fi_info mode bit FI_LOCAL_MR.  As
	      a	result,	it is recommended  that	 applications  migrating  from
	      libfabric	1.4 or earlier or wanting to support basic memory reg-
	      istration	set the	mr_mode	to FI_MR_VIRT_ADDR | FI_MR_ALLOCATED |
	      FI_MR_PROV_KEY  |	 FI_MR_LOCAL.	FI_MR_BASIC must be set	alone.
	      Other mr_mode bit	pairings are invalid.	Unlike	other  mr_mode
	      bits,  if	 FI_MR_BASIC  is set on	input to fi_getinfo(), it will
	      not be cleared by	the provider.  That is,	setting	FI_MR_BASIC to
	      one requests basic registration.

       The registrations functions -- fi_mr_reg, fi_mr_regv, and fi_mr_regattr
       -- are used to register one or more  memory  regions  with  fabric  re-
       sources.	  The  main  difference	between	registration functions are the
       number and type of parameters that they accept  as  input.   Otherwise,
       they perform the	same general function.

       By  default,  memory  registration  completes synchronously.  I.e.  the
       registration call will not return until the registration	has completed.
       Memory  registration  can complete asynchronous by binding the resource
       domain to an event queue	using the FI_REG_MR flag.  See fi_domain_bind.
       When memory registration	is asynchronous, in order to avoid a race con-
       dition between the registration call returning  and  the	 corresponding
       reading of the event from the EQ, the mr	output parameter will be writ-
       ten before any event associated with the	operation may be read  by  the
       application.   An  asynchronous	event will not be generated unless the
       registration call returns success (0).

       The fi_mr_reg call registers the	user-specified memory buffer with  the
       resource	 domain.  The buffer is	enabled	for access by the fabric hard-
       ware based on the provided access permissions.  See  the	 access	 field
       description for memory region attributes	below.

       Registered memory is associated with a local memory descriptor and, op-
       tionally, a remote memory key.  A memory	descriptor is a	provider  spe-
       cific identifier	associated with	registered memory.  Memory descriptors
       often map to hardware specific indices or keys associated with the mem-
       ory  region.  Remote memory keys	provide	limited	protection against un-
       wanted access by	a remote node.	Remote accesses	 to  a	memory	region
       must provide the	key associated with the	registration.

       Because	MR  keys  must be provided by a	remote process,	an application
       can use the requested_key parameter to indicate	that  a	 specific  key
       value be	returned.  Support for user requested keys is provider specif-
       ic and is determined by the mr_mode domain attribute.

       Remote RMA and atomic operations	indicate the location within a	regis-
       tered  memory  region by	specifying an address.	The location is	refer-
       enced by	adding the offset to either the	base virtual  address  of  the
       buffer or to 0, depending on the	mr_mode.

       The offset parameter is reserved	for future use and must	be 0.

       For  asynchronous  memory registration requests,	the result will	be re-
       ported to the user through an event queue associated with the  resource
       domain.	 If  successful, the allocated memory region structure will be
       returned	to the user through the	mr parameter.  The mr address must re-
       main  valid  until  the	registration operation completes.  The context
       specified with the registration request is returned with	the completion

       The   fi_mr_regv	 call  adds  support  for  a  scatter-gather  list  to
       fi_mr_reg.  Multiple memory buffers are registered as a	single	memory
       region.	Otherwise, the operation is the	same.

       The  fi_mr_regattr call is a more generic, extensible registration call
       that allows the user to specify the registration	request	using a	struct
       fi_mr_attr (defined below).

       Fi_close	is used	to release all resources associated with a registering
       a memory	region.	 Once unregistered, further access to  the  registered
       memory is not guaranteed.  Active or queued operations that reference a
       memory region being closed may fail or result in	 accesses  to  invalid
       memory.	 Applications  are  responsible	 for  ensuring that a MR is no
       longer needed prior to closing it.  Note	that accesses to a  closed  MR
       from  a	remote peer will result	in an error at the peer.  The state of
       the local endpoint will be unaffected.

       When closing the	MR, there must be no opened endpoints or counters  as-
       sociated	 with  the  MR.	 If resources are still	associated with	the MR
       when attempting to close, the call will return -FI_EBUSY.

       Obtains the local memory	descriptor associated with a MR.   The	memory
       registration  must  have	 completed  successfully  before invoking this

       Returns the remote protection key associated with  a  MR.   The	memory
       registration  must  have	 completed  successfully before	invoking this.
       The returned key	may be used in data transfer operations	at a peer.  If
       the FI_RAW_MR mode bit has been set for the domain, then	the memory key
       must be obtained	using the fi_mr_raw_key	function  instead.   A	return
       value  of  FI_KEY_NOTAVAIL will be returned if the registration has not
       completed or a raw memory key is	required.

       Returns the raw,	remote protection key and base address associated with
       a  MR.  The memory registration must have completed successfully	before
       invoking	this routine.  Use of this call	is required if	the  FI_RAW_MR
       mode  bit has been set by the provider; however,	it is safe to use this
       call with any memory region.

       On input, the key_size  parameter  should  indicate  the	 size  of  the
       raw_key buffer.	If the actual key is larger than what can fit into the
       buffer, it will return -FI_ETOOSMALL.  On output, key_size  is  set  to
       the  size  of  the  buffer needed to store the key, which may be	larger
       than the	input value.  The needed key_size can also be obtained through
       the mr_key_size domain attribute	(fi_domain_attr) field.

       A raw key must be mapped	by a peer before it can	be used	in data	trans-
       fer operations.	See fi_mr_map_raw below.

       Raw protection keys must	be mapped to a usable key  value  before  they
       can  be	used for data transfer operations.  The	mapping	is done	by the
       peer that initiates the RMA or atomic operation.	 The mapping  function
       takes  as  input	 the raw key and its size, and returns the mapped key.
       Use of the fi_mr_map_raw	function is  required  if  the	peer  has  the
       FI_RAW_MR  mode	bit  set,  but this routine may	be called on any valid
       key.  All mapped	keys must be freed by calling fi_mr_unmap_key when ac-
       cess to the peer	memory region is no longer necessary.

       This  call  releases any	resources that may have	been allocated as part
       of mapping a raw	memory key.  All mapped	keys must be freed before  the
       corresponding domain is closed.

       The  fi_mr_bind	function  associates a memory region with a counter or
       endpoint.  Counter bindings are needed by providers  that  support  the
       generation  of  completions based on fabric operations.	Endpoint bind-
       ings are	needed if the provider associates  memory  regions  with  end-
       points (see FI_MR_ENDPOINT).

       When  binding  with  a  counter,	the type of events tracked against the
       memory region is	based on the bitwise OR	of the following flags.

	      Generates	an event whenever a remote RMA write or	atomic	opera-
	      tion modifies the	memory region.	Use of this flag requires that
	      the endpoint through which the MR	is accessed  be	 created  with
	      the FI_RMA_EVENT capability.

       When binding the	memory region to an endpoint, flags should be 0.

       The  use	 of this call is required to notify the	provider of any	change
       to the physical	pages  backing	a  registered  memory  region  if  the
       FI_MR_MMU_NOTIFY	mode bit has been set.	This call informs the provider
       that the	page table entries associated with the region  may  have  been
       modified,  and the provider should verify and update the	registered re-
       gion accordingly.  The iov parameter is optional	and  may  be  used  to
       specify	which  portions	 of  the  registered region requires updating.
       Providers are only guaranteed to	update the specified address ranges.

       The refresh operation has the effect of disabling and  re-enabling  ac-
       cess  to	the registered region.	Any operations from peers that attempt
       to access the region will fail while the	refresh	is  occurring.	 Addi-
       tionally,  attempts  to	access the region by the local process through
       libfabric APIs may result in a page fault or other fatal	operation.

       The fi_mr_refresh call is only needed if	the physical pages might  have
       been updated after the memory region was	created.

       The  enable  call  is used with memory registration associated with the
       FI_MR_RMA_EVENT mode bit.  Memory regions created in the	disabled state
       must be explicitly enabled after	being fully configured by the applica-
       tion.  Any resource bindings to the MR must be done prior  to  enabling
       the MR.

       Memory  regions are created using the following attributes.  The	struct
       fi_mr_attr is passed into fi_mr_regattr,	but individual fields also ap-
       ply to other memory registration	calls, with the	fields passed directly
       into calls as function parameters.

	      struct fi_mr_attr	{
		  const	struct iovec *mr_iov;
		  size_t	     iov_count;
		  uint64_t	     access;
		  uint64_t	     offset;
		  uint64_t	     requested_key;
		  void		     *context;
		  size_t	     auth_key_size;
		  uint8_t	     *auth_key;
		  enum fi_hmem_iface iface;
		  union	{
		      uint64_t	       reserved;
		      int	       cuda;
		      int      ze
		  } device;

       This is an IO vector of addresses that will represent a	single	memory
       region.	The number of entries in the iovec is specified	by iov_count.

       The number of entries in	the mr_iov array.  The maximum number of memo-
       ry buffers that may be associated with a	single memory region is	speci-
       fied as the mr_iov_limit	domain attribute.  See fi_domain(3).

       Indicates  the type of operations that the local	or a peer endpoint may
       perform on registered memory region.  Supported access permissions  are
       the bitwise OR of the following flags:

	      The  memory  buffer  may be used in outgoing message data	trans-
	      fers.  This includes fi_msg and fi_tagged	send operations.

	      The memory buffer	may be used to receive inbound message	trans-
	      fers.  This includes fi_msg and fi_tagged	receive	operations.

	      The  memory buffer may be	used as	the result buffer for RMA read
	      and atomic operations on the initiator side.  Note that from the
	      viewpoint	of the application, the	memory buffer is being written
	      into by the network.

	      The memory buffer	may be used as the source buffer for RMA write
	      and atomic operations on the initiator side.  Note that from the
	      viewpoint	of the application, the	endpoint is reading  from  the
	      memory buffer and	copying	the data onto the network.

	      The  memory  buffer  may	be used	as the source buffer of	an RMA
	      read operation on	the target side.  The contents of  the	memory
	      buffer are not modified by such operations.

	      The  memory  buffer  may	be used	as the target buffer of	an RMA
	      write or atomic operation.  The contents of  the	memory	buffer
	      may be modified as a result of such operations.

       Note  that  some	 providers may not enforce fine	grained	access permis-
       sions.  For example, a memory region registered for FI_WRITE access may
       also  behave as if FI_SEND were specified as well.  Relaxed enforcement
       of such access is permitted, though not guaranteed,  provided  security
       is maintained.

       The offset field	is reserved for	future use and must be 0.

       An  application specified access	key associated with the	memory region.
       The MR key must be provided by a	remote process when performing RMA  or
       atomic  operations  to  a  memory region.  Applications can use the re-
       quested_key field to indicate that  a  specific	key  be	 used  by  the
       provider.  This allows applications to use well known key values, which
       can avoid applications needing to exchange and store keys.  Support for
       user  requested	keys  is  provider  specific  and is determined	by the
       mr_mode domain attribute.

       Application context associated with  asynchronous  memory  registration
       operations.   This  value is returned as	part of	any asynchronous event
       associated with the registration.  This field is	ignored	 for  synchro-
       nous registration calls.

       The  size  of key referenced by the auth_key field in bytes, or 0 if no
       authorization key is given.  This field is ignored unless the fabric is
       opened with API version 1.5 or greater.

       Indicates  the  key to associate	with this memory registration.	Autho-
       rization	keys are used to limit communication between endpoints.	  Only
       peer  endpoints	that  are programmed to	use the	same authorization key
       may access the memory region.  The domain  authorization	 key  will  be
       used  if	the auth_key_size provided is 0.  This field is	ignored	unless
       the fabric is opened with API version 1.5 or greater.

       Indicates the software interfaces used by the application  to  allocate
       and  manage the memory region.  This field is ignored unless the	appli-
       cation has requested the	FI_HMEM	capability.

	      Uses standard operating system calls and libraries, such as mal-
	      loc, calloc, realloc, mmap, and free.

	      Uses  Nvidia CUDA	interfaces such	as cuMemAlloc, cuMemAllocHost,
	      cuMemAllocManaged, cuMemFree, cudaMalloc,	cudaFree.

	      Uses  AMD	 ROCR  interfaces  such	 as  hsa_memory_allocate   and

	      Uses  Intel  L0  ZE  interfaces  such as zeDriverAllocSharedMem,

       Reserved	64 bits	for device identifier if using non-standard  HMEM  in-
       terface.	 This field is ignore unless the iface field is	valid.

       cuda   For FI_HMEM_CUDA,	this is	equivalent to CUdevice (int).

       ze     For FI_HMEM_ZE, this is equivalent to the	ze_device_handle_t in-
	      dex (int).

       Direct access to	an application's memory	by a remote peer requires that
       the  application	register the targeted memory buffer(s).	 This is typi-
       cally  done  by	calling	 one  of   the	 fi_mr_reg*   routines.	   For
       FI_MR_PROV_KEY, the provider will return	a key that must	be used	by the
       peer when accessing the memory region.  The application is  responsible
       for transferring	this key to the	peer.  If FI_MR_RAW mode has been set,
       the key must be retrieved using the fi_mr_raw_attr function.

       FI_RAW_MR allows	support	for providers that require more	 than  8-bytes
       for  their protection keys or need additional setup before a key	can be
       used for	transfers.  After a raw	key has	been retrieved,	it must	be ex-
       changed	with the remote	peer.  The peer	must use fi_mr_map_raw to con-
       vert the	raw key	into a usable 64-bit key.  The mapping	must  be  done
       even if the raw key is 64-bits or smaller.

       The raw key support functions are usable	with all registered memory re-
       gions, even if FI_MR_RAW	has not	been set.  It is recommended that por-
       table  applications  target  using those	interfaces; however, their use
       does carry extra	message	and memory footprint overhead, making it  less
       desirable for highly scalable apps.

       The follow flag may be specified	to any memory registration call.

	      This flag	indicates that the specified memory region will	be as-
	      sociated with a completion counter used to count RMA  operations
	      that access the MR.

	      This  flag indicates that	the underlying memory region is	backed
	      by persistent memory and will be used  in	 RMA  operations.   It
	      must  be specified if persistent completion semantics or persis-
	      tent data	transfers are required when accessing  the  registered

       Returns 0 on success.  On error,	a negative value corresponding to fab-
       ric errno is returned.

       Fabric errno values are defined in rdma/fi_errno.h.

	      The requested_key	is already in use.

	      The requested_key	is not available.  They	key may	be out of  the
	      range supported by the provider, or the provider may not support
	      user-requested memory registration keys.

	      Returned by fi_mr_bind if	the provider does not support  report-
	      ing events based on access to registered memory regions.

	      Returned	if  the	 specified  flags  are	not  supported	by the

       Many hardware NICs accessed by libfabric	require	that data  buffers  be
       registered  with	the hardware while the hardware	accesses it.  This en-
       sures that the virtual to physical address mappings for	those  buffers
       do  not change while the	transfer is occurring.	The performance	impact
       of registering memory regions can be significant.  As  a	 result,  some
       providers  make	use of a registration cache, particularly when working
       with applications that are unable to manage their own network  buffers.
       A registration cache avoids the overhead	of registering and unregister-
       ing a data buffer with each transfer.

       If a registration cache is going	to be used for host and	device memory,
       the device must support unified virtual addressing.  If the device does
       not support unified virtual addressing, either an additional  registra-
       tion  cache  is	required to track this device memory, or device	memory
       cannot be cached.

       As a general rule, if hardware requires the FI_MR_LOCAL	mode  bit  de-
       scribed	above,	but this is not	supported by the application, a	memory
       registration cache may be in use.  The following	environment  variables
       may be used to configure	registration caches.

	      This  defines  the  total	number of bytes	for all	memory regions
	      that may be tracked by the cache.	 If not	set, the cache has  no
	      limit  on	 how many bytes	may be registered and cached.  Setting
	      this will	reduce the amount of memory that is not	actively being
	      used  as	part  of  a  data  transfer  that is registered	with a
	      provider.	 By default, the cache size is unlimited.

	      This defines the total number of memory regions that may be reg-
	      istered  with the	cache.	If not set, a default limit is chosen.
	      Setting this will	reduce the number of regions that  are	regis-
	      tered,  regardless  of  their size, which	are not	actively being
	      used as part of a	data transfer.	Setting	this to	zero will dis-
	      able registration	caching.

	      The  cache  monitor  is  responsible for detecting system	memory
	      (FI_HMEM_SYSTEM) changes made between the	virtual	addresses used
	      by an application	and the	underlying physical pages.  Valid mon-
	      itor options are:	userfaultfd, memhooks, and disabled.   Select-
	      ing  disabled will turn off the registration cache.  Userfaultfd
	      is a Linux kernel	feature	used to	report virtual to physical ad-
	      dress  mapping  changes to user space.  Memhooks operates	by in-
	      tercepting relevant memory  allocation  and  deallocation	 calls
	      which may	result in the mappings changing, such as malloc, mmap,
	      free, etc.  Note that memhooks operates at the elf linker	layer,
	      and does not use glibc memory hooks.

	      The  CUDA	cache monitor is responsible for detecting CUDA	device
	      memory (FI_HMEM_CUDA) changes made between  the  device  virtual
	      addresses	used by	an application and the underlying device phys-
	      ical pages.  Valid monitor options are: 0	or 1.  Note  that  the
	      CUDA memory monitor requires a CUDA toolkit version with unified
	      virtual addressing enabled.

	      The ROCR cache monitor is	responsible for	detecting ROCR	device
	      memory  (FI_HMEM_ROCR)  changes  made between the	device virtual
	      addresses	used by	an application and the underlying device phys-
	      ical  pages.   Valid monitor options are:	0 or 1.	 Note that the
	      ROCR memory monitor requires a ROCR version with unified virtual
	      addressing enabled.

       fi_getinfo(3),	fi_endpoint(3),	 fi_domain(3),	fi_rma(3),  fi_msg(3),


Libfabric Programmer's Manual	  2020-08-11			      fi_mr(3)


Want to link to this manual page? Use this URL:

home | help